Electronic device and method for providing virtual IoT environment

Abstract

An electronic device according to various embodiments of the present disclosure may include a display, a camera, a memory for storing instructions, and a processor operably connected to the display, the camera, and the memory. Instructions stored in the memory, when executed by the processor, cause the electronic device to: activate the camera and display, on the display, an image acquired in real-time by the camera; analyzing the image to determine at least one space within the image; identifying at least one IoT (Internet of Things) device that can be placed in the at least one space; and displaying at least one virtual IoT device corresponding to the identified at least one IoT device in the determined at least one space of the image. Various other embodiments are possible.

Description

Technical Field

[0001] The present disclosure relates to an electronic device, and more particularly, to a method for providing a virtual IoT environment on an electronic device. Background Art

[0002] The Internet of Things (IoT) can be a technology in which various objects are equipped with computing and communication capabilities, enabling access via the Internet. For example, household appliances such as TVs, refrigerators, washing machines, and vacuum cleaners, various sensors such as temperature sensors and motion sensors, or ordinary devices such as lights and curtains can be used as IoT devices. A user can use an electronic device (e.g., a smart phone, a tablet PC, and a laptop PC) supporting a wireless communication function to remotely control the IoT devices.

[0003] With the development of IoT technology, a large number of IoT devices can be placed in a designated space such as a home, and in addition to the currently used IoT devices, a user may also need to use additional IoT devices. Summary of the Invention

[0004] [Technical Problem]

[0005] In the past, there has been no technology for recommending IoT devices that can be set so that a user can build an IoT environment in a designated space.

[0006] [Technical Solution]

[0007] An electronic device according to the present disclosure (or the specification or the invention) may include a display, a camera, a memory storing instructions, and a processor operably connected to the display, the camera, and the memory.

[0008] According to an embodiment, the instructions stored in the memory, when executed by the processor, may cause the electronic device to activate the camera and display an image obtained in real time through the camera on the display, analyze the image to determine at least one space within the image, identify at least one IoT (Internet of Things) device that can be set in the at least one space, and display at least one virtual IoT device corresponding to the at least one identified IoT device on at least one of the determined spaces of the image.

[0009] A method for providing a virtual IoT environment of an electronic device according to various embodiments of the present disclosure may include: activating a camera to display an image obtained in real time through the camera; analyzing the image to determine at least one space within the image; identifying at least one IoT (Internet of Things) device that can be set in the at least one space; and displaying at least one virtual IoT device corresponding to the at least one identified IoT device on at least one of the determined spaces of the image.

[0010] [Advantageous Effects of the Invention]

[0011] According to various embodiments of the present disclosure, an electronic device capable of recognizing a space and providing a virtual IoT device to a user, and a method for providing a virtual IoT environment for the electronic device may be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0013] Figure 2 is a block diagram of an electronic device according to various embodiments.

[0014] Figure 3 is a flowchart illustrating a method for providing a virtual IoT environment for an electronic device according to various embodiments.

[0015] Figure 4 is a flowchart illustrating a method for an electronic device to recognize a real object according to various embodiments.

[0016] Figure 5a and Figure 5b illustrate real IoT devices recognized from a camera image according to various embodiments.

[0017] Figure 6a and Figure 6b illustrate virtual IoT devices that can be set in a camera image according to various embodiments.

[0018] Figure 7 illustrate a camera image and control items for controlling an IoT device according to various embodiments.

[0019] Figure 8 illustrate a camera image and available items for a virtual IoT device according to various embodiments.

[0020] Figure 9 illustrate changes in a camera image and control items depending on a change in a camera viewpoint according to various embodiments.

[0021] Figure 10 illustrate an example of recommending a virtual IoT device depending on a real environment according to various embodiments. DETAILED DESCRIPTION

[0022] Figure 1 is a block diagram showing an electronic device 101 in a network environment 100 according to various embodiments. Refer to Figure 1, the electronic device 101 in the network environment 100 can communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 can communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection end 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identity module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the above components (e.g., the connection end 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the above components (e.g., the sensor module 176, the camera module 180, or the antenna module 197) may be implemented as a single integrated component (e.g., the display module 160).

[0023] The processor 120 may run software (e.g., the program 140) to control at least one other component (e.g., a hardware component or a software component) connected to the processor 120 of the electronic device 101, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., the sensor module 176 or the communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operationally independent of or combined with the main processor 121. For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or may be adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or as part of the main processor 121.

[0024] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (instead of the main processor 121) may control at least some of the functions or states related to at least one of the components of the electronic device 1011 (e.g., the display module 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control, together with the main processor 121, at least some of the functions or states related to at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190). According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 180 or the communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. For example, such learning may be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., the server 108). The learning algorithm may include, but is not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, or a combination of two or more of them, but is not limited thereto. Additionally or alternatively, the artificial intelligence model may include a software structure in addition to the hardware structure.

[0025] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0026] The program 140 may be stored as software in the memory 130, and the program 140 may include, for example, an operating system (OS) 142, middleware 144, or an application 146.

[0027] The input module 150 may receive commands or data to be used by other components of the electronic device 101 (e.g., the processor 120) from the outside of the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0028] The sound output module 155 may output a sound signal to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as playing multimedia or playing a record. The receiver may be used for receiving an incoming call. According to an embodiment, the receiver may be implemented separately from the speaker or as a part of the speaker.

[0029] The display module 160 may visually provide information to the outside (e.g., a user) of the electronic device 101. The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure an intensity of a force caused by the touch.

[0030] The audio module 170 may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module 170 may obtain a sound via the input module 150, or output a sound via the sound output module 155 or a headset of an external electronic device (e.g., the electronic device 102) directly (e.g., wiredly) or wirelessly connected to the electronic device 101.

[0031] The sensor module 176 may detect an operating state of the electronic device 101 (e.g., power or temperature) or an environmental state outside the electronic device 101 (e.g., a state of a user), and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0032] The interface 177 may support one or more specific protocols used to directly (e.g., wiredly) or wirelessly connect the electronic device 101 to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0033] The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (e.g., the electronic device 102). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headset connector).

[0034] The haptic module 179 may convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be recognized by a user via his sense of touch or kinesthesia. According to an embodiment, the haptic module 179 may include, for example, an electric motor, a piezoelectric element, or an electrical stimulator.

[0035] The camera module 180 may capture a still image or a moving image. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0036] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0037] The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0038] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently of the processor 120 (e.g., an application processor (AP)), and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). Each of these communication modules may communicate with an external electronic device via a first network 198 (e.g., a short-range communication network, such as Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network, such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or these various types of communication modules may be implemented as multiple separate components (e.g., multiple chips). The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an international mobile subscriber identity (IMSI)) stored in the user identification module 196.

[0039] The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies (e.g., New Radio (NR) access technologies). The NR access technology may support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low-latency communication (URLLC). The wireless communication module 192 may support high frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transfer rates. The wireless communication module 192 may support various technologies for ensuring performance in high frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., the electronic device 104), or a network system (e.g., the second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for achieving eMBB (e.g., 20 Gbps or greater), a loss coverage for achieving mMTC (e.g., 164 dB or less), or a U-plane latency for achieving URLLC (e.g., 0.5 ms or less for each of downlink (DL) and uplink (UL), or a round-trip of 1 ms or less).

[0040] The antenna module 197 may transmit a signal or power to the outside of the electronic device 101 (e.g., an external electronic device) or receive a signal or power from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna, and the antenna may include a radiating element formed of a conductive material or a conductive pattern formed in a substrate (e.g., a printed circuit board (PCB)) or formed on the substrate. According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme to be used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). Subsequently, a signal or power may be transmitted or received between the communication module 190 and an external electronic device via the selected at least one antenna. According to an embodiment, additional components (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as a part of the antenna module 197.

[0041] According to various embodiments, the antenna module 197 may form a millimeter-wave antenna module. According to an embodiment, the millimeter-wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., an array antenna), wherein the RFIC is disposed on a first surface (e.g., a bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a specified high-frequency band (e.g., a millimeter-wave band), and the plurality of antennas are disposed on a second surface (e.g., a top surface or a side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the specified high-frequency band.

[0042] At least some of the above components may be interconnected via an inter-peripheral communication scheme (e.g., a bus, a general-purpose input / output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0043] According to an embodiment, commands or data may be sent or received between the electronic device 101 and an external electronic device 104 via a server 108 connected to a second network 199. Each of the electronic devices 102 or 104 may be a device of the same type as the electronic device 101 or a device of a different type from the electronic device 101. According to an embodiment, all or some of the operations running on the electronic device 101 may be run on one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 is to automatically execute a function or service or is to execute a function or service in response to a request from a user or another device, the electronic device 101 may request one or more of the external electronic devices to execute at least part of the function or service instead of running the function or service, or in addition to running the function or service, the electronic device 101 may also request one or more of the external electronic devices to execute at least part of the function or service. The one or more external electronic devices that receive the request may execute the requested at least part of the function or service, or execute additional functions or additional services related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least part of a reply to the request with or without further processing of the result. For this purpose, for example, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology or IoT-related technology.

[0044] Figure 2 is a block diagram of an electronic device according to various embodiments.

[0045] Referring to Figure 2 , the electronic device 200 may include a display 210, a camera 220, a communication module 230, a processor 250, and a memory 260, and may implement various embodiments of the present disclosure even if some of the illustrated configurations are omitted or replaced. The electronic device 200 may also include Figure 1 the configuration and / or function of the electronic device 101 in

[0046] According to various embodiments, the electronic device 200 may execute a preview mode of the camera 220 to capture the surrounding environment (e.g., a house), and if there is an IoT (Internet of Things) device that can be connected in the surrounding environment, operate the identified IoT device using the electronic device 200, and if there is no connectable IoT device, display a virtual device corresponding to the IoT device, thereby displaying a virtual operation state. Hereinafter, the IoT device (or real IoT device) may indicate an electronic device capable of providing IoT functions, such as a TV, an IP camera, a lamp, or a robotic cleaner.

[0047] According to an embodiment, if there is a connectable IoT device, the electronic device 200 may automatically identify the device when capturing the device through the camera 220, establish a connection with the electronic device 200, and turn on the connected IoT device. In addition, if the user needs to purchase a specified item (or add-on) for the operation of the IoT device, the electronic device may pre-display the operation state of providing the item in the camera image (or preview image) through augmented reality (AR).

[0048] According to various embodiments, the display 210 may display an image provided from the processor 250. The display 210 may be implemented as one of a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display, but is not limited thereto. According to an embodiment, the display 210 may be implemented as a flexible display 210 in which at least a partial region is flexible, and a display region that is exposed to the outside and displays image information may vary. For example, the display 210 may be configured as a foldable display or a rollable display. According to an embodiment, the display 210 may include a touch screen that detects a touch input of a user. For example, the display 210 may include a capacitive touch screen, but is not limited thereto, and various types of touch screens, such as a pressure-sensitive and an infrared type touch screen, may be used. The display 210 may include Figure 1 at least some configurations and / or functions of the display module 160 in

[0049] According to various embodiments, the camera 220 may capture an external environment and provide image information to the processor 250. The electronic device 200 may include at least one camera located on the front side where the display 210 is placed and / or on the rear side in the opposite direction. According to an embodiment, the camera 220 may include a lens assembly including: at least one lens for collecting light emitted from an external environment (or an object); an image sensor (e.g., a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor) for converting the light collected through the lens assembly into an electrical signal to generate image data; and an image signal processor for performing various processes on the image data obtained from the image sensor. At least some of the above configurations of the camera 220 may be omitted or replaced with other configurations. The camera 220 may provide an image of the captured external environment to the processor 250 in real time through an interface (e.g., a mobile industry processor interface). The camera 220 may include Figure 1 at least some configurations and / or functions of the camera module 180 in

[0050] According to various embodiments, the communication module 230 may include various hardware and / or software elements for supporting wireless communication with an external device. For example, the communication module 230 may communicate with an external device (e.g., an IoT device) within a predetermined distance through a short-range wireless communication method such as wireless LAN (Wi-Fi) communication, Bluetooth, or ultra-wideband (UWB), and may communicate with an external base station through a cellular communication such as 4G LTE (long term evolution) and 5G NR (new radio). In addition to the above wireless communication methods, the communication module 230 may also support other standard or non-standard wireless communications. The communication module 230 may include Figure 1 at least some configurations and / or functions of the communication module 190 in

[0051] According to various embodiments, the memory 260 may include known volatile and non-volatile memories. The memory 260 may store various instructions that may be executed by the processor 250. Such instructions may include control commands such as arithmetic and logical operations, data transfer, and input / output, which may be recognized by the processor 250. The memory 260 may include Figure 1 at least some configurations and / or functions of the memory 130 in Figure 1 and may store at least some of the programs 140 in

[0052] According to various embodiments, the processor 250 is a configuration capable of performing operations or data processing related to control and / or communication of corresponding components of the electronic device 200, and may include Figure 1At least some configurations of the processor 120 in []. The processor 250 is operably, electrically, and / or functionally connected to internal components of the electronic device 200, such as a display 210, a camera 220, a communication module 230, and a memory 260.

[0053] According to various embodiments, although the computing and data processing functions that can be implemented by the processor 250 in the electronic device 200 are not limited, the present disclosure will describe various embodiments in which the electronic device 200 analyzes a real-time image obtained from the camera 220 to identify a real IoT device, sets a virtual IoT device in an empty space in the camera image, controls the operation of the real IoT device and / or the virtual IoT device, and / or provides affordance information associated with the virtual IoT device. The operations of the processor 250 described below can be executed by executing instructions stored in the memory 260.

[0054] According to various embodiments, the processor 250 can activate the camera 220 and display a camera image obtained from the camera 220 in real time on the display 210. For example, the camera image can be a preview image, and if the viewpoint of the camera changes, the surrounding environment displayed through the camera image can change.

[0055] According to various embodiments, the processor 250 can analyze the camera image to identify at least one object located within the camera image and an empty space where the object is not located. For example, the object can be a real IoT device that can provide IoT functions (e.g., a TV, a lamp, a robotic cleaner, an electric curtain, etc.), or a common object that cannot provide IoT functions (e.g., a common curtain, a sofa, a chair, etc.). The user can use the rear camera of the electronic device 200 to take pictures of a designated space such as a living room, a room, or an office. For example, the electronic device 200 can identify the objects located in the camera image and / or the empty space where the object is not located by scanning the space of the camera image.

[0056] According to various embodiments, the processor 250 may identify a real IoT device from an object located in a camera image. According to an embodiment, the processor 250 may identify a real IoT device by transmitting and receiving wireless signals using the communication module 230. For example, the communication module 230 may transmit a broadcast packet containing information such as a device name, address, and universally unique identifier (UUID) during a Bluetooth scanning process, and an IoT device located in the vicinity may perform a predetermined operation when receiving the broadcast packet, such as notifying a master device connected to the IoT device or requesting a connection. In addition, the processor 250 may identify an IoT device located in the vicinity using various short-range wireless communications such as UWB and Wi-Fi through the communication module 230, and analyze the relative direction and / or distance with respect to the electronic device 200 to determine the position of the IoT device within the camera image. In addition, the processor 250 may filter out devices that transmit signals of a predetermined or higher intensity using various short-range wireless communications, and register devices that transmit signals of a predetermined or higher intensity. The more types of short-range wireless communications used, the closer the proximity of the device can be identified more accurately by the signal strength, and the position of the device can be identified more accurately by the phase difference. Therefore, it can be utilized to set a virtual device at a more precise position during 3D rendering.

[0057] According to an embodiment, the processor 250 may identify an object in the camera image by analyzing the image of the acquired camera image. For example, the processor 250 may include an AI camera (e.g., Bixby Vision) capable of detecting various objects included in a real-time acquired camera image. Using the AI camera, the processor 250 may identify objects that do not support the function of transmitting and receiving wireless signals (e.g., curtains, lights, brackets, etc.), as well as devices that can be identified by transmitting and receiving wireless signals.

[0058] According to various embodiments, the processor 250 may display the identified real IoT device in the camera image displayed on the display 210 in a recognizable manner. For example, the processor 250 may display an indicator indicating the identified real IoT device in a manner that overlaps or is adjacent to the real IoT device, highlight the real IoT device, and / or provide a different visual effect to the area including the real IoT device.

[0059] According to various embodiments, the processor 250 may display one or more control items on the display 210 respectively corresponding to the identified real IoT devices. Here, the control items may include at least one button capable of controlling the operation of the real IoT device. When a plurality of real IoT devices are identified from the camera image, the processor 250 may display the control items corresponding to the IoT devices respectively. According to an embodiment, the processor 250 may display the control items in an area separated from the camera image on the display 210. According to another embodiment, the processor 250 may display the control items in a pop-up form (overlaying the camera image) on the display 210. According to an embodiment, the processor 250 may display the control items in a specified area based on a user input of an indicator for indicating a real IoT device or a virtual IoT device.

[0060] According to various embodiments, the processor 250 may send a control command to the real IoT device based on a user input of the control item. For example, based on a user input of the power button of the control item, the processor 250 may send a power on / off command to the connected TV through the communication module 230.

[0061] According to various embodiments, the processor 250 may identify an empty space in the surrounding environment by analyzing the camera image and determine a virtual IoT device that can be set in the empty space. The electronic device 200 may pre-store information related to various types of IoT devices or obtain the information from a cloud server, and if an empty space is identified, select an appropriate IoT device from the pre-stored information and / or the list of IoT devices received from the cloud server.

[0062] According to various embodiments, the processor 250 may display the determined virtual IoT device in the camera image. For example, the virtual IoT device may be a graphic object having substantially the same shape as the corresponding device. The processor 250 may display the virtual IoT device in an augmented reality manner on the surrounding environment of the camera image.

[0063] According to various embodiments, the processor 250 may display control items corresponding to the virtual IoT device. In the case where a real IoT device exists in the camera image, the processor 250 may display both the control items for controlling the corresponding real IoT device and the control items for controlling the corresponding virtual IoT device.

[0064] According to various embodiments, the processor 250 may display the execution status of the virtual IoT device in response to a user input. For example, if an input to the start / stop button on the control of the robot cleaner is detected, the electronic device 200 may generate a real operation screen such as the movement of the robot cleaner and display it within the camera image. The processor 250 may display the execution status of the virtual IoT device in an augmented reality manner on the surrounding environment of the camera image. Thus, even though the virtual IoT device (i.e., the robot cleaner) does not exist in the real environment, the user may predict the operation status when the robot cleaner is set in the real environment.

[0065] According to various embodiments, the processor 250 may display available items related to the virtual IoT device. The available items may be displayed in a pop-up form adjacent to the virtual IoT device in the camera image, or may be displayed in an area separate from the camera image. The available items may include text and / or image information, which includes product information of the virtual IoT device and a link to a website for purchasing the virtual IoT device. In response to a user input to the available items, the processor 250 may access an application or website where the corresponding virtual IoT device can be purchased and display the relevant web page. The user may purchase the real IoT device corresponding to the virtual IoT device from the web page and set it in the surrounding environment for utilization. According to an embodiment, the processor 250 may display the available items in a specified area based on a user input to an indicator indicating the real IoT device or the virtual IoT device.

[0066] Figure 3 is a flowchart illustrating a method for providing a virtual IoT environment of an electronic device according to various embodiments.

[0067] Figure 3 The illustrated method may be executed by an electronic device (e.g., Figure 2 the electronic device 200 in), and the above technical features will be omitted from the following description.

[0068] According to various embodiments, in operation 310, the electronic device may activate a camera (e.g., Figure 2 the camera 220 in) and display a camera image (e.g., a preview image) obtained in real time from the camera on a display (e.g., Figure 2 the display 210 in).

[0069] According to various embodiments, in operation 320, the electronic device may analyze the camera image and scan the space within the camera image. The user may capture an image of a designated space such as a living room, a room, or an office using the rear camera of the electronic device. For example, the electronic device may identify objects located within the camera image and / or empty spaces where objects are not located by scanning the space of the camera image. The electronic device may perform operations 330 to 345 based on the result of identifying the objects, and may perform operations 350 to 390 based on the result of identifying the empty spaces. Operations 330 to 345 and operations 350 to 390 may be performed in parallel with each other, and at least some operations may be performed simultaneously.

[0070] According to various embodiments, in operation 330, the electronic device may identify real IoT devices located within the camera image.

[0071] According to an embodiment, the electronic device may identify real IoT devices by transmitting and receiving wireless signals. For example, the electronic device may transmit a broadcast packet containing information such as a device name, an address, and a Universally Unique Identifier (UUID) during a Bluetooth scanning process, and an IoT device located in the vicinity may perform a predetermined operation such as notifying a master device connected to the IoT device or making a connection request when the broadcast packet is received. In addition, the processor 250 may identify IoT devices located in the vicinity using various short-range wireless communications such as UWB and Wi-Fi, and analyze the relative direction and / or distance with respect to the electronic device to determine the position of the IoT device within the camera image. In addition, the electronic device may filter out devices that transmit signals of a predetermined or higher intensity using various short-range wireless communications, and register devices that transmit signals of a predetermined or higher intensity. The more types of short-range wireless communications used, the closer the proximity of the device can be more accurately identified by the signal intensity, and the position of the device can be more accurately identified by the phase difference. Therefore, it can be used to set virtual devices at more precise positions during 3D rendering.

[0072] According to an embodiment, the electronic device may identify an object within the camera image by image analysis of the acquired camera image. For example, the electronic device may include an AI camera (e.g., Bixby Vision) capable of detecting various objects included in a camera image acquired in real time. Using the AI camera, the electronic device may identify objects that do not support wireless signal transmission and reception functions (e.g., curtains, lights, brackets, etc.), as well as devices that can be identified by wireless signal transmission and reception.

[0073] According to an embodiment, the electronic device may establish a connection with the identified real IoT device. For example, if a connection with a specified IoT device is not currently established, the electronic device may establish a connection via short-range wireless communication (e.g., Wi-Fi) and register the connection information with a cloud server that provides IoT services.

[0074] According to various embodiments, in operation 335, the electronic device may display the real IoT device identified in the camera image in a recognizable manner. For example, the electronic device may display an indicator indicating the identified real IoT device in a manner that overlaps or is adjacent to the real IoT device, highlight the real IoT device, and / or provide a different visual effect to the area including the real IoT device.

[0075] According to an embodiment, the electronic device may display one or more controls corresponding to the identified real IoT device on the display. The electronic device may display a control area including at least one control in a manner separated from the area where the camera image is displayed (e.g., at the bottom of the camera image), and as will be described below, the control area may include at least one control corresponding to the real IoT device and at least one control corresponding to the virtual IoT device.

[0076] According to various embodiments, in operation 340, the electronic device may receive a user input for a control. For example, the controls corresponding to a TV as a real IoT device may include a power button, a channel control button, and / or a volume control button, and the user input may be an input to a specified button of the control.

[0077] According to various embodiments, in operation 345, the electronic device may send an operation control command for the real IoT device in response to the user input for the control. For example, if the user selects the TV power button on the control, a TV on / off command may be directly sent to the TV via the established short-range wireless communication and / or sent to the TV via a cloud server on the network.

[0078] According to various embodiments, in operation 350, the electronic device may identify an empty space in the camera image. For example, the electronic device may identify an object or a space in the camera image by transmitting / receiving wireless signals and / or using image analysis of an AI camera, and may identify a space having a predetermined size or larger where no object is identified as an empty space.

[0079] According to various embodiments, in operation 355, the electronic device may determine virtual IoT devices that can be set in the identified empty space. The electronic device may pre-store information related to various types of IoT devices or obtain information from a cloud server, and if an empty space is identified, select appropriate IoT devices from the pre-stored information and / or the list of IoT devices received from the cloud server.

[0080] According to an embodiment, the electronic device may identify the pattern of the identified space, extract their features, and determine virtual IoT devices suitable for the space. For example, if the environment captured by a camera image has many objects and little empty space and is messy, the electronic device may select a device that helps with tidying up and has a small volume (e.g., a handheld cleaner). Alternatively, if the captured environment is in a well-organized state with many objects and little empty space, the electronic device may select an additional device that is convenient and helps with better organizing the surrounding environment or a device with a small volume (e.g., an air purifier, a water purifier, an air freshener, a smart photo frame, etc.). Alternatively, if the captured environment has few objects and much empty space and is messy, the electronic device may select a device that helps with tidying up and has a large volume (e.g., a robotic cleaner, a storage room, etc.). Alternatively, if the captured environment is in a well-organized state with few objects and much empty space, the electronic device may select an additional device that is convenient and helps with better organizing the surrounding environment or a device with a large volume (e.g., a robotic cleaner, a smart speaker, etc.).

[0081] According to various embodiments, in operation 360, the electronic device may display the determined virtual IoT devices in the camera image. The electronic device may display the virtual IoT devices in an augmented reality manner on the surrounding environment of the camera image.

[0082] According to an embodiment, the electronic device may display the virtual IoT devices so as to be distinguishable from objects including real IoT devices in the camera image. In addition, the electronic device may display an indicator indicating the virtual IoT device adjacent to or overlapping with the virtual IoT device, and display control items corresponding to the virtual IoT device.

[0083] According to various embodiments, in operation 370, the electronic device may receive a user input corresponding to the control item of the virtual IoT device. For example, if the selected virtual IoT device is a robotic cleaner, the control item may include a start / stop button of the robotic cleaner.

[0084] According to various embodiments, in operation 375, in response to a user input, the electronic device may display an operation state of a virtual IoT device. The electronic device may display the operation state of the virtual IoT device in an augmented reality manner on the surrounding environment of the camera image. For example, if an input to a start / stop button on a control item of a robotic cleaner is detected, the electronic device may generate a real operation screen such as the movement of the robotic cleaner and display it in the camera image. Thus, although the robotic cleaner as a virtual IoT device does not exist in the real environment, the user can predict the operation state of the robotic cleaner when it is actually set up.

[0085] According to various embodiments, in operation 380, the electronic device may display available items related to the virtual IoT device. The available items may be displayed in a pop-up form adjacent to the virtual IoT device in the camera image, or may be displayed in an area separated from the camera image. The available items may include text and / or image information, which includes product information of the virtual IoT device and a link to a website for purchasing the virtual IoT device.

[0086] According to various embodiments, in operation 385, the electronic device may receive a user input to the available items. According to various embodiments, in operation 390, the electronic device may access a website capable of purchasing the corresponding virtual Internet of Things device in response to the user input and display the relevant web page.

[0087] Figure 4 is a flowchart illustrating a method for an electronic device to identify a real object according to various embodiments.

[0088] Figure 4 The illustrated method may be executed by an electronic device (e.g., Figure 2 the electronic device 200 in), and the description of the above technical features will be omitted below.

[0089] According to various embodiments, in operation 410, the electronic device may activate a camera (e.g., Figure 2 the camera 220 in) to perform a preview mode, and display the acquired camera image (or preview image) in real time on a display (e.g., Figure 2 the display 210 in). The user may use the rear camera of the electronic device to capture a designated space such as a living room, a room, or an office, and the captured camera image may include a plurality of objects that belong or do not belong to IoT devices.

[0090] According to various embodiments, in operation 420, the electronic device may identify a short-range wireless communication signal transmitted from an adjacent IoT device. According to an embodiment, the electronic device may support short-range wireless communications such as Wi-Fi, Bluetooth (or Bluetooth Low Energy (BLE)), and Ultra-Wideband (UWB). The electronic device may identify an adjacent IoT device by transmitting and receiving short-range wireless communication signals. For example, the electronic device may transmit a broadcast packet containing information such as a device name, address, and Universally Unique Identifier (UUID) during a Bluetooth scanning process, and an IoT device located in the vicinity may perform a predetermined operation when receiving the broadcast packet, such as notifying the master device connected to the IoT device or making a connection request. In addition, the electronic device may use various short-range wireless communications such as UWB and Wi-Fi to identify IoT devices located nearby, and analyze the relative direction and / or distance with respect to the electronic device to determine the position of the IoT device within the camera image. In addition, the electronic device may filter out devices that transmit signals of a predetermined intensity or higher using various short-range wireless communications, and register devices that transmit signals of a predetermined intensity or higher. The more types of short-range wireless communications used, the closer the proximity of the device can be identified more accurately by the signal intensity, and the position of the device can be identified more accurately by the phase difference. Therefore, it can be utilized to set virtual devices at more precise positions during 3D rendering.

[0091] According to various embodiments, in operation 430, the electronic device may use an AI camera to identify at least one object included in the camera image. For example, the electronic device may include an AI camera (e.g., Bixby Vision) capable of detecting various objects included in a camera image acquired in real time. Using the AI camera, the electronic device may identify objects that do not support wireless signal transmission and reception functions (e.g., curtains, lights, brackets, etc.), as well as devices that can be identified through wireless signal transmission and reception. According to various embodiments, operations 420 and 430 may be performed sequentially or at least partially simultaneously.

[0092] According to various embodiments, in operation 440, the electronic device may identify an object displayed within the camera image. For example, the electronic device may identify at least one object included in the camera image by wireless signal transmission and reception in operation 420 and / or object recognition using an AI camera in operation 430. The identified objects may include: IoT devices equipped with IoT functions, such as TVs, air conditioners, or robotic cleaners; objects that do not currently have IoT functions but can have IoT functions added to them through a hub, such as lights or curtains; and / or objects that cannot provide IoT functions, such as flower pots or sofas.

[0093] According to various embodiments, in operation 450, the electronic device may identify an object belonging to the IoT device category from among the objects identified in the camera image. For example, the electronic device may identify, as an IoT device, an object belonging to the IoT device list stored in the memory and / or the database of the cloud server from among the objects identified in operation 440. According to an embodiment, the electronic device may identify, from the database, a device that can be added with IoT functionality among ordinary objects (e.g., curtains, lights, brackets, etc.) that do not support wireless signal transmission and reception, in order to distinguish it from other objects.

[0094] Figure 5a and Figure 5b The illustration shows a real IoT device identified from a camera image according to various embodiments.

[0095] Figure 5a and Figure 5b The illustration shows a camera image captured in real time by a camera (e.g., camera 220 in Figure 2 ) of an electronic device (e.g., the electronic device 200 in Figure 2 ) and displayed on a display (e.g., display 210 in Figure 2 ).

[0096] Referring to Figure 5a and Figure 5b , a TV 521, which is an IoT device capable of being connected to the electronic device via wireless communication, a light 561, which is an object capable of providing IoT functionality by being connected to an IoT hub, and a curtain that can provide IoT functionality by installing an additional component (or item) may be located within the captured surrounding environment.

[0097] According to an embodiment, an electronic device (e.g., the electronic device 200 in Figure 2 ) may identify one or more objects located within the camera image 510 by identifying a short-range wireless communication signal received from an external device and / or image analysis of an AI camera (e.g., Bixby Vision), and identify a real IoT device from among the identified objects. For example, among the TV 521, the light 561, and the curtain, which are objects identified in Figure 5a , the electronic device may identify the TV521 belonging to the IoT device list of the database as a real IoT device.

[0098] According to an embodiment, the electronic device may display an indicator 520 indicating the identified real IoT device (e.g., TV 521) within the camera image 510. The indicator 520 may be displayed to overlap and / or be adjacent to the real IoT device.

[0099] According to an embodiment, the electronic device may display a control item 525 capable of controlling the operation of the identified real IoT device. For example, the control item 525 corresponding to the real IoT device (i.e., TV 521) may include a power button, a channel control button, and / or a volume control button. According to an embodiment, as Figure 7 and Figure 8 shown, the electronic device may display at least one control item in an area different from the camera image. According to another embodiment, the electronic device may display the control item 525 in a pop-up form in the area of the camera image 510.

[0100] According to an embodiment, the electronic device may send an operation control command of the real IoT device in response to a user input on the control item 525. For example, if the user selects the power button of the TV 521 on the control item 525, the on / off command for the TV 521 may be directly sent to the TV 521 through short-range wireless communication established between the TV 521 and the communication module (e.g., Figure 2 the communication module 230 in Figure 2 ), and / or it may be sent to the TV 521 through a cloud server on the network. The TV 521 may be turned on according to the control command received from the electronic device. Accordingly, the user may recognize that the real TV 521 is turned on in the camera image 530.

[0101] According to an embodiment, the electronic device may provide an available item 540 corresponding to the identified real IoT device. Referring to Figure 5a , the available item 540 may include text and / or image information about additional functions that can be further provided by the real IoT device, and may include a link 545 for accessing a website that provides additional functions or sells additional products. If the user selects the link 545 in the available item 540, the linked website may be displayed, and the user may obtain additional information about the IoT device or purchase additional products from the website.

[0102] Referring to Figure 5b , the lamp 561 located within the captured environment may be an object that is connected to the IoT hub to provide IoT functions but is not yet connected to the electronic device.

[0103] According to an embodiment, the electronic device may display an indicator 560 corresponding to the recognized object (i.e., the lamp 561) within the camera image 550. For example, the electronic device may display the indicator 560 adjacent to or at least partially overlapping the object (i.e., the lamp 561). The electronic device may display a control item 565 corresponding to the lamp 561, and based on a user input to the control item 565, send a control command to the IoT hub connected to the lamp 561 to turn on the lamp 561. According to an embodiment, if a touch input to the indicator 560 is detected, the electronic device may display the control item 565 in an area separated from the camera image 550, or display the control item 565 as a pop-up form on at least a part of the camera image 550.

[0104] According to an embodiment, the electronic device may display an available item 580 corresponding to the object. For example, the available item 580 corresponding to the lamp 561, which is an object not yet connected to the electronic device, may include text and / or image information indicating that it is connectable, and a link 585 to an application or website that can be accessed to perform the connection. If the user selects the link 585 of the available item 580, the electronic device may attempt to connect and register with the hub connected to the lamp. According to an embodiment, if a touch input to the indicator 560 is detected, the electronic device may display the available item 580 in an area separated from the camera image 550, or display it as a pop-up form on at least a part of the camera image 550.

[0105] According to an embodiment, if the electronic device is connected to the lamp (or the hub connected to the lamp), it may display a control item corresponding to the lamp. Thereafter, in response to a user input to the control item, the electronic device may send a control command to the lamp, and thus, may turn on the lamp 561, as Figure 5b shown.

[0106] Figure 6a and Figure 6b illustrate virtual IoT devices that can be set in a camera image according to various embodiments.

[0107] According to various embodiments, an electronic device (e.g., Figure 2 the electronic device 220 in

[0108] may determine at least one space in the camera image, identify an IoT device that can be set in the determined space, and display a virtual IoT device corresponding to the identified IoT device in the determined space within the image. Here, the virtual IoT device may be a graphical object in the form of a real IoT device. Figure 6a, the electronic device may identify the space between the TV and the curtain in the camera image 610 captured in real time as an empty space. The electronic device may determine the robot cleaner as an IoT device that can be set in the empty space, and display a virtual IoT device 621 in the camera image, where the virtual IoT device 621 is a graphical object in the form of a robot cleaner. The electronic device may display an indicator 620 adjacent to the virtual IoT device 621 to indicate the virtual IoT device 621.

[0109] According to an embodiment, the electronic device may display a control item 625 capable of controlling the virtual operation of the virtual IoT device 621. The control item 625 corresponding to the virtual IoT device 621 may be substantially the same as the control item corresponding to a real IoT device (e.g., Figure 5a the control item 525 in).

[0110] According to an embodiment, based on a user input to the control item 625, the electronic device may display a virtual operation state 622 corresponding to the operation state of the virtual IoT device 621 during the operation within the camera image 630. For example, if the start button for the robot cleaner is pressed, the robot cleaner may move to another space in the house. The virtual operation state 622 may display the same effect as the movement of the real robot cleaner as the movement of the graphical object. Thus, through the image 630, the user can predict the real operation state when the corresponding IoT device is purchased and installed, and the real environment and the operation state of the virtual IoT device 621 overlap with each other in the image 630.

[0111] According to an embodiment, the electronic device may provide an offer item 640 corresponding to the virtual IoT device 621. Referring to Figure 6a , it may include text and / or image information, which includes information about the robot cleaner corresponding to the virtual IoT device 621 and a link 645 pointing to an application or website for purchasing the robot cleaner. If the user selects the link of the offer item 640, the electronic device may access the application or website for purchasing the robot cleaner and provide the corresponding screen to the user through the display.

[0112] Referring to Figure 6b , among the objects recognized from the camera image 650, the curtain 661 may be an object that can provide IoT functions by installing additional products. For example, different from the curtain in the real environment, an electric curtain added with an automatic opening / closing device can provide the function of automatically opening and closing the electric curtain through an input on the electronic device.

[0113] According to an embodiment, an electronic device may display an indicator 660 corresponding to an identified object (e.g., an electric curtain) and a control item 665 capable of controlling the electric curtain. Based on a user input to the control item, the electronic device may display a virtual operation state 662 of the electric curtain as a virtual IoT device. For example, if the user selects a close button on the control item 665, the electronic device may display the virtual operation state 662 within the camera image 670, which provides an effect similar to opening the curtain.

[0114] According to an embodiment, the electronic device may provide an available item 680 corresponding to the virtual IoT device. Referring to Figure 6b , it may include text and / or image information, which includes information about the electric curtain corresponding to the virtual IoT device and a link 685 to an application or website for purchasing the electric curtain. If the user selects the link 685 of the available item 680, the electronic device may access the application or website for purchasing the electric curtain and provide its corresponding screen to the user through the display.

[0115] Figure 7 The figure illustrates a camera image and control items for controlling an IoT device according to various embodiments.

[0116] According to various embodiments, an electronic device 700 (e.g., Figure 2 the electronic device 200 in Figure 2 ) may display a camera image 710 (or preview image) obtained in real time through a camera (e.g., Figure 2 the camera 220 in Figure 7 ) on a first area of a display 701 (e.g.,

[0117] the display 210 in Figure 7 ). The electronic device 700 may display control items 731, 733, 735, and 737 corresponding to respective real IoT devices identified from the camera image 710 and / or virtual IoT devices set in the space of the camera image 710 on a second area 760 (or control item area) of the display 701. Referring to

[0118] According to an embodiment, the electronic device 700 may identify one or more objects located in the camera image 710 by recognizing a short-range wireless communication signal and / or image analysis of an AI camera (e.g., Bixby Vision), and identify a genuine IoT device from the identified objects. For example, the electronic device 700 may identify the TV 721 and the lamp 723, which belong to the list of IoT devices in the database, among the objects such as the TV 721, the lamp 723, and the curtain 727 identified in Figure 7 as genuine IoT devices.

[0119] According to an embodiment, the electronic device 700 may determine an IoT device to be set in the empty space within the camera image 710 and display a virtual IoT device corresponding to the determined IoT device in the camera image 710. For example, the electronic device 700 may determine that the robot cleaner 725 is an IoT device that can be set in the space between the TV 721 and the curtain 727, and determine an electric curtain that can replace the curtain 727, and display the virtual IoT device 727 corresponding to the electric curtain.

[0120] According to an embodiment, the electronic device 700 may display control items 731, 733, 735, and 737 corresponding to the genuine IoT device and / or the virtual IoT device, respectively, in a second area of the display 701. For example, referring to Figure 7 , a control item 731 capable of controlling the TV 721 as a genuine IoT device, a control item 733 capable of controlling the lamp 723, a control item 735 capable of controlling the robot cleaner 725 as a virtual IoT device, and a control item 737 capable of controlling the electric curtain 727 may be displayed. Each of the control items 731, 733, 735, and 737 may include image / text information indicating the IoT device, the current state, and / or at least one button capable of controlling the operation of the device. According to an embodiment, the electronic device 700 may provide an integrated control item 738 capable of controlling multiple IoT devices at once. For example, if the good morning button of the integrated control item 738 is selected, control may be executed such that the electric curtain 727 is opened, the lamp 723 is turned off, and the robot cleaner 725 is operated.

[0121] According to an embodiment, if a genuine IoT device is recognized from the camera image, or if a virtual IoT device is determined by image analysis, the electronic device 700 may display a control item corresponding to the genuine IoT device or the virtual IoT device in one area of the display. According to another embodiment, based on a user input to an indicator indicating the genuine IoT device or the virtual IoT device, the electronic device 700 may display a control item in one area of the display.

[0122] According to an embodiment, the electronic device 700 may send an operation control command of a real IoT device based on a user input corresponding to a control item of the real IoT device. In Figure 7 in (a) of Figure 7 , if the user selects the close button on the control item 737 of the electric curtain 727, the electronic device 700 may send a control command to close the electric curtain 727. Further, in

[0123] According to an embodiment, based on a user input corresponding to a control item of a virtual IoT device, the electronic device 700 may display a virtual operation state corresponding to an operation state of the IoT device during an operation in the camera image 710. In Figure 7 in (c) of Figure 7 , if the user selects the start button of the control item 735 of the robot cleaner 725 (i.e., the virtual IoT device), as

[0124] Figure 8 shown in (d) of

[0125] the electronic device 700 may display a virtual operation state 740 within the camera image 710, and the virtual operation state 740 includes the movement of a graphic object showing the same effect as the movement of the real robot cleaner 725. Figure 8

[0126] Figure 8 Referring to Figure 8 , the electronic device 800 may display the camera image 810 on a first area of the display 801, and display display control items 831, 833, 835, and 837 corresponding to real IoT devices and virtual IoT devices respectively on a second area of the display 801. In Figure 8 in (a) of Figure 8 , if the user presses the close button of the electric curtain as the virtual IoT device, the electronic device 800 may implement a virtual operation state showing the same effect as the closing of the electric curtain within the camera image 810, as

[0127] According to an embodiment, the electronic device 800 may provide an item 880 corresponding to a virtual IoT device. For example, the item 880 may include text and / or image information, the text and / or image information including information on an electric curtain corresponding to the virtual IoT device, and a link 885 pointing to an application or website for purchasing the electric curtain. If the user selects the link 885 of the item 880, the electronic device 800 may access the application or website for purchasing the electric curtain and provide the corresponding screen to the user through the display 801.

[0128] Figure 9 The figure shows changes in a camera image and control items depending on a change in a camera viewpoint according to various embodiments.

[0129] According to an embodiment, when changing the viewpoint of a camera (e.g., Figure 2 the camera 220 in Figure 2 such that a displayed real IoT device is added to be shown in the image, the electronic device 900 (e.g.,

[0130] the electronic device 200 in

[0131] Figure 9 (A) of Figure 9 shows a state where the camera is capturing the left side of the surrounding environment. From Figure 9 the viewpoint (A), the electric curtain 971 can be identified as a real IoT device in the camera image. In addition, the electronic device 900 may determine an empty space in the camera image where an IoT device can be set and a virtual IoT device that can be set in the empty space. Referring to

[0132] Figure 9 (B) of Figure 9From the viewpoint (B), the lamp 973 and the electric curtain 971 can be recognized as real IoT devices from the camera image. In addition, the electronic device 900 can determine that the robot cleaner can be set in the lower space 982. The electronic device 900 can also display, on the display 901, a control item 99 for the lamp 973, which is a further recognized real IoT device.

[0133] Figure 9 The (C) diagram shows the state where the camera is capturing the right side of the surrounding environment. From Figure 9 the viewpoint (C), the lamp 973 and the electric curtain 971 can be recognized as real IoT devices from the camera image. In addition, the electronic device 900 can determine that the TV can be set as a virtual IoT device in the right space 984. The electronic device 900 can display a virtual object of the TV within the camera image and further display a control item 994 for the TV. In addition, depending on the change of the camera viewpoint, Figure 9 the space 982 for setting the robot cleaner, which was recognized in (A) and (B), may become invisible. The electronic device 900 can remove the virtual object and the control item 992 of the robot cleaner according to the change of the viewpoint.

[0134] Figure 10 The diagram shows an example of recommending virtual IoT devices according to a real environment according to various embodiments.

[0135] Figure 10 (a) and (b) show different surrounding environments. The electronic device can analyze the objects and empty spaces in the camera image and determine the virtual IoT devices that can be set in the empty spaces according to the attributes of the surrounding environment. For example, as Figure 10 shown in (a), multiple empty spaces 1023, 1025, and 1027 can be recognized within the environment being captured, or as Figure 10 shown in (b), only an empty space can be recognized in the narrow lower area 1071 within the environment being captured.

[0136] According to an embodiment, if the environment captured by the camera image 1010 is chaotic, with many objects and few empty spaces, a device that helps with tidying up and has a small volume (e.g., a handheld cleaner) can be selected.

[0137] According to an embodiment, if the captured environment is in a well-organized state with many objects and few empty spaces, an additional device that is convenient and helps to better organize the surrounding environment (e.g., an air purifier, a water purifier, an air freshener, a smart photo frame, etc.) or a device with a small volume can be selected.

[0138] In an embodiment, if the captured environment is cluttered with few objects and many empty spaces, a device that helps with organization and has a large volume (e.g., a robotic cleaner, a storage room, etc.) may be selected. Alternatively, if the captured environment is in a well-organized state with few objects and many empty spaces, an additional device that is convenient and helps better organize the surrounding environment or a device with a large volume (e.g., a robotic cleaner, a smart speaker, etc.) may be selected.

[0139] According to an embodiment, the electronic device may set a virtual IoT device in an empty space of the surrounding environment within the camera image or display it as overlapping with other objects. The electronic device may preferentially set the virtual IoT device in an empty space, and if only small empty spaces exist in the surrounding environment, set the virtual IoT device to overlap with the empty spaces and surrounding objects, and if no empty space exists, set the IoT device to overlap with other objects at an optimal position.

[0140] According to an embodiment, if there is no connectable real IoT device in the surrounding environment captured in the camera image and if a new IoT device needs to be purchased, the electronic device may recommend and display a virtual IoT device so that the user can experience it in advance. In addition, if there is no connectable real IoT device in the surrounding environment captured in the camera image and if an additional component is provided to enable the IoT function when connected to a general device, the electronic device may induce the purchase of the additional component for IoT execution.

[0141] According to an embodiment, if there is a connectable real IoT device in the surrounding environment captured in the camera image and if the IoT device has never been connected to the electronic device and is in a closed state, the electronic device may recommend turning on the IoT device to operate and use the IoT device after connection. In addition, if the IoT device has never been connected to the electronic device and is in an open state, the electronic device may recommend inducing connection for operation, connecting, and then using the IoT device. In addition, if the IoT device has been connected to the electronic device and is currently in a closed state, the electronic device may induce turning on the IoT device for operation and then continue to use the IoT device. In addition, if the IoT device has been connected to the electronic device and is currently in an open state, the electronic device may induce continuous use of the IoT device and provide suggestions on other IoT devices that will be beneficial to use together.

[0142] An electronic device according to various embodiments of the present disclosure may include a display, a camera, a memory storing instructions, and a processor operatively connected to the display, the camera, and the memory.

[0143] According to an embodiment, instructions stored in a memory, when executed by a processor, may cause an electronic device to: activate a camera and display, on a display, an image obtained in real time through the camera, analyze the image to determine at least one space within the image, identify at least one Internet of Things (IoT) device that can be disposed in the at least one space, and display, on at least one determined space of the image, at least one virtual IoT device corresponding to the at least one identified IoT device.

[0144] According to an embodiment, the instructions may cause the electronic device to: identify, through analysis of the image, at least one real object included in the image, and identify an IoT device connectable to the identified real object.

[0145] According to an embodiment, the instructions may cause the electronic device to: display, within the image, a virtual operation state corresponding to an operation state of an IoT device during operation.

[0146] According to an embodiment, the instructions may cause the electronic device to: display at least one control item corresponding to the virtual IoT device, and display, within the image, a virtual operation state based on a user input to the control item.

[0147] According to an embodiment, the instructions may cause the electronic device to: when displaying a plurality of virtual IoT devices, display a plurality of control items respectively corresponding to the plurality of virtual IoT devices.

[0148] According to an embodiment, the instructions may cause the electronic device to: display an available item corresponding to the virtual IoT device, and provide, based on a user input to the available item, a purchase screen for purchasing an IoT device corresponding to the virtual IoT device.

[0149] According to an embodiment, the electronic device may further include a communication module supporting short-range wireless communication, and the instructions may cause the electronic device to use the communication module to identify at least one adjacent real IoT device, and identify the identified real IoT device from the image.

[0150] According to an embodiment, the instructions may cause the electronic device to: display at least one control item corresponding to the real IoT device, and send, based on a user input to the control item, a control signal indicating an operation of the real IoT device.

[0151] According to an embodiment, if a real IoT device is added to be displayed within the image due to a change in the viewpoint of the camera, display a control item corresponding to the added real IoT device, and if a real IoT device displayed within the image is excluded due to a change in the viewpoint of the camera, remove a control item corresponding to the excluded real IoT device.

[0152] According to an embodiment, an instruction may cause an electronic device to identify at least one object included in an image through analysis of the image, and determine at least one IoT device that can be disposed in at least one space based on at least one of the size, distribution, or density of the identified real object.

[0153] A method for providing a virtual IoT environment of an electronic device according to various embodiments of the present disclosure may include: activating a camera to display an image acquired in real time through the camera, analyzing the image to determine at least one space within the image, identifying at least one IoT (Internet of Things) device that can be disposed in the at least one space, and displaying at least one virtual IoT device corresponding to the at least one identified IoT device on at least one determined space of the image.

[0154] According to an embodiment, the method may further include: displaying a virtual operation state corresponding to an operation state of the IoT device during operation within the image.

[0155] According to an embodiment, the method may further include: displaying at least one control item corresponding to the virtual IoT device, and the display of the virtual operation state may include displaying the virtual operation state within the image based on a user input to the control item.

[0156] According to an embodiment, the method may further include: displaying an available item corresponding to the virtual IoT device, and providing a purchase screen for purchasing an IoT device corresponding to the virtual IoT device based on a user input to the available item.

[0157] According to an embodiment, the method may further include: identifying at least one adjacent real IoT device, and identifying the identified real IoT device from the image.

[0158] An electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to the above-described electronic devices.

[0159] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for the corresponding embodiments. For the description of the drawings, like reference numerals may be used to refer to like or related elements. It will be understood that a singular noun corresponding to a term may include one or more things, unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any one or all possible combinations of the items listed together in the corresponding one of the plurality of phrases. As used herein, terms such as "first" and "second" or "1st" and "2nd" may be used to simply distinguish a corresponding component from another component, and do not limit the component in other respects (e.g., importance or order). It will be understood that, in the case where the terms "operably" or "communicatively" are used or where the terms "operably" or "communicatively" are not used, if one element (e.g., a first element) is referred to as "coupled with another element (e.g., a second element)", "coupled to another element (e.g., a second element)", "connected with another element (e.g., a second element)", or "connected to another element (e.g., a second element)", it means that the one element can be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0160] As used in connection with the various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware and may be used interchangeably with other terms (e.g., "logic", "logic block", "portion", or "circuit"). A module may be a single integrated component adapted to perform one or more functions or the smallest unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0161] The various embodiments described herein can be implemented as software (e.g., program 140) including one or more instructions that are readable by a machine (e.g., electronic device 101) and stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can call at least one of the one or more instructions stored in the storage medium and run the at least one instruction with or without using one or more other components. This enables the machine to operate to perform at least one function in accordance with the at least one instruction called. The one or more instructions can include code generated by a compiler or code that can be run by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Herein, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between data being stored semi-permanently in the storage medium and data being stored temporarily in the storage medium.

[0162] According to an embodiment, a method according to various embodiments of the present disclosure can be included and provided in a computer program product. The computer program product can be traded between a seller and a purchaser as a product. The computer program product can be published in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or can be published online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM), or can be directly distributed (e.g., downloaded or uploaded) between two user devices (e.g., smart phones). If it is published online, at least part of the computer program product can be generated temporarily, or at least part of the computer program product can be stored at least temporarily in a machine-readable storage medium (such as the memory of a manufacturer's server, an application store's server, or a forwarding server).

[0163] According to various embodiments, each of the above components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be separately provided in different components. According to various embodiments, one or more of the above components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding one of the multiple components performed the one or more functions before integration. According to various embodiments, the operations performed by a module, a program, or another component may be performed sequentially, in parallel, repeatedly, or in a heuristic manner, or one or more of the operations may be run in a different order or omitted, or one or more other operations may be added.

Claims

1. An electronic device, comprising: a display; a camera; a memory configured to store instructions; and a processor operably connected to the display, the camera, and the memory, wherein, when the instructions stored in the memory are executed by the processor, the electronic device is caused to: activate the camera and display, on the display, an image acquired in real time through the camera; analyze the image to determine at least one space within the image; identify at least one Internet of Things (IoT) device that can be disposed in the at least one space; and display, on at least one determined space of the image, at least one virtual IoT device corresponding to the at least one identified IoT device.

2. The electronic device according to claim 1, wherein, The instructions are configured to cause the electronic device to: identify, through analysis of the image, at least one real object included in the image, and identify an IoT device that can be connected to the identified real object.

3. The electronic device according to claim 1 or 2, wherein, The instructions are configured to cause the electronic device to: display, within the image, a virtual operating state corresponding to an operating state of the IoT device during operation.

4. The electronic device according to any one of claims 1-3, wherein, The instructions are configured to cause the electronic device to: display at least one control item corresponding to the virtual IoT device, and display, within the image, the virtual operating state based on a user input to the control item.

5. The electronic device according to claim 4, wherein, The instructions are configured to cause the electronic device to: when multiple virtual IoT devices are displayed, display multiple control items respectively corresponding to the multiple virtual IoT devices.

6. The electronic device according to any one of claims 1-5, wherein, The instructions are configured to cause the electronic device to: display an available item corresponding to the virtual IoT device, and provide, based on a user input to the available item, a purchase screen for purchasing an IoT device corresponding to the virtual IoT device.

7. The electronic device according to any one of claims 1-6 further includes a communication module that supports short-range wireless communication, wherein, The instructions are configured to cause the electronic device to: identify at least one adjacent real IoT device using the communication module, and identify the identified real IoT device from the image.

8. The electronic device according to any one of claims 1-7, wherein, The instructions are configured to cause the electronic device to: display at least one control item corresponding to the real IoT device, and send, based on a user input to the control item, a control signal instructing the real IoT device to operate.

9. The electronic device according to claim 8, wherein, The instructions are configured to cause the electronic device to: when a real IoT device is added to the image display due to a change in the viewpoint of the camera, display a control item corresponding to the added real IoT device, and when a real IoT device displayed within the image is excluded from the image due to a change in the viewpoint of the camera, remove the control item corresponding to the excluded real IoT device.

10. The electronic device according to claim 9, wherein, The instructions are configured to cause the electronic device to: identify, through analysis of the image, at least one real object included in the image, and determine, based on at least one of dimensions, distribution, or density of the identified real object, at least one IoT device that can be disposed in the at least one space.

11. A method for providing a virtual IoT environment for an electronic device, the method comprising: Activate the camera to display an image obtained in real time through the camera; Analyze the image to determine at least one space within the image; Identify at least one Internet of Things (IoT) device that can be set in the at least one space, and Display at least one virtual IoT device corresponding to the at least one identified IoT device on at least one determined space of the image.

12. The method according to claim 11, further comprising: Displaying a virtual operating state corresponding to the operating state of the IoT device during operation within the image.

13. The method according to claim 12, further comprising: Displaying at least one control item corresponding to the virtual IoT device, wherein the display of the virtual operating state includes displaying the virtual operating state within the image based on a user input to the control item.

14. The method according to any one of claims 11-13, further comprising: Displaying available items corresponding to the virtual IoT device, and Providing a purchase screen for purchasing an IoT device corresponding to the virtual IoT device based on a user input to the available items.

15. The method according to any one of claims 11-14, further comprising: Identifying at least one adjacent real IoT device, and Identifying the identified real IoT device from the image.

Images