Electronic device for automatically arranging external devices in space, operation method therefor, and storage medium

Abstract

According to an embodiment, an electronic device may comprise a camera, a display, at least one processor, and a memory for storing instructions. According to an embodiment, the instructions may be configured to, when individually or collectively executed by the at least one processor, cause the electronic device to: identify a plurality of candidate positions for an external device in a plurality of image frames obtained by photographing a space by using the camera; on the basis of a photographing time of the external device and a photographing posture of the electronic device, calculate reliabilities for the plurality of candidate positions, respectively; determine, as a position of the external device, a candidate position having the highest reliability among the reliabilities; and display, through the display, an object corresponding to the external device at the determined position within an indoor map view representing the space.

Description

Electronic device for automatically arranging external devices within a space, its operating method and storage medium

[0001] One embodiment disclosed in this document relates to an electronic device for automatically arranging external devices within a space, a method of operating the same, and a storage medium.

[0002] The Internet of Things (IoT), which integrates communication capabilities into objects and connects them to a network, is widely used in everyday life. This IoT, through the convergence and integration of existing IT technologies with various industries, can be applied to diverse fields such as smart homes, smart buildings, smart cities, smart or connected cars, smart grids, healthcare, smart appliances, and advanced medical services.

[0003] Among these, smart home environments can leverage various IoT devices to provide a variety of Internet of Things-based services. Electronic devices like smartphones can run applications to manage each IoT device.

[0004] IoT-based services (e.g., Samsung) that control and / or manage external devices (e.g., IoT devices) TM In the case of SmartThings service, various services can be provided using various IoT devices. One of the various services provided by IoT-based services is spatial information service (e.g. Samsung TM The map view service provided by the SmartThings service is a server of an IoT-based service (e.g. Samsung TM It may be a service that provides information on external devices registered in the SmartThings server (e.g., home space and / or office space) located in a set space in the form of a map.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0006] According to one embodiment, an electronic device may include a camera, a display, at least one processor, and a memory storing instructions.

[0007] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may be configured to cause the electronic device to identify a plurality of candidate locations for an external device from a plurality of image frames captured in space using the camera.

[0008] According to one embodiment, the instructions may be configured to cause the electronic device to calculate reliabilities for each of the plurality of candidate locations based on a shooting time of the external device and a shooting pose of the electronic device.

[0009] In one embodiment, the instructions may be configured to cause the electronic device to determine a candidate location having the highest reliability among the reliabilities as the location of the external device.

[0010] According to one embodiment, the instructions may be configured to cause the electronic device to display, through the display, an object corresponding to the external device at the determined location within an indoor map view representing the space.

[0011] According to one embodiment, a method for automatically arranging external devices within a space in an electronic device may include an operation of identifying a plurality of candidate locations for the external devices from a plurality of image frames captured within a space using a camera.

[0012] According to one embodiment, the method may include an operation of calculating reliabilities for each of the plurality of candidate locations based on a shooting time of the external device and a shooting pose of the electronic device.

[0013] According to one embodiment, the method may include an operation of determining a candidate location having the highest reliability among the reliabilities as the location of the external device.

[0014] According to one embodiment, the method may include an action of displaying an object corresponding to the external device at the determined location within an indoor map view representing the space.

[0015] In a non-transitory storage medium storing instructions, the instructions, when individually or collectively executed by at least one processor of an electronic device, are configured to cause the electronic device to perform at least one operation, wherein the at least one operation may include an operation of identifying a plurality of candidate locations for an external device from a plurality of image frames captured in space using a camera.

[0016] According to one embodiment, the at least one operation may include an operation of calculating reliabilities for each of the plurality of candidate locations based on a shooting time of the external device and a shooting pose of the electronic device.

[0017] In one embodiment, the at least one operation may include determining a candidate location having the highest reliability among the reliabilities as the location of the external device.

[0018] In one embodiment, the at least one action may include displaying an object corresponding to the external device at the determined location within an indoor map view representing the space.

[0019] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0020] FIG. 2 is a diagram illustrating a process for providing an IoT-based service in a wireless communication network according to one embodiment.

[0021] Figure 3 is an internal block diagram of an electronic device according to one embodiment.

[0022] FIG. 4 is an example screen diagram for managing a selected space among the divided spaces of an indoor environment according to one embodiment.

[0023] FIG. 5 is an example screen diagram illustrating a method of calling external devices related to a selected space according to one embodiment.

[0024] FIG. 6 is a flowchart illustrating the operation of an electronic device for automatically arranging external devices within a space according to one embodiment.

[0025] FIG. 7 is a drawing illustrating a shooting method for acquiring the positions of external devices within a selected space according to one embodiment.

[0026] FIG. 8 is a drawing for explaining a method for identifying and estimating the location of an external device using a photographed object image according to one embodiment.

[0027] FIG. 9 is a diagram illustrating data used to identify and estimate the location of an external device using a photographed object image according to one embodiment.

[0028] FIG. 10 is a drawing showing the result of an external device being placed at an identified location in space during shooting according to one embodiment.

[0029] Figure 11 is an example screen diagram showing the automatic placement of external devices within a selected space according to one embodiment.

[0030] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0031] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0032] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0033] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0034] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0035] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0036] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0037] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0038] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0039] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0040] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0041] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0042] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0044] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0046] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0047] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0048] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0049] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0050] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0051] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] In the detailed description below, reference numerals in the drawings may be used interchangeably or omitted for components that can be easily understood through the preceding embodiments, and their detailed descriptions may also be omitted. The electronic device (101) according to one embodiment disclosed in this document may be implemented by selectively combining components of different embodiments, and components of one embodiment may be replaced by components of another embodiment. For example, it should be noted that the present invention is not limited to specific drawings or embodiments.

[0054] FIG. 2 is a diagram for explaining a process for providing an IoT-based service in a wireless communication network (200) according to one embodiment.

[0055] Referring to FIG. 2, the wireless communication network (200) may be an IoT network (e.g., a smart home network).

[0056] According to one embodiment, an electronic device (101) (e.g., electronic device (101) of FIG. 1) controls and / or manages external device(s) (or external electronic device(s)) (220) (e.g., IoT devices) (e.g., electronic device (102) or electronic device (104) of FIG. 1) based on IoT technology, an IoT-based service (e.g., Samsung TM It can provide SmartThings service.

[0057] According to one embodiment, the hub (210) may be located in a set space (e.g., a home space and / or an office space).

[0058] According to one embodiment, the server (108) can register external device(s) (220) that use IoT-based services and remotely control the registered external device(s) (220). The server (108) can remotely control the external device(s) (220) by linking with the electronic device (101).

[0059] According to one embodiment, the external device(s) (220) may be a device that provides an IoT-based service by interfacing with the electronic device (101), the hub (210), and / or the server (108). According to one embodiment, the external device(s) (220) may be an IoT device (e.g., a home appliance such as a TV, a light, an air conditioner, a blind, an air purifier, a camera, an oven, and / or a speaker). According to one embodiment, the external device(s) (220) may be located in a set space. In one embodiment, the wireless communication network (200) may further include other external devices (e.g., an application market server).

[0060] In one embodiment, the server (108) can receive data from external device(s) (220). In one embodiment, the server (108) can communicate with the electronic device (101), the hub (210), and / or the external device(s) (220) via the Internet. The server (108) can communicate with other external devices registered with other servers (not shown) that provide IoT-based services.

[0061] According to one embodiment, the hub (210) can communicate with external device(s) (220) based on any one of various communication methods. The various communication methods can include Zigbee and / or Z-Wave. According to one embodiment, the hub (210) can communicate with external device(s) (220) via a local area network (LAN).

[0062] According to one embodiment, the external device(s) (220) may be located in various spaces of an indoor environment within a wireless communication network (200) (e.g., a living room, a family room, a kitchen, an entrance, a laundry storage, and a first to Nth office, a kitchen, and an entrance in a home space, and an office space) and may communicate with the electronic device (101) via a server (108).

[0063] According to one embodiment, the external device(s) (220) may include various types of electronic devices that operate based on IoT-based services. In one embodiment, the external device(s) (220) may include sensors or switches for detecting the surrounding environment. The external device(s) (220) may operate based on user input or automatically when a set condition is satisfied.

[0064] According to one embodiment, the electronic device (101) can receive data from external device(s) (220) and display a screen corresponding to the received data. According to one embodiment, the electronic device (101) can control external device(s) (220) located in various spaces within a wireless communication network (200) using an IoT-based application (e.g., Samsung TM When the SmartThings application is executed, it can output a user interface (UI) related to the IoT-based application being executed.

[0065] In the case of IoT-based services that control and / or manage external devices (220), various services can be provided using various IoT devices. One of the various services provided by IoT-based services is spatial information service (e.g., Samsung TMThe map view service provided by the SmartThings service is a server (e.g. server (108)) of an IoT-based service (e.g. Samsung TM It may be a service that provides information on external devices registered in the SmartThings server (e.g., a home space and / or office space) located in a set space in the form of a map.

[0066] In the case of a spatial information service, an application can be executed to control external devices (220) existing in various spaces. However, when the application is executed for the first time, the user must manually set up a group of external devices (220) or set up the arrangement of objects within the user interface (UI). For example, the user can control the external devices belonging to the designated groups by dividing multiple external devices (220) located within the home into groups by space, such as the bedroom, living room, and kitchen, according to the user's selection. In addition, when registering a new external device (220) or moving it, a new space must be created and added on the UI according to the user's selection, or the arrangement of objects must be directly moved within the UI. Therefore, there may be the inconvenience of having to repeatedly perform the actions of selecting and arranging objects one by one.

[0067] Therefore, if the location of external devices within a space can be automatically identified and controlled without the user having to manually change it on the UI when placing external devices, convenience of use can be increased.

[0068] Accordingly, in one embodiment of the present disclosure, an electronic device, an operating method thereof, and a storage medium for photographing a space and automatically arranging external devices related to the space on an indoor map view corresponding to the space can be provided.

[0069] According to one embodiment, when arranging external devices registered in a space, the user can increase convenience by automatically identifying the locations of external devices within the space and arranging them without the user having to manually arrange them on the UI.

[0070] According to one embodiment, when the physical location or position of external devices registered for a space changes, the arrangement of the external devices can be changed to correspond to the changed location by taking a picture of the space without the user having to manually change it on the UI.

[0071] According to one embodiment, by taking into account the characteristics of device placement by space, the placement positions of external devices can be corrected when placing external devices within a space, thereby providing a spatial information service with improved accuracy during automatic placement.

[0072] In the following description, it should be noted that “IoT-based service” and “SmartThings service” may be used interchangeably, “spatial information service” and “map view service” may be used interchangeably, and “spatial information” may be used interchangeably with “screen displaying spatial information.”

[0073] FIG. 3 is an internal block diagram of an electronic device according to an embodiment. To facilitate understanding of the description of FIG. 3, reference will be made to FIG. 4 and FIG. 5 . FIG. 4 is an example screen diagram for managing a selected space among the divided spaces of an indoor environment according to an embodiment, and FIG. 5 is an example screen diagram for explaining a method for calling external devices related to a selected space according to an embodiment.

[0074] Referring to FIG. 3, the electronic device (101) (e.g., the electronic device (101) of FIGS. 1 and 2) may include a camera (380) (e.g., the camera module (180) of FIG. 1), a display (360) (e.g., the display module (160) of FIG. 1), at least one processor (320) (e.g., the processor (120) of FIG. 1) and a memory (330) (e.g., the memory (130) of FIG. 1). According to one embodiment, the electronic device (101) may further include at least one sensor (376) (e.g., the sensor module (176) of FIG. 1) and a communication circuit (390) (e.g., the communication module (190) of FIG. 1).

[0075] According to one embodiment, the memory (330) is an application for executing an IoT-based service (e.g., Samsung TM It can store images, user information, documents, databases or related data for providing UI and UI related to SmartThings service application.

[0076] According to one embodiment, the communication circuit (390) can perform communication with external devices (220) under the control of the processor (320). According to one embodiment, the communication circuit (390) can perform communication using at least one communication method among communication methods including Zigbee, Z-Wave, Wi-Fi, Bluetooth, Ultra-Wide Band (UWB), Wireless USB, and Near Field Communication (NFC). For example, the communication circuit (390) can perform low-power wireless communication using Zigbee or Z-Wave communication to communicate with the external devices (220). According to one embodiment, the communication circuit (390) can support short-range communication such as Bluetooth and NFC (near field communication) in addition to Zigbee or Z-Wave communication.

[0077] According to one embodiment, the processor (320) can manage external devices (e.g., IoT devices) (220) that have been onboarded using an IoT-based application. For example, the processor (320) can perform management and control functions for onboarded external devices (220) that exist in various indoor spaces.

[0078] According to one embodiment, when an application for executing an IoT-based service is executed under the control of the processor (320), the display (360) may display a UI related to the executed application. According to one embodiment, the display (360) may display a map (or map view) that visually represents the locations of external devices (220) within a space. For example, the map may be a floor plan diagram within an indoor space (or a building), and each external device (220) corresponding to each space may be displayed on the map using objects such as icons. Here, the objects representing the external devices (220) may be replaced with expressions such as a visual object, an icon, an indicator, content, an image, a widget, and an affordance, and there is no limitation on their types.

[0079] The map displayed through the display (360) represents at least one space among multiple spaces within the building, and may represent a two-dimensional map and / or a three-dimensional map. The two-dimensional map may be a floor plan (or layout), and the three-dimensional map may be a map for representing virtual reality (VR), augmented reality (AR), or mixed reality (MR).

[0080] According to one embodiment, when an IoT-based application is executed, in order to automatically arrange external devices (220) within a space, the processor (320) may display a UI on the display (360) for receiving a user input for selecting one of a plurality of spaces. For example, when the 'living room' space is selected as illustrated in FIG. 4, the processor (320) may call up (or obtain) map data corresponding to the 'living room' space. The map data may include a map image representing a floor plan and / or indoor map information. Here, the indoor map information may include partition information for dividing each of the plurality of spaces (e.g., partition information by walls) and coordinate information for each of the spaces, and may include information (e.g., identification information, product information) of an external device located in each of the spaces. Map data may be stored in memory within a server (108) (e.g., server (108) of FIG. 2), and the processor (320) may obtain (or download) the map data by connecting to the server (108) when executing the application.

[0081] According to one embodiment, the processor (320) may obtain information (e.g., identification information, product information) of an external device located in a 'living room' space based on map data. In addition, the processor (320) may display a map (405) corresponding to the 'living room' space on the display (360) as illustrated in FIG. 4 based on the map data. For example, referring to FIG. 4, 'total devices' (410) represents external devices registered for the 'living room' space, and exemplifies a case where '5' external devices are registered for the 'living room' space. Here, 'positioned devices' (415) represents external devices that have been placed for the 'living room' space, and '0' may indicate that there are no placed external devices.

[0082] According to one embodiment, the processor (320) may provide a function (or item, menu) for automatically arranging external devices registered in a space, as illustrated in FIG. 5. As illustrated in 500a, when the auto-positioning function (505) is selected, the processor (320) may provide (or display) detailed functions (510) for the auto-positioning, as illustrated in 500b. For example, the detailed functions (510) may include a function for arranging all external devices (515) and a function for arranging some of the external devices (520). The function for arranging all external devices (515) may be a function for providing a list of all external devices registered for the 'living room' space, and the function for arranging some of the external devices (520) may be a function for providing a list of external devices not arranged in a map view, for example, the 'living room' space. For example, when selecting the full placement (515) function, a full list (525) of external devices registered for the 'living room' space may be displayed on the map view as illustrated in 500c. The next operation (530) for automatically placing external devices registered for the 'living room' space may be a shooting operation for the actual space.

[0083] For example, in response to a user selection to automatically place an external device registered for the 'living room' space, the processor (320) may execute the camera (380).

[0084] According to one embodiment, the processor (320) may display an image captured by the camera (380) in an actual space on the display (360). The user may rotate at a position within the set space and capture at least one area (e.g., a wall) within the space in all 360 directions using the camera (380) of the electronic device (101). The set space and the at least one area may be selected based on a user input. For example, the user may rotate at the same position and capture multiple images toward a space partitioned by a wall, i.e., at least one area (e.g., a wall) within the space, using the camera (380).

[0085] According to one embodiment, the processor (320) may acquire a plurality of image frames by photographing a space using a camera (380). The processor (320) may detect (or identify) at least one external device by analyzing the plurality of image frames (or still images and / or moving images) photographing the space.

[0086] According to one embodiment, the processor (320) may perform an operation of detecting (or identifying) an object corresponding to an external device from each captured image frame by referring to a list of external devices registered for a space. The processor (320) may analyze a plurality of image frames to confirm that an external device exists at a specific location within the space. For example, the processor (320) may compare an image of the captured space with a representative image of the registered external device to confirm that an external device exists at a specific location within the space. Here, when referring to the list of registered external devices, the processor (320) may be able to not only detect an external device from the captured image, but also identify the type of the detected external device.

[0087] According to one embodiment, the processor (320) may detect (or identify) at least one external device and recognize (or identify) the type of the detected external device by analyzing a plurality of image frames using an artificial intelligence (AI) model (e.g., a machine learning model or a generative AI model). In this way, the method for detecting and tracking an object corresponding to an external device within a plurality of image frames that capture a space may include, in addition to a method of referencing a list of external devices and a method of using an AI model, a method of confirming that an external device exists in a selected area based on a user confirmation signal (e.g., touch input) for a specific area within the captured image, and the method is not limited thereto.

[0088] According to one embodiment, the processor (320) may identify a plurality of candidate locations for the external device in response to detecting (or identifying) the external device from a plurality of image frames that capture a space. According to one embodiment, the plurality of candidate locations for the detected (or identified) external device may be determined using sensor data acquired from at least one sensor (376) of the electronic device (101) at the time of capture. For example, since the external devices exist inside a space, they may be positioned based on unique orientation information (e.g., azimuth) based on an absolute coordinate system. Therefore, if orientation information is acquired when capturing an external device inside a space, a candidate location for placing the external device may be identified based on the acquired orientation information. Here, there may be a plurality of candidate locations based on the acquired orientation information, and in one embodiment, the sensor data (or sensor information) of the electronic device (101) may be used to determine a candidate location corresponding to an actual location of the external device among the plurality of candidate locations.

[0089] According to one embodiment, at least one sensor (376) may include a gyro sensor or an orientation sensor for sensing the attitude of the electronic device (101), and examples of sensors included in the sensor (376) may not be limited thereto.

[0090] In one embodiment, orientation information obtained through at least one sensor (376) of the electronic device (101) may be used to determine a candidate location corresponding to the location of an actual external device among a plurality of candidate locations.

[0091] According to one embodiment, the processor (320) can identify the orientation with respect to an external device. The processor (320) can identify the orientation with respect to an external device when photographing a space in which the external device is present, using at least one sensor (376). For example, a orientation sensor can be used to obtain orientation information (or azimuth angle, orientation value) when photographing an external device, and the orientation information can be stored by matching the detected external device with the photographed image frame when the external device is detected within the image frame.

[0092] In one embodiment, in order to determine a candidate position corresponding to the actual position of the external device among a plurality of candidate positions, the photographing pose information of the electronic device (101) obtained through at least one sensor (376) of the electronic device (101) may be used. For example, the electronic device (101) may include a gyro sensor that senses the rotation of the electronic device (101) around one axis among the rotation (roll) around the X-axis, the rotation (pitch) around the Y-axis, and the rotation (yaw) around the Z-axis. For example, when a user tilts the electronic device (101) left and right while holding it in his hand, the rotation (roll) value about the X-axis changes, and the change in the rotation (roll) value about the X-axis may indicate a change in the photographing pose. Therefore, if the rotation (roll) value about the X-axis falls outside a critical range, even if the user took the picture from a fixed position, it means that the picture was taken by tilting the electronic device (101) too much. Therefore, in one embodiment, photographing pose information corresponding to a rotation (roll) value that does not fall outside the critical range may be used.

[0093] In one embodiment, the time of capture may be utilized to determine a candidate location corresponding to the actual location of an external device among multiple candidate locations. For example, when an external device is detected (or identified) from multiple video frames, the time at which the external device was detected (or captured) may be stored in a match with the corresponding video frame.

[0094] In one embodiment, an offset may be used to determine a candidate location corresponding to an actual location of an external device among a plurality of candidate locations. The processor (320) may obtain an offset representing a difference in position between detection boxes surrounding the external device within the image frame from the center position of the image frame for each image frame. For example, an area in which an object corresponding to the external device exists (or is displayed) within the image frame may be identified, a detection box surrounding the object may be determined, and coordinate values ​​representing the detection box may be generated. For example, the detection box may have a rectangular shape, and the offset may be obtained using the center of the detection box (detection box center x) / frame width.

[0095] In one embodiment, to determine a candidate position corresponding to the actual position of the external device among a plurality of candidate positions, the shooting time of the external device and the shooting posture of the electronic device (101) may be used together with the obtained offset.

[0096] Here, among multiple candidate locations, the candidate location corresponding to the location of the actual external device can be determined by calculating the reliability for each candidate location.

[0097] According to one embodiment, the processor (320) may calculate reliability scores for each of a plurality of candidate locations and determine the candidate location with the highest reliability among the calculated reliability scores as the actual location of the external device. The processor (320) may complete the automatic placement operation by displaying an object corresponding to the external device at the determined location within an indoor map view representing the space.

[0098] According to one embodiment, in order to improve the accuracy of placement of external devices on an indoor map view, the processor (320) may use pre-stored space-specific device placement characteristic information. The processor (320) may correct the determined position using the pre-stored space-specific device placement characteristic information, and then update the placement of the external devices by displaying an object corresponding to the external devices at the corrected position within the indoor map view. In one embodiment, the placement update process (or operation) for the external devices may be automatically triggered upon entering a spatial information service (e.g., a map view service), automatically triggered based on a set cycle, or triggered based on a user input via a specific menu (e.g., an automatic placement menu) on the spatial information service.

[0099] According to one embodiment, the device arrangement characteristics for each space may be stored in advance and may be set to reflect user characteristics. For example, in the case of an external device such as a TV, considering the characteristic of the TV being arranged at the center of the wall when placed in a living room, the electronic device (101) may compare the position corresponding to the azimuth of the TV with the center of the corresponding wall, and based on the comparison result, correct the placement position of the TV to correspond to the center of the wall. In addition, in the case of a TV, considering the characteristic of being placed opposite a sofa and a bed, the electronic device (101) may correct the placement position of the TV to be placed opposite the furniture based on furniture information (e.g., furniture arrangement position) about the sofa and the bed.

[0100] For example, if the external device is an air conditioner, considering its characteristic of being mainly placed in a corner within a space, the electronic device (101) can correct the placement position of the air conditioner to correspond to the space's partition information (e.g., corner, vertex). In addition, considering the air conditioner piping, the location can be corrected to be located near a window or an exterior wall.

[0101] For example, if the external device is an air purifier, the electronic device (101) can correct the placement position of the air purifier by taking into account the characteristics of being placed next to a TV, a sofa, or at both ends of a wall.

[0102] For example, if the external device is a washing machine, considering the characteristic of being placed primarily on a short wall in a balcony space, the electronic device (101) can be adjusted to place the washing machine on the shorter of the two adjacent walls. Additionally, if the external device is placed in a position overlapping a door, the electronic device (101) can be adjusted to place it on the left or right of the door.

[0103] As described above, the device placement characteristics for each space may be stored in advance and may be set to reflect user characteristics, but may not be limited thereto.

[0104] Meanwhile, the above-described process of detecting external devices and determining and placing one of the candidate locations as the actual location, assuming that different types of external devices exist within a space, may be described. However, external devices of the same type may also exist within the space. In this case, the processor (320) may operate as follows to detect multiple external devices of the same type, and to determine and place the actual location among the candidate locations for each of the detected external devices.

[0105] According to one embodiment, the processor (320) can confirm that there are multiple identical external devices by referring to a list of registered external devices. If there are multiple identical external devices, the processor (320) can calculate reliabilities for multiple candidate locations for each of the external devices, and determine the candidate location with the highest reliability and the candidate location with the next highest reliability among the calculated reliabilities as the locations of each of the identical external devices.

[0106] According to one embodiment, the processor (320) may determine one candidate location for each of a plurality of candidate locations for each of the same type of external devices by using the orientation information of each external device at the time of shooting. For example, when a candidate location with the highest reliability is determined for a first external device, the processor (320) may determine a candidate location with the next highest reliability among the candidate locations for a second external device of the same type as the first external device as the candidate location for the second external device. At this time, in order to prevent a candidate location that is too close to the first external device from being determined (or selected) as a candidate location for the second external device, candidate locations within a first threshold range (e.g., a threshold orientation angle) may be excluded based on the candidate location with the highest reliability for the first external device. Here, orientation information of the first external device at the time of shooting may be used so that candidate locations within the first threshold range may be excluded. Accordingly, the processor (320) can determine the candidate position with the next highest reliability among the remaining candidate positions, excluding the candidate position corresponding to the first threshold range based on the orientation information at the time of shooting of the first external device, as the actual position for the second external device.

[0107] Meanwhile, in the above, a process for determining the actual location of each of the same type of external devices when multiple of the same type of external devices are detected in each of different video frames has been described, but the same type of external devices may also be detected in one video frame.

[0108] According to one embodiment, when external devices of the same type are identified in at least one of a plurality of image frames, the processor (320) may reduce the first threshold range to a second threshold range smaller than the first threshold range. For example, assuming that two external devices of the same type are detected (or identified) in one image frame, the two external devices may be positioned adjacently. Therefore, considering the situation in which the first external device and the second external device are positioned close to each other, the candidate location with the highest reliability among the plurality of candidate locations for the first and second external devices may be determined as the location for the first external device, and then candidate locations within the second threshold range may be excluded based on the candidate location with the highest reliability among the plurality of candidate locations. Therefore, based on the orientation information when capturing the first external device, the processor (320) may determine the candidate location with the next highest reliability among the remaining candidate locations as the actual location for the second external device, excluding the corresponding candidate location within the second threshold range based on the candidate location with the highest reliability.

[0109] According to one embodiment, the electronic device (101) may include a camera (180, 380), a display (160, 360), at least one processor (120, 320), and a memory (130, 330) for storing instructions.

[0110] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may be configured to cause the electronic device to identify a plurality of candidate locations for an external device from a plurality of image frames captured in space using the camera.

[0111] According to one embodiment, the instructions may be configured to cause the electronic device to calculate reliabilities for each of the plurality of candidate locations based on a shooting time of the external device and a shooting pose of the electronic device.

[0112] In one embodiment, the instructions may be configured to cause the electronic device to determine a candidate location having the highest reliability among the reliabilities as the location of the external device.

[0113] According to one embodiment, the instructions may be configured to cause the electronic device to display, through the display, an object corresponding to the external device at the determined location within an indoor map view representing the space.

[0114] According to one embodiment, the instructions may be configured to cause the electronic device to obtain orientation information of the external device at the time of shooting in response to identification of the external device from the plurality of image frames, and to identify a plurality of candidate locations for the external device based on the orientation information of the external device at the time of shooting.

[0115] According to one embodiment, the instructions may be configured to cause the electronic device to identify the external device from the plurality of image frames based on a list of external devices registered for the space.

[0116] According to one embodiment, the instructions may be configured to cause the electronic device to obtain an offset representing a difference in position between detection boxes surrounding the external device within the image frame from a center position of the image frame for each of the image frames, and to calculate reliabilities for each of the plurality of candidate positions based on the photographing time of the external device and the photographing pose of the electronic device together with the obtained offset.

[0117] According to one embodiment, the instructions may be configured to cause the electronic device to determine, based on a list of external devices registered for the space, if there are multiple external devices of the same type, a candidate location having the highest reliability among the reliabilities and a candidate location having the next highest reliability as the locations of each of the external devices of the same type.

[0118] According to one embodiment, the instructions may be configured such that the electronic device determines, as the locations of each of the same type of external devices, the candidate location having the highest reliability and the candidate location having the next highest reliability among the remaining candidate locations after the exclusion, excluding the candidate location corresponding to a first threshold orientation based on the orientation information of the external device at the time of photographing among the plurality of candidate locations for the external device.

[0119] According to one embodiment, the instructions may be configured to cause the electronic device to, when external devices of the same type are identified within at least one image frame among the plurality of image frames, reduce a first threshold direction to a second threshold direction, and, based on orientation information of the external device at the time of photographing among a plurality of candidate locations for the external device, exclude a candidate location corresponding to the external device within the second threshold direction based on the candidate location having the highest reliability, and determine the candidate location having the highest reliability and a candidate location having a next highest reliability among the candidate locations remaining after the exclusion as the locations of each of the external devices of the same type.

[0120] According to one embodiment, the plurality of video frames may be captured from 360 degrees around the space according to the rotation of the electronic device at the same location within the space using the camera.

[0121] According to one embodiment, the instructions may be configured to cause the electronic device to correct the determined position based on pre-stored space-specific device arrangement characteristic information, and to display an object corresponding to the external device at the corrected position within an indoor map view representing the space through the display.

[0122] FIG. 6 is a flowchart illustrating an operation of an electronic device for automatically arranging external devices within a space according to an embodiment. Referring to FIG. 6, the operation method may include operations 605 to 620. Each operation of the operation method of FIG. 6 may be performed by an electronic device (e.g., the electronic device (101) of FIGS. 1 to 3 ) and at least one processor of the electronic device (e.g., the processor (120) of FIG. 1 , the processor (320) of FIG. 3 ). In an embodiment, at least one of operations 605 to 620 may be omitted, the order of some operations may be changed, or another operation may be added.

[0123] Referring to FIG. 6, in operation 605, the electronic device (101) can identify multiple candidate locations for an external device from multiple image frames captured by using a camera (180, 380) in the space. Here, the multiple image frames may be captured from 360 degrees in all directions according to the rotation of the electronic device (101) at the same location in the space using the camera.

[0124] According to one embodiment, the electronic device (101) can obtain orientation information of the external device at the time of shooting in response to the identification of the external device from the plurality of image frames, and identify a plurality of candidate locations for the external device based on the orientation information of the external device at the time of shooting.

[0125] According to one embodiment, the electronic device (101) can identify the external device from the plurality of image frames based on a list of external devices registered for the space.

[0126] In operation 610, the electronic device (101) may calculate reliabilities for each of the plurality of candidate locations based on the shooting time of the external device and the shooting posture of the electronic device. For example, the electronic device (101) may obtain shooting posture information using a gyro sensor that senses the rotation of the electronic device (101), and the sensor value indicating the shooting posture may include a rotation (roll) value about the X-axis.

[0127] According to one embodiment, the electronic device (101) can obtain an offset representing a positional difference between detection boxes surrounding the external device within the image frame from the center position of the image frame for each of the image frames.

[0128] According to one embodiment, the electronic device (101) may calculate reliabilities for each of the plurality of candidate locations based on the acquired offset, the shooting time of the external device, and the shooting pose of the electronic device (101). For example, the electronic device (101) may acquire the shooting pose of the electronic device (101) using at least one sensor (376).

[0129] In operation 615, the electronic device (101) can determine the candidate location having the highest reliability among the reliabilities as the location of the external device.

[0130] In operation 620, the electronic device (101) can display an object corresponding to the external device at the determined location within the indoor map view representing the space.

[0131] According to one embodiment, the electronic device (101) may, based on a list of external devices registered for the space, determine a candidate location having the highest reliability and a candidate location having the next highest reliability as the locations of each of the external devices of the same type, when there are multiple external devices of the same type.

[0132] According to one embodiment, the electronic device (101) may exclude candidate locations corresponding to within a first threshold orientation based on the orientation information of the external device at the time of photographing among a plurality of candidate locations for the external device, based on the candidate location with the highest reliability. The electronic device (101) may determine the candidate location with the highest reliability and the candidate location with the next highest reliability among the candidate locations remaining after the exclusion as the locations of each of the external devices of the same type.

[0133] According to one embodiment, the electronic device (101) may reduce the first threshold direction to a second threshold direction when external devices of the same type are identified within at least one of the plurality of image frames. The electronic device (101) may exclude candidate locations corresponding to within the second threshold direction based on the candidate location with the highest reliability based on the orientation information of the external device at the time of photographing among the plurality of candidate locations for the external device. The electronic device (101) may determine the candidate location with the highest reliability and the candidate location with the next highest reliability among the candidate locations remaining after the exclusion as the locations of each of the external devices of the same type.

[0134] According to one embodiment, the electronic device (101) can correct the determined location based on pre-stored space-specific device arrangement characteristic information. The electronic device (101) can display an object corresponding to the external device at the corrected location within an indoor map view representing the space.

[0135] FIG. 7 is a diagram illustrating a photographing method for acquiring the positions of external devices within a selected space according to one embodiment. To facilitate understanding of the description of FIG. 7, reference will be made to FIG. 8. FIG. 8 is a diagram illustrating a method for identifying and estimating the positions of external devices using photographed object images according to one embodiment.

[0136] Referring to FIG. 7, a user can use a camera (380) of an electronic device (101) to photograph at least one area (e.g., a wall) within a space (705) from all 360 degrees while rotating at a position within the set space (705). For example, a user can use a camera (380) to photograph a space partitioned by a wall, i.e., at least one area (e.g., a wall) within the space, multiple times while rotating at the same position.

[0137] For example, when a user takes a still image and / or video toward at least one area (e.g., a wall) of a space where an external device (e.g., a TV) is present, the electronic device (101) can obtain a first image frame (700) corresponding to a front view. When the user takes a picture of the space while rotating in the A direction from the same position during the shooting, the electronic device (101) can obtain a second image frame (700a) corresponding to a view corresponding to the A direction, and when the user takes a picture of the space while rotating in the B direction, the electronic device (101) can obtain a third image frame (700b) corresponding to a view corresponding to the B direction.

[0138] According to one embodiment, in response to a user input for automatically arranging external devices within a space, the electronic device (101) may execute a camera (380) to display a captured image as illustrated in FIG. 8 through a display (360). Referring to FIG. 8, the captured image (800) displays a detection box (810) surrounding the external device, and the user's location within the set space (705) may be displayed using an object (825) within the section information (815) corresponding to the space (705) of FIG. 7. At this time, a candidate location after identification of the external device within the detection box (810) may also be displayed using an object (820). In addition, a spatial information service screen (830) may be displayed on at least a portion of the captured image screen so that the user can know that the spatial information service is running when capturing.

[0139] FIG. 9 is a diagram illustrating data used to identify and estimate the location of an external device using a captured object image according to one embodiment. To facilitate understanding of the description of FIG. 9, reference will be made to FIG. 10. FIG. 10 is a diagram illustrating the results of placing an external device at an identified location within space during capture according to one embodiment.

[0140] Referring to FIG. 9, data used to estimate the position of an external device may include at least one of identification information of the external device, direction information at the time of shooting of the external device, shooting time of the external device, shooting posture of the electronic device (101), or offset.

[0141] The identification information of the external device (e.g., "name": "tv") can be obtained by referring to the list of external devices registered for the space. For example, if the full placement function (515) is selected for the 'living room' space as in FIG. 5, the electronic device (101) can obtain the identification information of the external device based on the list of five external devices (e.g., one TV, two air purifiers, one refrigerator, and one air conditioner) that are registered but not placed for the 'living room' space.

[0142] The azimuth information (e.g., "azimut": ~) when shooting an external device may be the azimuth of the external device when shooting a space in which the external device is located. For example, an azimuth sensor may be used to obtain azimuth information (or azimuth angle, azimuth value) when shooting an external device, and the azimuth information may be stored by matching the detected external device with the captured video frame when the external device is detected within the video frame.

[0143] The capture time of the external device (e.g., "time": ~) may be the time at which the external device is detected (or captured) for each video frame when the external device is detected (or identified) from multiple video frames. The capture time of the external device may be stored in a manner matching the corresponding video frame.

[0144] The shooting posture (e.g., “roll”: ~) of the electronic device (101) may indicate a change in the rotation (roll) value about the X-axis. For example, a gyro sensor may be used to obtain the shooting posture of the electronic device (101). For example, when the user tilts the electronic device (101) left and right while holding it in his hand, the rotation (roll) value about the X-axis changes, and the change in the rotation (roll) value about the X-axis may indicate a change in the shooting posture. Therefore, if the rotation (roll) value about the X-axis goes out of the critical range, even if the user took the photo from a fixed position, it means that the photo was taken by tilting the electronic device (101) too much. Therefore, in one embodiment, shooting posture information corresponding to a rotation (roll) value that does not go out of the critical range may be used. For example, the rotation (roll) value is based on 0, and if it is less than the first threshold value (e.g., -10) or greater than the second threshold value (e.g., 10), it may be considered an abnormal posture and may not be used as data for estimating the position of the external device. Here, the threshold values ​​are explained as -10 and 10 for convenience of explanation, but these are only examples and the values ​​are not limited to these.

[0145] The offset may indicate the difference in position between detection boxes surrounding the external device within the image frame from the center position of the image frame for each image frame. For example, the offset may be a value indicating where the detection box (e.g., the detection box (810) of FIG. 8) is located with respect to the center of the captured image. Assuming that the center of the detection box is '0.5' when the center of the detection box coincides with the center of the captured image, if the offset is greater than 0 and less than the first offset value (e.g., 0.5), the detection box for the external device may be located to the left of the captured image, and if the offset is greater than the first offset value and less than the second offset value (e.g., 1), the detection box for the external device may be located to the right of the captured image. Here, the first offset value and the second offset value are described as 0.5 and 1 as examples for convenience of explanation, but this is merely an example and the values ​​are not limited thereto.

[0146] Figure 9 illustrates a case where the roll value is excluded for each video frame if it is less than a first threshold value (e.g., -10) or greater than a second threshold value (e.g., 10).

[0147] As illustrated in FIG. 9, the electronic device (101) can obtain orientation information of the external device (e.g., TV) when identifying the external device at the time of photographing the external device. Based on the orientation information of the external device at the time of photographing the external device, the electronic device (101) can identify multiple candidate locations for the external device. The electronic device (101) can estimate that the external device is present at locations corresponding to, for example, azimuth angles of 60 degrees, 61 degrees, 62 degrees, and 100 degrees, and can identify locations corresponding to each azimuth angle as candidate locations.

[0148] According to one embodiment, the electronic device (101) may determine a candidate location corresponding to an actual location of an external device among a plurality of candidate locations by calculating a reliability for each candidate location. The reliability for each candidate location may be obtained based on the following mathematical formula (or placement algorithm). For example, the reliability for each candidate location may be determined using the shooting time of the external device (or the time at which the external device was detected (or recognized)), the shooting posture and offset of the electronic device, based on the mathematical formula of Table 1 below. Here, if the current time and the time calculated using the shooting time of the external device differ by a specified time (e.g., 100 seconds) or more, the video frame matching the shooting time of the external device may not be used when calculating the reliability.

[0149]

[0150] According to one embodiment, the reliability may be the result of (1) + (2) + (3) of the mathematical formula in Table 1. For example, FIG. 9 illustrates cases in which the azimuth angle is 60 degrees in three cases, 61 degrees in one case, 62 degrees in one case, and 100 degrees in two cases in multiple video frames. If the reliability is calculated using the data {"name": "tv", "azimut": 60, "time": "2 minutes 10 seconds", "roll": 0, "offset": 0.45} for the first video frame with an azimuth angle of 60 degrees, the result may be as shown in Table 2 below.

[0151]

[0152] For example, referring to Table 2, for the first candidate location of the external device in the first video frame (e.g., location corresponding to a 60-degree azimuth angle), the reliability can be calculated as '(1-(2 minutes - 32 seconds - 2 minutes 10 seconds) / 100) * (1-abs(0.5-0.45)*4) * (1- 0 / 20)=> 0.78 * 0.8 * 1'. For the first candidate location of the external device in the second video frame (e.g., location corresponding to a 60-degree azimuth angle), the reliability can be calculated as '(0.79 * 1 * 0.95)'. For the first candidate location of the external device in the third video frame (e.g., location corresponding to a 60-degree azimuth angle), the reliability can be calculated as '(0.8 * 0.95 * 0.8)'. Therefore, the final confidence level for the first candidate location of the external device can be '1.98' as the sum of each confidence level.

[0153] On the other hand, for the second candidate location of the external device in the fourth video frame (e.g., the location corresponding to 61 degrees azimuth), the confidence may be '0.49', and for the third candidate location of the external device in the fifth video frame (e.g., the location corresponding to 62 degrees azimuth), the confidence may be '0.54'.

[0154] Accordingly, the electronic device (101) can determine the candidate location (e.g., 60 degrees) with the highest reliability among multiple candidate locations for the external device (e.g., TV) as the location of the external device. Accordingly, as illustrated in 1000a of FIG. 10, since the location of the external device is determined in real time during shooting, as illustrated in 1000b, when the user's gaze is directed toward the wall (820) based on the user's location (or the location of the electronic device (101)) (825) within the space, an object (1010) corresponding to the external device can be placed at the determined location. For example, the electronic device (101) can display an object corresponding to the external device (e.g., TV) at the determined location (e.g., 60 degrees) within an indoor map view representing a 'living room space'.

[0155] According to one embodiment, external devices of the same type may exist within a space. The electronic device (101) can confirm that there are multiple identical external devices by referring to a list of registered external devices. When there are multiple identical external devices, the electronic device (101) can calculate reliabilities for multiple candidate locations for each of the external devices, and determine the candidate location with the highest reliability and the candidate location with the next highest reliability among the calculated reliabilities as the locations of each of the identical external devices.

[0156] According to one embodiment, the electronic device (101) may determine one candidate position for each of a plurality of candidate positions for the same type of external devices by using the orientation information of each external device at the time of shooting. For example, when a candidate position with the highest reliability is determined for a first external device, the electronic device (101) may determine a candidate position with the next highest reliability among the candidate positions for a second external device of the same type as the first external device as the candidate position for the second external device. For example, when there is one registered external device (e.g., a TV), a candidate position corresponding to a 60-degree azimuth angle may be determined as the placement position of the external device (e.g., the TV).

[0157] If there are two registered external devices (e.g., TVs), in order to prevent a candidate location that is too close to the first external device from being determined (or selected) as a candidate location for the second external device, candidate locations within a first threshold range (e.g., a threshold azimuth angle) based on the candidate location with the highest confidence for the first external device can be excluded. For example, if the candidate location with the highest confidence corresponds to a azimuth angle of 60 degrees, candidate locations within the first threshold range (e.g., 15 degrees) based on 60 degrees can be excluded. For example, all external devices within the first threshold range with an azimuth angle of, for example, 45 to 75 degrees based on 60 degrees can be regarded as the same external device.

[0158] Here, the orientation information of the first external device at the time of shooting can be used so that candidate locations within the first threshold range can be excluded. Accordingly, the electronic device (101) can, based on the orientation information of the first external device at the time of shooting, exclude the candidate location corresponding to the first threshold range based on the candidate location with the highest reliability, and determine the candidate location with the next highest reliability among the remaining candidate locations (e.g., the candidate location corresponding to a 100-degree azimuth with a reliability of '1.74' in Table 2) as the actual location for the second external device.

[0159] Meanwhile, in the above, a process for determining the actual location of each of the same type of external devices when multiple of the same type of external devices are detected in each of different video frames has been described, but the same type of external devices may also be detected in one video frame.

[0160] According to one embodiment, when external devices of the same type are identified in at least one of a plurality of image frames, the electronic device (101) may reduce the first threshold range to a second threshold range smaller than the first threshold range. For example, assuming that two external devices of the same type are detected (or identified) in one image frame, the two external devices may be disposed adjacently. Therefore, considering the situation in which the first external device and the second external device are disposed close to each other, the candidate location with the highest reliability among the plurality of candidate locations for the first and second external devices may be determined as the location for the first external device, and then candidate locations within the second threshold range may be excluded based on the candidate location with the highest reliability among the plurality of candidate locations. For example, external devices having an azimuth angle of, for example, 55 degrees to 65 degrees, which differ by the second threshold range (for example, 5 degrees) based on 60 degrees, may be regarded as the same external device. That is, external devices with an azimuth of 66 degrees or more or an azimuth of 54 degrees or less may be considered as different external devices, although of the same type.

[0161] Accordingly, the electronic device (101) can determine, based on the azimuth information at the time of shooting of the first external device, the candidate location with the next highest reliability among the remaining candidate locations, excluding the candidate location corresponding to within the second threshold range based on the candidate location with the highest reliability, as the actual location for the second external device of the same type as the first external device.

[0162] Figure 11 is an example screen diagram showing the automatic placement of external devices within a selected space according to one embodiment.

[0163] Referring to FIG. 11, when automatic arrangement of external devices within a space is completed, the electronic device (101) can display objects (1105, 1110, 1115, 1120, 1125) corresponding to the identified external devices at each determined location within the indoor map view (405) representing the 'living room space'. Here, the electronic device (101) can further display an object (1130) to indicate that the external devices have been newly arranged.

[0164] According to one embodiment, by taking into account the characteristics of device placement by space, the placement positions of external devices can be corrected when placing external devices within a space, thereby providing a spatial information service with improved accuracy during automatic placement.

[0165] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0166] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0167] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0168] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0169] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0170] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0171] According to one embodiment, in a non-transitory storage medium storing instructions, the instructions are configured to cause the electronic device (101) to perform at least one operation when individually or collectively executed by at least one processor (320), wherein the at least one operation may include: identifying a plurality of candidate locations for an external device from a plurality of image frames captured by using a camera (180, 380) to capture a space; calculating reliabilities for each of the plurality of candidate locations based on a capturing time of the external device and a capturing posture of the electronic device; determining a candidate location having the highest reliability among the reliabilities as a location of the external device; and displaying an object corresponding to the external device at the determined location within an indoor map view representing the space.

Claims

1. In an electronic device (101), Camera (180, 380); display(160, 360); At least one processor (120, 320); and A memory (130, 330) for storing instructions, wherein the instructions are individually or collectively executed by the at least one processor, Identifying multiple candidate locations for an external device from multiple video frames captured using the above camera, Based on the shooting time of the external device and the shooting posture of the electronic device, the reliability for each of the plurality of candidate locations is calculated, Determine the candidate location with the highest reliability among the above reliabilities as the location of the external device, An electronic device set to display an object corresponding to the external device at the determined location within an indoor map view representing the space through the display.

2. In the first paragraph, when the instructions are individually or collectively executed by the at least one processor, the electronic device, In response to the identification of the external device from the plurality of video frames, the orientation information of the external device at the time of shooting is acquired, An electronic device configured to identify multiple candidate locations for the external device based on azimuth information when the external device is photographed.

3. In the first or second paragraph, when the instructions are individually or collectively executed by the at least one processor, the electronic device, An electronic device configured to identify the external device from the plurality of image frames based on a list of external devices registered for the space.

4. In any one of paragraphs 1 to 3, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: For each of the above video frames, an offset representing the difference in position between detection boxes surrounding the external device within the video frame from the center position of the video frame is obtained, An electronic device configured to calculate reliabilities for each of the plurality of candidate locations based on the shooting time of the external device and the shooting posture of the electronic device, together with the obtained offset.

5. In any one of paragraphs 1 to 4, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on the list of external devices registered for the above space, if there are multiple external devices of the same type, An electronic device configured to determine the candidate location having the highest reliability among the above reliabilities and the candidate location having the next highest reliability as the locations of each of the same type of external devices.

6. In any one of paragraphs 1 to 5, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Among the multiple candidate locations for the external device, based on the orientation information at the time of shooting of the external device, excluding the candidate location corresponding to within the first threshold orientation based on the candidate location with the highest reliability, An electronic device set to determine the candidate location with the highest reliability and the candidate location with the next highest reliability among the candidate locations remaining after the exclusion as the locations of each of the external devices of the same type.

7. In the first to sixth paragraphs, when the instructions are individually or collectively executed by the at least one processor, the electronic device, If the same type of external devices are identified in at least one of the plurality of image frames, the first threshold direction is reduced to the second threshold direction, Among the multiple candidate locations for the external device, based on the orientation information at the time of shooting of the external device, excluding the candidate location corresponding to within the second threshold orientation based on the candidate location with the highest reliability, An electronic device set to determine the candidate location with the highest reliability and the candidate location with the next highest reliability among the candidate locations remaining after the exclusion as the locations of each of the external devices of the same type.

8. In the first to seventh paragraphs, the plurality of video frames, An electronic device that photographs the space from 360 degrees in all directions according to the rotation of the electronic device at the same location within the space using the camera.

9. In the first to eighth paragraphs, when the instructions are individually or collectively executed by the at least one processor, the electronic device, Based on the previously stored space-specific device placement characteristic information, the determined location is corrected, An electronic device set to display an object corresponding to the external device at the corrected location within an indoor map view representing the space through the display.

10. In a method for automatically arranging external devices within a space in an electronic device (101), An operation of identifying multiple candidate locations for an external device from multiple image frames captured in space using a camera (180, 380); An operation of calculating reliabilities for each of the plurality of candidate locations based on the shooting time of the external device and the shooting posture of the electronic device; An operation of determining a candidate location having the highest reliability among the above reliabilities as the location of the external device; and A method for automatically arranging external devices within a space, comprising an action of displaying an object corresponding to the external device at the determined location within an indoor map view representing the space.

11. In the 10th paragraph, the operation of identifying multiple candidate locations for the external device is as follows: An operation of acquiring azimuth information of the external device at the time of shooting in response to the identification of the external device from the plurality of video frames; and A method for automatically arranging external devices within a space, comprising an operation of identifying a plurality of candidate locations for the external device based on orientation information at the time of photographing the external device.

12. A method for automatically arranging external devices in a space, further comprising an operation of identifying the external device from the plurality of image frames based on a list of external devices registered for the space in the 10th or 11th clause.

13. In any one of paragraphs 10 to 12, the operation of calculating the reliability for each of the plurality of candidate locations comprises: An operation of obtaining an offset representing a difference in position between detection boxes surrounding the external device within the image frame from the center position of the image frame for each of the image frames; A method for automatically arranging external devices in a space, comprising an operation of calculating reliabilities for each of the plurality of candidate locations based on the shooting time of the external device and the shooting posture of the electronic device, together with the obtained offset.

14. In any one of paragraphs 10 to 13, Based on the list of external devices registered for the above space, if there are multiple external devices of the same type, A method for automatically arranging external devices in a space, further comprising an action of determining a candidate location having the highest reliability among the above reliabilities and a candidate location having the next highest reliability as the locations of each of the same type of external devices.

15. In a non-transitory storage medium storing instructions, the instructions are configured to cause the electronic device (101) to perform at least one operation when individually or collectively executed by at least one processor (320), wherein the at least one operation is: An operation of identifying multiple candidate locations for an external device from multiple image frames captured in space using a camera (180, 380); An operation of calculating reliabilities for each of the plurality of candidate locations based on the shooting time of the external device and the shooting posture of the electronic device; An operation of determining a candidate location having the highest reliability among the above reliabilities as the location of the external device; and A storage medium comprising an action of displaying an object corresponding to the external device at the determined location within an indoor map view representing the space.

Images