Electronic device and method of operating the same

The three-dimensional frame is generated by taking images of actual space by electronic devices and adjusting its size and position according to user input, which solves the problem of inconvenience of user interface in the prior art, realizes rapid setting and monitoring of interactive areas, and improves user experience.

CN112041803BActive Publication Date: 2024-11-08SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980029421.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-04-30
Filing Date
2019-03-15
Publication Date
2024-11-08
Estimated Expiration
2039-03-15

AI Technical Summary

Technical Problem

When the existing interactive system sets and monitors the interactive area, the user interface is inconvenient to use, and it is difficult to quickly set or change the content corresponding to the interactive area.

Method used

An electronic device is provided that includes a display, a user input interface and a processor that generates a three-dimensional box by taking an image of the actual space, and adjusts its size and position according to user input to set an interaction area in the actual space and monitors the interaction events in the area.

Benefits of technology

It improves the convenience and flexibility of users to set up interactive zones, can quickly respond to event changes in the interactive zone, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112041803B_ABST
    Figure CN112041803B_ABST
Patent Text Reader

Abstract

An electronic device and an operating method thereof are provided. The electronic device includes: a display; a user input interface; a memory configured to store one or more instructions; and a processor configured to execute the stored one or more instructions to perform a control operation, so that an image acquired by photographing an actual space to be monitored is displayed on the display, a three-dimensional (3D) frame is generated based on at least one point included in the image, a size and / or position of the 3D frame is adjusted based on a user input received through the user input interface, and an interactive area corresponding to the 3D frame is set in the actual space.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an electronic device and an operating method thereof, and more particularly to an electronic device capable of setting an interactive zone for detecting an event in an interactive system and an operating method thereof. Background Art

[0002] The interactive system may include a monitoring device capable of monitoring the interactive zone. The monitoring device may monitor the interactive zone by using a depth sensor. The depth sensor may generate a depth image indicating the depth of the corresponding area. The depth image may be generated by stereo triangulation, optical triangulation, structured light, interferometry, coded aperture, etc.

[0003] The interactive system may set an interactive zone in the actual space and monitor whether an event has occurred in the set interactive zone. Summary of the invention

[0004] Technical Solution

[0005] Provided are an electronic device for providing a user interface capable of setting an interactive area based on an image acquired by photographing a real space, and an operating method thereof.

[0006] Beneficial effects

[0007] The electronic device according to the embodiment can improve the use convenience of the user by providing a user interface through which the user can easily set the interactive zone.

[0008] The electronic device according to the embodiment can improve the use convenience of the user by providing a user interface through which the user can easily set or change the content corresponding to the interactive area. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 FIG. 1 illustrates an interactive system according to an embodiment;

[0011] Figure 2 is a block diagram of an electronic device according to an embodiment;

[0012] Figure 3 is a block diagram of an electronic device according to another embodiment;

[0013] Figure 4 is a block diagram of a display device according to an embodiment;

[0014] Figure 5 is a signaling diagram of an operating method of an interactive system according to an embodiment;

[0015] Figure 6 is a flowchart of a method for operating an electronic device according to an embodiment;

[0016] Figure 7 FIG. 1 illustrates a method for setting an interactive zone performed by an electronic device according to an embodiment;

[0017] Fig. 8A FIG. 1 illustrates a method for setting an interactive zone performed by an electronic device according to an embodiment;

[0018] Figure 8B FIG. 1 illustrates a method for setting an interactive zone performed by an electronic device according to an embodiment;

[0019] Fig.9A FIG. 1 illustrates a method for setting an interactive zone performed by an electronic device according to an embodiment;

[0020] Fig. 9B FIG. 1 illustrates a method for setting an interactive zone performed by an electronic device according to an embodiment;

[0021] Fig.10 FIG. 1 illustrates a method for setting content corresponding to an interactive area performed by an electronic device according to an embodiment;

[0022] Fig.11 FIG. 1 illustrates a method for setting content corresponding to an interactive area performed by an electronic device according to an embodiment;

[0023] Fig.12 illustrating the operation of an interactive system including an electronic device and a display device according to an embodiment;

[0024] Fig.13 The figure illustrates a method for setting tag information in an interactive area according to an embodiment;

[0025] Fig.14A FIG. 1 illustrates a method for storing a set interactive area performed by an electronic device according to an embodiment;

[0026] Fig. 14B FIG. 1 illustrates a method for storing a set interactive area performed by an electronic device according to an embodiment;

[0027] Fig.15 The diagram illustrates an operation method of an interactive system according to an embodiment;

[0028] Fig.16 The diagram illustrates an operation method of an interactive system according to an embodiment;

[0029] Fig.17 is a signaling diagram of an operating method of an interactive system according to an embodiment;

[0030] Fig.18 The diagram illustrates an operation method of an interactive system according to an embodiment;

[0031] Fig.19 The diagram illustrates an operation method of an interactive system according to an embodiment;

[0032] Fig. 20 FIG. 1 illustrates a method for setting an interactive area performed by a display device according to an embodiment;

[0033] Fig.21 illustrates the operation of the display device according to the embodiment; and

[0034] Fig. 22 is a block diagram of a display device according to an embodiment. DETAILED DESCRIPTION

[0035] Best Practices

[0036] Provided are an electronic device for providing a user interface capable of setting an interactive area based on an image acquired by photographing a real space, and an operating method thereof.

[0037] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0038] According to an embodiment of the present disclosure, an electronic device includes: a display; a user input interface; a memory configured to store one or more instructions; and at least one processor configured to execute the one or more instructions stored in the memory to: control to display a captured image of an actual space to be monitored on the display, determine a three-dimensional (3D) box based on at least one point included in the captured image, adjust at least one of a size and a position of the 3D box based on user input received through the user input interface, and set an interaction area corresponding to the 3D box in the actual space.

[0039] According to another embodiment of the present disclosure, an operating method of an electronic device includes: controlling to display a captured image of an actual space to be monitored on a display; determining a three-dimensional (3D) frame based on at least one point included in the captured image; adjusting at least one of a size and a position of the 3D frame based on user input; and setting an interactive area corresponding to the 3D frame in the actual space.

[0040] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium includes a program recorded thereon for executing an operating method of an electronic device in a computer.

[0041] According to another embodiment of the present disclosure, an electronic device includes: a memory configured to store one or more instructions; and at least one processor configured to execute the one or more instructions stored in the memory to: control a captured image of an actual space to be monitored outputted for a display, determine a three-dimensional (3D) frame based on at least one point included in the captured image, adjust at least one of a size and a position of the 3D frame based on a user input relative to the displayed image, and determine an interactive area corresponding to the 3D frame in the actual space.

[0042] According to another embodiment of the present disclosure, a method for operating a display device includes: transmitting a captured image of an actual space to be monitored to an electronic device; receiving information about a three-dimensional interactive zone in the actual space and content to be executed or output based on an event detected in the interactive zone from the electronic device; monitoring the interactive zone; and controlling to execute or output content based on an event detected in the monitored interactive zone.

[0043] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium includes a program recorded thereon for executing an operating method of a display device in a computer.

[0044] Implementation

[0045] The terms used in this disclosure are only used to describe specific embodiments and do not have any intention to limit the present disclosure. The terms used herein are those general terms currently understood in the prior art, but some terms may be changed according to the intentions, conventions, or new technologies in the prior art of those skilled in the art. Moreover, the applicant may choose to specify the terms, and in this case, their detailed meanings will be described in the specific embodiments. Thus, the terms used in the specification should not be understood as simple names, but based on the meaning of the terms and the entire specification.

[0046] Throughout the specification, when a component "includes" an element, unless otherwise specified, it should be understood that the component does not exclude another element but may additionally include another element. In addition, terms such as "... unit", "... module", etc. refer to a unit that performs at least one function or operation, and the unit may be implemented as hardware or software or as a combination of hardware and software.

[0047] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein commonly understood parts may be omitted to clearly describe the embodiments. In addition, throughout the specification, the same reference numerals represent the same elements. In this regard, the embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Therefore, certain embodiments are specifically described below with reference to the accompanying drawings to explain aspects.

[0048] As used herein, the term "and / or" includes any or all combinations of one or more of the associated listed items. When expressions such as "at least one of," precede a series of elements, they modify the entire series of elements and do not modify the individual elements of the series. For example, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0049] Figure 1 The interactive system according to the embodiment is shown in FIG. The interactive system according to the embodiment may include an electronic device 100 and a display device 200 .

[0050] The electronic device 100 according to the embodiment may be implemented in various forms. For example, the electronic device 100 may be implemented by various electronic devices such as cellular phones, smart phones, laptop computers, desktop personal computers (PCs), tablet PCs, e-book terminals, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation machines, MP3 players, video cameras, Internet protocol televisions (IPTVs), digital televisions (DTVs), wearable devices, etc. However, it should be understood that the electronic device 100 is not limited thereto.

[0051] In addition, the display device 200 according to the embodiment may be a sign or a TV, but this is merely illustrative, and the display device 200 may be implemented by any electronic device including a display. For example, the display device 200 may be implemented by various electronic devices such as a cellular phone, a tablet PC, a digital camera, a video camera, a laptop computer, a desktop PC, an e-book terminal, a digital broadcast terminal, a PDA, a PMP, a navigation machine, an MP3 player, a wearable device, etc. As an example, the embodiment may be implemented in a display device having a large display such as a sign or a TV, etc., but the embodiment is not limited thereto. In addition, the display device 200 may be a fixed type or a mobile type and may be a digital broadcast receiver capable of receiving a digital broadcast signal.

[0052] The term "user" in this specification indicates a person who controls a function or operation of an electronic device, a display device, an Internet of Things (IoT) device, or the like, and may include an administrator or an installation engineer, or a consumer or a general user.

[0053] The electronic device 100 and the display device 200 according to the embodiment may communicate with each other. For example, the electronic device 100 may transmit or receive data, signals, etc. to or from the display device 200 by using short-range communication. Short-range communication may include a Wi-Fi scheme, a near field communication (NFC) scheme, a Bluetooth scheme, an infrared data association (IrDA) scheme, a ZigBee scheme, etc., but is not limited thereto.

[0054] According to an embodiment, the electronic device 100 may obtain an image by photographing (or capturing) (an image of) the actual space 10 to be monitored. The electronic device 100 may use a camera included in the electronic device 100 to obtain an image by photographing the space to be monitored. Alternatively, the electronic device 100 may receive an image captured by an external device by communicating with the external device. For example, according to an embodiment, the display device 200 may photograph the actual space 10 to be monitored, which may be located in front of the display device 200, by using a camera embedded in the display device 200, a camera separately provided on the display device 200, a camera connected to the display device 200, and the like. The display device 200 may transmit the captured image to the electronic device 100 by communicating with the electronic device 100.

[0055] According to an embodiment, the electronic device 100 may set at least one interactive area 50 in the real space 10 based on an image acquired by photographing the real space 10. For example, the electronic device 100 may display the image acquired by photographing the real space 10 on a display and generate (or determine) a three-dimensional (3D) frame (e.g., a virtual 3D frame or space) based on at least one point included in the image. The electronic device 100 may adjust the size and / or position of the 3D frame based on a user input, and set an interactive area 50 corresponding to the 3D frame in the real space 10.

[0056] In addition, the electronic device 100 may set or determine at least one content corresponding to the generated 3D frame, and control the display device 200 so that the display device 200 displays the set content when an event is detected in the interactive area 50. In this case, the event may include an event in which an object is detected in the interactive area 50, an event in which an object is not detected in the interactive area 50, an event in which a motion of an object in the interactive area 50 is detected, an event in which a specific motion in the interactive area 50 is detected, and the like, but is not limited thereto. That is, in various embodiments, various events may be used or considered.

[0057] According to an embodiment, the electronic device 100 may transmit at least one of information about the set 3D frame (information about the interactive zone 50 ), information about an event, and content information corresponding to the 3D frame (interactive zone 50 ) to the display device 200 .

[0058] According to an embodiment, the display device 200 may determine the interactive zone 50 in the real space 10 based on information about the 3D frame (e.g., coordinate value information of the 3D frame, etc.). In addition, the display device 200 may monitor whether an event occurs in the interactive zone 50 based on event information. For example, the display device 200 may monitor whether an object is detected in the interactive zone 50 based on information about “object detected event”. In addition, the display device 200 may display when an event occurs (e.g., an object is detected) in the interactive zone 50 based on content information.

[0059] Figure 2 is a block diagram of an electronic device 100 according to an embodiment.

[0060] refer to Figure 2 , the electronic device 100 according to the embodiment may include a communication interface 110 , a processor 120 , a memory 130 , a display 140 , and a user input interface 150 .

[0061] According to an embodiment, the communication interface 110 may include at least one component for communication between the electronic device 100 and an external device (eg, a display device, a server, etc.).

[0062] The communication interface 110 may include one or more components (e.g., circuits, hardware, transceivers, etc.) employed alone or in combination for communicating through a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, radio frequency transmission, etc. In addition, the communication interface 110 may directly send and receive data or signals to and from an external device or an external server in a wireless manner by using a wireless LAN (WLAN) (e.g., Wi-Fi), etc.

[0063] For example, the communication interface 110 may receive an image acquired by photographing the actual space to be monitored from the external device. In addition, the communication interface 110 may transmit information about the interactive zone, event information, content information, etc. to the display device 200.

[0064] According to an embodiment, the processor 120 may execute one or more programs (or instructions) stored in the memory 130. The processor 120 may include a single core, a dual core, a triple core, a quad core, or a plurality of cores. In addition, the processor 120 may include a plurality of processors. For example, the processor 120 may be implemented by a main processor and a sub-processor operating in a sleep mode.

[0065] According to an embodiment, the memory 130 may store various data, programs, or applications for operating and controlling the electronic device 100 .

[0066] In addition, the program stored in the memory 130 may include one or more instructions. The program (one or more instructions) or application stored in the memory 130 may be executed by the processor 120.

[0067] According to an embodiment, the processor 120 may display an image acquired by photographing the actual space to be monitored on the display 140. In this case, the image displayed on the display 140 may be an image that three-dimensionally represents the actual space based on the depth value information of the objects included in the actual space. In addition, the processor 120 may generate a 3D frame based on at least one point included in the image. The processor 120 may adjust the size and / or position of the generated 3D frame based on the user input received through the user input interface 150. The processor 120 may be controlled so that an interactive area corresponding to the 3D frame is set in the photographed actual space. In addition, the processor 120 may set an event and content corresponding to the 3D frame, and control so that when an event occurs (or an event is detected) in the interactive area, the corresponding content is executed or provided.

[0068] According to an embodiment, the display 140 generates a driving signal by converting at least one of an image signal, a data signal, an on-screen display (OSD) signal, a control signal, etc. processed by the processor 120. The display 140 may be implemented by a plasma display panel (PDP), a liquid crystal display (LCD), an organic light emitting diode (OLED), a flexible display, etc., and may also be implemented by a 3D display. In addition, by configuring the display 140 with a touch screen, the display 140 may be used as an input device in addition to being used as an output device.

[0069] According to an embodiment, the display 140 may display a user interface screen image for setting the interactive area under the control of the processor 120. For example, the display 140 may display at least one of an image acquired by photographing the space to be monitored, a generated 3D frame, an interface screen image for adjusting the position and size of the 3D frame, an interface screen image for setting content corresponding to the 3D frame, and the like.

[0070] The user input interface 150 is an interface through which a user inputs data for controlling the electronic device 100. For example, the user input interface 150 may include at least one of a keypad, a dome switch, a touch pad (capacitive cover touch pad, resistive cover touch pad, infrared (IR) beam touch pad, surface acoustic wave touch pad, integrated strain gauge touch pad, piezoelectric touch pad, etc.), a scroll wheel, a jog switch, a button, a plurality of buttons, a rotatable dial, a voice input and recognition device, etc., but is not limited thereto.

[0071] According to an embodiment, the user input interface 150 may receive at least one of a user input for adjusting the viewing angle of the displayed image, a user input for selecting at least one point or object included in the image to generate a 3D frame, a user input for adjusting the size and position of the 3D frame, and the like, but is not limited thereto.

[0072] Figure 3 is a block diagram of an electronic device 300 according to another embodiment. Figure 3 The electronic device 300 may be Figure 1 An embodiment of an electronic device 100 in FIG.

[0073] refer to Figure 3 , the electronic device 300 according to an embodiment may include a processor 330 , a sensor 320 , a communication interface 340 , an output interface 350 , a user input interface 360 ​​, an audio / video (A / V) input interface 370 , and a memory 380 .

[0074] Figure 3 The communication interface 340, the processor 330, the display 351, the memory 380 and the user input interface 360 ​​may correspond to Figure 2 The communication interface 110, the processor 120, the display 140, the memory 130 and the user input interface 150 in the embodiment of the present invention may be omitted. Figure 2 The same or similar descriptions made.

[0075] The communication interface 340 may include at least one component for communication between external devices such as a server, etc. For example, the communication interface 340 may include at least one of a short-range wireless communication interface 341, a mobile communication interface 342, a broadcast reception interface 343, and the like.

[0076] The short-range wireless communication interface 341 may include at least one of a Bluetooth communication interface, an NFC interface, a WLAN (Wi-Fi) communication interface, a ZigBee communication interface, an IrDA communication interface, a Wi-Fi Direct (WFD) communication interface, an ultra-wideband (UWB) communication interface, an Ant+ communication interface, and the like, but is not limited thereto.

[0077] The mobile communication interface 342 transmits and receives wireless signals to and from at least one of a base station, an external terminal, a server, etc. in a mobile communication network. Here, the wireless signal may include a voice call signal, a video call signal, or various types of data according to text / multimedia message transmission and reception.

[0078] The broadcast receiving interface 343 receives a broadcast signal and / or broadcast related information from the outside through a broadcast channel. Here, the broadcast channel may include a satellite channel and a terrestrial channel. According to an implementation example, the electronic device 300 may not include the broadcast receiving interface 343.

[0079] The communication interface 340 may receive at least one user image from an external device.

[0080] The output interface 350 may output at least one of an audio signal, a video signal, and a vibration signal and may include a display 351 , an acoustic output interface 352 , a vibration motor 353 , and the like.

[0081] The acoustic output interface 352 may output audio data received through the communication interface 340 or stored in the memory 380. In addition, the acoustic output interface 352 may output an acoustic signal related to a function (e.g., a call signal reception sound, a message reception sound, or an alarm sound) performed by the electronic device 300. The acoustic output interface 352 may include a speaker, a buzzer, and the like. For example, when the recognized user is not an authorized user, the acoustic output interface 352 may output an alarm.

[0082] The vibration motor 353 may output a vibration signal. For example, the vibration motor 353 may output a vibration signal corresponding to the output of audio data and / or video data (e.g., a call signal reception sound, a message reception sound, etc.). In addition, when a touch is input on the touch screen, the vibration motor 353 may output a vibration signal.

[0083] The processor 330 (which may include a plurality of processors) controls the overall operation of the electronic device 300. For example, the processor 330 may control the communication interface 340, the output interface 350, the user input interface 360, the sensor 320, the A / V input interface 370, etc. by executing a program stored in the memory 380.

[0084] The sensor 320 may detect at least one of biological information of a user, a state of the electronic device 300 , or an environmental state of the electronic device 300 , and transmit the detected information to the processor 330 .

[0085] The sensor 320 may include at least one of a magnetic sensor 321, an acceleration sensor 322, a temperature / humidity sensor 323, an IR sensor 324, a gyro sensor 325, a position sensor (e.g., a global positioning system (GPS)) 326, an atmospheric pressure sensor 327, a proximity sensor 328, an RGB (illuminance) sensor 329, etc., but is not limited thereto. The function of each sensor can be intuitively inferred from its name by those skilled in the art, and therefore a detailed description thereof is omitted herein.

[0086] The A / V input interface 370 receives an input of an audio signal and / or a video signal and may include a camera 371, a microphone 372, etc. The camera 371 may acquire image frames of still images, moving pictures, etc. through an image sensor in a video call mode or a capture mode. The image captured through the image sensor may be processed by the processor 330 or a separate image processor.

[0087] The image frames processed by the camera 371 may be stored in the memory 380 and / or transmitted to the outside through the communication interface 340. Two or more cameras 371 may be included according to the configuration aspect of the electronic device 300.

[0088] Microphone 372 can receive external acoustic signals and process the external acoustic signals into electronic voice data. For example, microphone 372 can receive acoustic signals from an external device or a user. Microphone 372 can use various noise cancellation algorithms to cancel the noise generated during the process of receiving the external acoustic signal.

[0089] The memory 380 may store programs and / or instructions for processing and control of the processor 330 and store input / output data (eg, applications, contents, time band information of an external device, an address book, etc.).

[0090] The memory 380 may include at least one type of storage medium among a flash memory type memory, a hard disk type memory, a multimedia card micro memory, a card type memory (e.g., a secure digital (SD) or an eXtreme Card (XD) memory), a random access memory (RAM), a static RAM (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk, etc. In addition, the electronic device 300 may operate a web storage or a cloud server having a storage function of the memory 380 through the Internet, interoperate with it, or be connected to it.

[0091] The programs stored in the memory 380 may be classified into a plurality of modules according to their functions, such as a user interface (UI) module 381, a touch screen module 382, ​​an alarm module 383, and the like.

[0092] The UI module 381 may provide each application with a designated UI, a designated graphic UI (GUI), etc. that interoperates with the electronic device 300. The touch screen module 382 may sense a user's touch gesture on the touch screen and transmit information about the touch gesture to the processor 330.

[0093] The touch screen module 382 may recognize and analyze a touch code. The touch screen module 382 may be configured by separate hardware including a controller.

[0094] The alarm module 383 may generate a signal for notifying the occurrence of an event of the electronic device 300. Examples of events occurring in the electronic device 300 may include call signal reception, message reception, key signal input, and schedule alarms. The alarm module 383 may output an alarm signal in the form of a video signal through the display 351, in the form of an audio signal through the acoustic output interface 352, and / or in the form of a vibration signal through the vibration motor 353.

[0095] exist Figure 2 and Figure 3 The block diagrams of the electronic devices 100 and 300 shown in the figure are merely illustrative. Each component in the block diagram may be integrated, added, or omitted according to the specifications of the electronic devices 100 and 300 actually implemented. That is, two or more components may be integrated into one component, one component may be separated into two or more components, and / or one or more components may be omitted, depending on the circumstances. In addition, the functions performed in each frame will describe the embodiments, and their detailed operations or devices are not limited.

[0096] Figure 4 is a block diagram of a display device 200 according to an embodiment.

[0097] refer to Figure 4 , the display device 200 according to the embodiment may include a sensor 210 , a communication interface 220 , a processor 230 , a memory 240 , and a display 250 .

[0098] According to an embodiment, the sensor 210 may sense a state of the display device 200 or an environmental state and transmit the sensed information (or the sensed state) to the processor 230 .

[0099] The sensor 210 may include an image sensor and / or a depth sensor. The display device 200 may acquire image frames of still images, videos, etc. through the image sensor. In this case, the image captured by the image sensor may be processed by the processor 230 or a separate image processor.

[0100] For example, the image sensor may photograph (or capture) (an image of) the actual space to be monitored, recognize objects in the interactive zone, sense the movement of objects, etc. Herein, the objects may include humans, human hands, human fingers, etc., but are not limited thereto.

[0101] In addition, the depth sensor may measure the depth value of an actual object present in the actual space to be monitored, etc. The depth value may correspond to the distance from the depth sensor to the specific object. In this case, as the distance from the depth sensor to the specific object becomes farther, the depth value may increase (although it should be understood that one or more other embodiments are not limited thereto).

[0102] According to an embodiment, the depth sensor may obtain a depth value of an object in various methods. For example, the depth sensor may measure the depth value by using at least one of a time of flight (TOF) scheme, a stereoscopic vision scheme, or a structured light pattern scheme.

[0103] The image sensor and the depth sensor may be configured by separate sensors or configured as one sensor. In addition, two or more image sensors and / or depth sensors may be included according to the configuration aspect of the display device 200.

[0104] According to an embodiment, the sensor 210 may include at least one of a microphone, a human detection sensor (eg, an IR sensor, an ultrasonic sensor, etc.), an illumination sensor, or a temperature / humidity sensor in addition to the image sensor and the depth sensor, but is not limited thereto.

[0105] The microphone can receive an external acoustic signal and process the external acoustic signal into electronic voice data. For example, the microphone can receive an acoustic signal from an external device or a user. The microphone can use various noise cancellation algorithms to cancel the noise generated during the process of receiving the external acoustic signal.

[0106] The functions of the human detection sensor (eg, IR sensor, ultrasonic sensor, etc.), illumination sensor, and temperature / humidity sensor may be intuitively inferred from their names by those skilled in the art, and thus a detailed description thereof is omitted herein.

[0107] According to an embodiment, the communication interface 220 may include at least one component for communication between the display device 200 and an external device (eg, an electronic device, a server, etc.).

[0108] The communication interface 220 may include one or more components used alone or in combination for communicating through a LAN, a WAN, a VAN, a mobile radio communication network, a satellite communication network, radio frequency transmission, etc. In addition, the communication interface 220 may directly send and receive data or signals to and from an external device or an external server in a wireless manner by using a WLAN (e.g., Wi-Fi), Bluetooth, NFC, etc.

[0109] For example, the communication interface 220 may receive information about the interactive zone, event information, and content information from the electronic device 100. Alternatively, the communication interface 220 may receive content to be executed by the display device 200 from the electronic device 100 or an external server.

[0110] According to an embodiment, the processor 230 can execute one or more programs stored in the memory 240. The processor 230 may include a single core, a dual core, a triple core, a quad core, or a plurality of cores. In addition, the processor 230 may include a plurality of processors. For example, the processor 230 may be implemented by a main processor and a sub-processor operating in a sleep mode.

[0111] According to an embodiment, the memory 240 may store various data, programs, or applications for operating and controlling the display device 200 .

[0112] In addition, the program stored in the memory 240 may include one or more instructions. The program (one or more instructions) or application stored in the memory 240 may be executed by the processor 230.

[0113] According to an embodiment, the processor 230 may determine the interactive area based on information about the interactive area (e.g., coordinate value information of the 3D frame, etc.) and control to monitor the interactive area. For example, the processor 230 may monitor whether a preset event occurs in the interactive area, and when the occurrence of the event is detected, execute the content corresponding to the occurrence of the event.

[0114] According to an embodiment, the display 250 generates a driving signal by converting at least one of an image signal, a data signal, an OSD signal, a control signal, etc. processed by the processor 230. The display 250 may be implemented by a PDP, an LCD, an OLED, a flexible display, etc., and may also be implemented by a 3D display. In addition, by being configured with a touch screen, the display 250 may be used as an input device in addition to being used as an output device. According to an embodiment, the display 250 may display an execution screen image of content corresponding to the occurrence of an event.

[0115] Figure 5 is a signaling diagram of an operating method of an interactive system according to an embodiment.

[0116] The interactive system according to the embodiment may include the electronic device 100 (or the electronic device 300 ) and the display device 200 .

[0117] refer to Figure 5In operation S410, the display device 200 may capture the actual space to be monitored. For example, the display device 200 may capture the actual space to be monitored by using an image sensor and / or a depth sensor and obtain depth value information of one or more objects included in the space to be monitored.

[0118] In operation S420, the display device 200 may transmit the captured image and / or depth value information of one or more objects included in the image to the electronic device 100. According to another embodiment, the depth value information may be determined by the electronic device 100 through, for example, analysis of the captured image.

[0119] The electronic device 100 may display the captured image of the real space on the display. For example, the electronic device 100 may display an image that three-dimensionally indicates the real space based on depth value information of an object included in the real space.

[0120] In operation S430, the electronic device 100 may set an interactive zone based on the displayed image and set content corresponding to the interactive zone.

[0121] For example, the electronic device 100 may generate a 3D frame based on at least one point included in the image. In addition, the electronic device 100 may adjust the size and / or position of the generated 3D frame based on user input. In addition, the electronic device 100 may set an interactive area corresponding to the 3D frame in the actual space captured. In addition, the electronic device 100 may set an event and content corresponding to the 3D frame, and when an event occurs in the interactive area corresponding to the 3D frame, control is performed so as to execute or output the corresponding content.

[0122] Reference below Figure 6 , Figure 7 , Fig. 8A , Figure 8B , Fig.9A Fig. 9B , Fig.10 and Fig.11 To describe in more detail Figure 5 Operation 430.

[0123] In operation S440, the electronic device 100 may transmit at least one of information about the interactive area (eg, coordinate value information of the 3D box), information about the event, and information about the content to the display device 200.

[0124] In operation S450, the display device 200 may monitor the interactive zone based on the information about the interactive zone received from the electronic device 100. For example, the display device 200 may determine the interactive zone corresponding to the 3D frame in the actual space photographed by the display device 200 and monitor whether a preset event occurs in the interactive zone.

[0125] When the occurrence of an event is detected in operation S460, in operation S470, the display device 200 may execute or output content corresponding to the occurrence of the event. The content may include at least one of image content, text content, audio content, video content, game content, etc., but is not limited thereto. In this case, the content may be received from the electronic device 100 or an external server according to an embodiment, or may be pre-stored in the display device 200.

[0126] Figure 6 is a flowchart of an operating method of the electronic device 100 according to an embodiment.

[0127] Figure 6 is a detailed flow chart of the operation of setting the interactive area for the electronic device 100, for example, Figure 5 A detailed flowchart corresponding to operation S430 is shown.

[0128] refer to Figure 6 , in operation S510, the electronic device 100 according to the embodiment may display an image acquired by photographing a real space.

[0129] For example, the electronic device 100 may receive an image acquired by photographing the real space from the display device 200 or another external device according to an embodiment. In this case, the image acquired by photographing the real space may include depth value information of objects included in the real space, but is not limited thereto. According to another embodiment, the electronic device 100 may capture an image and / or determine depth value information.

[0130] In operation S520, the electronic device 100 may generate a 3D frame based on at least one point included in the image. For example, when a user input for selecting any one point of the image is received, the electronic device 100 may determine whether the selected point is included in the object. When the selected point is not included in the object, the electronic device 100 may generate a 3D frame with a preset size. Otherwise, when the selected point is included in the object, the electronic device 100 may generate a 3D frame according to the shape and size of the corresponding object. Alternatively, the electronic device 100 may detect the objects included in the image and generate the 3D frame by reconfiguring the detected objects in 3D. The electronic device 100 may display the generated 3D frame on the image.

[0131] In addition, the electronic device 100 can adjust the viewing angle of the displayed image based on user input (e.g., drag input of the image). When the viewing angle of the displayed image is changed, the position, size, direction, etc. of the object in the image can be changed. The electronic device 100 can store images with different viewing angles and provide the stored images as shortcuts.

[0132] In operation S530, the electronic device 100 may adjust the size and / or position of the generated 3D frame.

[0133] For example, the electronic device 100 may display a horizontal / vertical resizing object on the surface of the generated 3D frame seen from the front based on the viewing angle of the image. The horizontal / vertical resizing object may adjust the size in the horizontal direction and / or vertical direction relative to the surface seen from the front. In addition, the electronic device 100 may display a rotation adjustment object through which the 3D frame is rotatable.

[0134] The electronic device 100 may adjust the size and / or position of the 3D frame based on a user input on a horizontal / vertical size adjustment object and / or a rotation adjustment object.

[0135] In operation S540, the electronic device 100 may control so as to set an interaction area corresponding to the 3D frame in the real space.

[0136] For example, the electronic device 100 may transmit the coordinate value information of the 3D frame to the display device 200. The display device 200 may determine an interactive zone in the real space based on the received coordinate value information of the 3D frame and monitor whether an event occurs in the determined interactive zone.

[0137] Figure 7 , Fig. 8A , Figure 8B , Fig.9A and Fig. 9B The figure illustrates a method of setting an interaction zone performed by the electronic device 100 according to one or more embodiments.

[0138] refer to Figure 7 , the electronic device 100 may display an image 601 acquired by photographing the actual space to be monitored on a display. The image 601 acquired by photographing the actual space to be monitored may be received from an external device (e.g., the display device 200) or captured by the electronic device 100. In addition, the image 601 acquired by photographing the actual space to be monitored may include at least one object, and the electronic device 100 may display the captured image 601 as a 3D image based on depth value information of the object.

[0139] The electronic device 100 may generate a 3D frame 610 based on at least one point included in the image 601. For example, when a user input for selecting any one point on the image is received, the electronic device 100 may generate a 3D frame having a preset size. Alternatively, when the selected point is included in an object, the electronic device 100 may generate a 3D frame according to the shape and / or size of the corresponding object (e.g., length, width, and depth dimensions having a predetermined relationship or correspondence with the shape and / or size of the corresponding object).

[0140] Alternatively, if Fig. 8A As shown in , the electronic device 100 can detect an object included in the image 710 and generate 3D frames 701, 702, 703, 704, 705, 706, and 707 by reconfiguring the detected object in 3D. In this case, the electronic device 100 can generate the 3D frames 701, 702, 703, 704, 705, 706, and 707 by using depth value information of the object.

[0141] The electronic device 100 may adjust the viewing angle of the image 710 based on user input. Fig. 8A As shown in FIG. 1 , when a drag input 720 of touching a point of the image 710 having the first viewing angle and moving in a certain direction is received, the electronic device 100 may display an image 730 such that the image 730 has a Figure 8B The second perspective shown in .

[0142] The electronic device 100 may store images with different viewing angles and provide the stored images as shortcuts. For example, the electronic device 100 may store an image 710 ( Fig. 8A ) and an image 730 (shown in FIG. Figure 8B ) and provide images 710 and 730 as shortcuts. Alternatively, the electronic device 100 may store images with viewing angles captured by a camera and provide the stored images as shortcuts.

[0143] The electronic device 100 may adjust the size and / or position of the generated 3D frame. Fig.9AAs shown in , the electronic device 100 can display a horizontal / vertical resizing object 820 on a surface 810 (hereinafter, referred to as the first surface 810) seen from the front based on the viewing angle of the image in the generated 3D frame. For example, when the horizontal / vertical resizing object 820 is displayed and then a drag input in the horizontal direction is received, the electronic device 100 can increase or decrease the horizontal direction size of the first surface 810. In this case, together with the adjustment of the size of the first surface 810, the size of the 3D frame including the first surface 810 (e.g., the horizontal or vertical length of the 3D frame) can be adjusted. However, the embodiment is not limited thereto.

[0144] In addition, the electronic device 100 may display a rotation adjustment object 830. When the rotation adjustment object 830 is displayed and then a drag input in a clockwise or counterclockwise direction is received, the electronic device 100 may rotate the 3D frame in a clockwise or counterclockwise direction based on a straight line perpendicular to the first surface 810 and passing through the center of the first surface 810.

[0145] In addition, if Fig. 9B As shown in , when the viewing angle of the image is changed, the electronic device 100 can display a horizontal / vertical size adjustment object 850 and a rotation adjustment object 860 on a surface 840 (hereinafter, referred to as the second surface 840) seen in front of the image based on the changed viewing angle. Therefore, the electronic device 100 can adjust the height of the 3D box when receiving a drag input in the vertical direction or adjust the horizontal length of the 3D box when receiving a drag input in the horizontal direction. Alternatively, the electronic device 100 can rotate the 3D box in a clockwise or counterclockwise direction based on a straight line perpendicular to the second surface 840 when receiving a drag input in the clockwise or counterclockwise direction.

[0146] In addition, the electronic device 100 may provide a background image lock function (or a viewing angle lock function) so as not to change the viewing angle of the displayed image while adjusting the size and position of the 3D frame. For example, the electronic device 100 may display a background image lock icon 870. In this case, when an input for selecting the background image lock icon 870 is received, the electronic device 100 may maintain the viewing angle of the image without changing even when a drag input on the image is received.

[0147] The electronic device 100 may set an interaction zone corresponding to the 3D frame in the real space based on the coordinate value information of the 3D frame displayed on the image. Therefore, according to an embodiment, the display device 200 may monitor whether a preset event occurs in the set interaction zone.

[0148] Fig.10 and Fig.11The figure illustrates a method of setting content corresponding to an interactive zone, performed by the electronic device 100, according to one or more embodiments.

[0149] refer to Fig.10 When setting a 3D frame, the electronic device 100 may set content corresponding to the 3D frame. Fig.10 As shown in , the electronic device 100 may display a content list 910 including a plurality of contents. The content may include at least one of image content, text content, audio content, video content, game content, etc., but is not limited thereto.

[0150] When receiving an input 920 for selecting any one of the multiple 3D frames displayed on the display, the electronic device 100 may display a first area 930 corresponding to a first event and a second area 940 corresponding to a second event. In this case, the first event may be an event in which an object is detected in an interactive area corresponding to the 3D frame. In addition, the second event may be an event in which an object is not detected in an interactive area corresponding to the 3D frame. However, it should be understood that one or more other embodiments are not limited to the first and second events as described above.

[0151] The electronic device 100 may receive a selection of one or more contents included in the content list 910 to the first area 930 or the second area 940 (e.g., an input of dragging and dropping one or more contents included in the content list 910 to the first area 930 or the second area 940). For example, the electronic device 100 may display thumbnail images 931 and 932 (or other identifiers) of the first content and the second content, respectively, in the first area 930 upon receiving an input of dragging and dropping the first content and the second content to the first area 930. In addition, the electronic device 100 may display a thumbnail image 933 (or other identifier) ​​of the third content in the second area 940 upon receiving an input of dragging and dropping the third content to the second area 940.

[0152] Therefore, the electronic device 100 may control or provide a setting so that when the first event is detected to occur in the interactive area corresponding to the 3D frame (or based on the detection of the first event occurring in the interactive area corresponding to the 3D frame), the display device 200 executes or outputs the first content and the second content. In addition, the electronic device 100 may control or provide a setting so that when the second event is detected to occur in the interactive area corresponding to the 3D frame (or based on the detection of the second event occurring in the interactive area corresponding to the 3D frame), the display device 200 executes or outputs the third content.

[0153] refer to Fig.11 When multiple contents are set as one event, the electronic device 100 can set a sub-interaction area.

[0154] For example, when (or based on) the occurrence of a first event is detected in the interactive area corresponding to the selected 3D frame 1010, the display device 200 may be configured to execute or output a plurality of (first to fourth) contents A, B, C, and D. In this case, Fig.11 As shown in , the electronic device 100 may generate and display a sub-frame 1020 corresponding to a sub-interaction area 1030. The sub-interaction area 1030 may be set as an area in front of (eg, directly in front of) the display device 200, but is not limited thereto.

[0155] The size and / or position of the sub-frame 1020 corresponding to the sub-interaction area 1030 may also be adjusted by using the horizontal / vertical size adjustment object 1040 and / or the rotation adjustment object 1050. The sub-interaction area 1030 may include a plurality of (first to fourth) indicator areas corresponding to the plurality of contents A, B, C, and D, respectively. In addition, the electronic device 100 may adjust the size and / or position of each indicator area.

[0156] For example, the sub-frame 1020 may include a plurality of indicator frames corresponding to the plurality of contents A, B, C, and D, respectively, and a plurality of indicator regions corresponding to the plurality of indicator frames, respectively, may be included in the sub-interaction region 1030. The electronic device 100 and / or the display device 200 may be configured so that when an event (e.g., detection of an object) occurs in a first indicator region (or based on the occurrence of an event), the first content A is executed or output, when the event occurs in a second indicator region, the second content B is executed or output, when the event occurs in a third indicator region, the third content C is executed or output, and when the event occurs in a fourth indicator region, the fourth content D is executed or output.

[0157] In addition, according to an embodiment, the display device 200 may display an indicator 1060 indicating the positions of the first to fourth indicator regions. In addition, the display device 200 may highlight and display an indicator corresponding to the content currently displayed thereon.

[0158] Fig.12 The diagram illustrates the operation of an interactive system including the electronic device 100 and the display device 200 according to an embodiment.

[0159] The electronic device 100 may transmit the generated coordinate value information of the 3D frame, information about an event corresponding to the 3D frame, and information about content corresponding to the event to the display device 200 .

[0160] Therefore, the display device 200 may determine the interactive zone 1150 corresponding to the 3D frame in the real space based on the coordinate value information of the 3D frame and monitor whether an event (e.g., detection of an object) occurs in the interactive zone 1150. When an object is detected in the interactive zone 1150, the display device 200 may execute preset content.

[0161] In addition, the display device 200 may transmit information about the position of the detected object (eg, coordinate value information) to the electronic device 100. Fig.12 As shown in , the electronic device 100 can display information about the location of the detected object using points, surfaces, frames, etc.

[0162] Fig.13 The figure illustrates a method for setting tag information in an interactive area according to an embodiment.

[0163] refer to Fig.13 , the electronic device 100 may set or determine tag information for the 3D frame. For example, at least one of a product name (e.g., model name and / or number), a manufacturer, a manufacturing year, a release year, a color, a price, etc. of a product located or to be located in an interactive area corresponding to the 3D frame is set as tag information, but is not limited thereto.

[0164] When an event is detected to occur in the interactive zone (or based on the event being detected to occur in the interactive zone), the electronic device 100 or another device (e.g., an external server) may collect multiple tag information corresponding to the interactive zone. For example, a first tag information (e.g., GALAXY S9, black, $800) may be set or determined for the first interactive zone, a second tag information (e.g., GALAXY NOTE, blue, $700) may be set or determined for the second interactive zone, and a third tag information (e.g., GALAXY NOTE 8, black, $700) may be set or determined for the third interactive zone. When (or based on) an event in which an object (e.g., an object) is detected in the first interaction zone occurs five times, an event in which an object (e.g., an object) is detected in the second interaction zone occurs twice, and an event in which an object (e.g., an object) is detected in the third interaction zone occurs once, data may be collected five times for GALAXY S9 and three times for GALAXYNOTE 8 according to model name, data may be collected six times for black and twice for blue according to color, and data may be collected five times for USD 800 and three times for USD 700 according to price.

[0165] Therefore, the electronic device 100 or other devices (eg, an external server) may analyze data of the interaction between the occurrence of an event and sales by using the number of times the tag information is collected.

[0166] Fig.14A and Fig. 14B The figure illustrates a method of storing a set interaction zone performed by the electronic device 100 according to an embodiment.

[0167] The electronic device 100 may generate a plurality of 3D frames and set a plurality of interaction areas respectively corresponding to the plurality of 3D frames.

[0168] For example, Fig.14A As shown in , the electronic device 100 may generate three 3D boxes 1310, 1320, and 1330, and set or control in the real space to set first to third interaction areas respectively corresponding to the three 3D boxes 1310, 1320, and 1330. In this case, the electronic device 100 may store a layout including the three 3D boxes 1310, 1320, and 1330 as a first file. For example, coordinate value information and label information of the three 3D boxes 1310, 1320, and 1330 may be stored.

[0169] In addition, if Fig. 14B As shown in , the electronic device 100 may generate four 3D boxes 1340, 1350, 1360, and 1370 and set or control in the real space to set fourth to seventh interaction areas corresponding to the four 3D boxes 1340, 1350, 1360, and 1370, respectively. In this case, the electronic device 100 may store a layout including the four 3D boxes 1340, 1350, 1360, and 1370 as a second file. For example, coordinate value information and label information of the four 3D boxes 1340, 1350, 1360, and 1370 may be stored.

[0170] Therefore, the electronic device 100 can change and set the content corresponding to the 3D frame (or the interactive area) by using the stored first or second file, thereby easily constituting an interactive system.

[0171] Fig.15 The diagram illustrates an operating method of an interactive system according to an embodiment.

[0172] refer to Fig.15, the electronic device 100 may be set and / or controlled so that when multiple events occur simultaneously, multiple contents corresponding to the multiple events are executed simultaneously. For example, when an event occurs simultaneously in the first interaction zone 1410 and the second interaction zone 1420 set in the actual space (or determined for the actual space) by the electronic device 100 according to the embodiment, it may be set (for example, according to user input or by default) so that both the first content corresponding to the first interaction zone 1410 and the second content corresponding to the second interaction zone 1420 are executed or output. Therefore, when an event occurs simultaneously in the first interaction zone 1410 and the second interaction zone 1420 (or is detected simultaneously), the display device 200 may divide the screen into two areas (the first area 1430 and the second area 1440) (or display the divided screen), and display the first content on the first area 1430 and the second content on the second area 1440.

[0173] In addition, when the display device 200 displays the first content on the full screen in response to the first event occurring in the first interactive area 1410, and the event occurs in the second interactive area 1420 within the preset time, the display device 200 may divide the screen into two areas (the first area 1430 and the second area 1440). In this case, the preset time may be determined based on the user input or the execution time of the first content. In addition, the display device 200 may display the first content on the first area 1430 and the second content on the second area 1440, but is not limited thereto in one or more other embodiments.

[0174] Fig.16 The diagram illustrates an operating method of an interactive system according to an embodiment.

[0175] refer to Fig.16 According to an embodiment, the interactive system may include an electronic device 100, a display device 200, and an Internet of Things (IoT) device 1400. According to an embodiment, the electronic device 100 may be set or controlled to allow the IoT device 1400 to operate when an event is detected to occur in a set interactive area. The IoT device 1400 may be, for example, a lighting device, but is not limited thereto.

[0176] For example, Fig.16 As shown in , the electronic device 100 may be configured or controlled to turn on the lighting device 1400 when (or based on) detecting that a first event (e.g., an event in which a certain object is detected) occurs in the interactive area 1450 corresponding to the 3D frame 1460. In addition, the electronic device 100 may be configured or controlled to turn off the lighting device 1400 when (or based on) detecting that a second event (e.g., an event in which a certain object is not detected) occurs in the interactive area 1450 corresponding to the 3D frame 1460.

[0177] In this case, the display device 200 may monitor the interactive area 1450, and transmit information indicating the occurrence of the first event or a turn-on command to the lighting device 1400 when the first event is detected to occur in the interactive area 1450. In addition, when the second event is detected to occur in the interactive area 1450, the display device 200 may transmit information indicating the occurrence of the second event or a turn-off command to the lighting device 1400.

[0178] The lighting device 1400 may receive information indicating at least one of the occurrence of an event, a turn-on command, a turn-off command, etc. from the display device 200 and turn on or off the lighting device 1400 according to the received information (eg, the occurrence of an event).

[0179] Fig.17 is a signaling diagram of an operating method of an interactive system according to an embodiment.

[0180] refer to Fig.17 , the interactive system according to the embodiment may include an electronic device 100 or 300 , a monitoring device 400 , and an IoT device 500 .

[0181] In operation S1510, the electronic device 100 according to the embodiment may obtain (e.g., capture) an image by photographing the actual space to be monitored. The image obtained by photographing the actual space to be monitored may be captured by the electronic device 100, the monitoring device 400, an external device, an accessory device, etc.

[0182] The electronic device 100 may display the captured image of the real space on a display (eg, an internal display or an external display). For example, the electronic device 100 may display an image that three-dimensionally indicates the real space based on depth value information of an object included in the real space.

[0183] In operation S1520, the electronic device 100 may set an interactive zone based on the displayed image and set an operation of the IoT device 500 corresponding to the interactive zone. Figure 5 Operation S430 and Figure 6 , Figure 7 , Fig. 8A , Figure 8B , Fig.9A and Fig. 9B The method of setting the interactive zone by the electronic device 100 is described in detail, and thus a detailed description thereof is omitted below.

[0184] Furthermore, the electronic device 100 may be configured or controlled to allow the IoT device 500 to perform a preset operation when (or based on) an event occurs in the interaction zone.

[0185] The electronic device 100 may transmit information about the interactive area (e.g., coordinate value information of the 3D box, etc.) and information about the event to the monitoring device 400 in operation S1530 and transmit information about the operation of the set IoT device 500 to the IoT device 500 in operation S1540. Alternatively or additionally, the electronic device 100 may transmit information about the operation of the set IoT device 500 to the monitoring device 400.

[0186] In operation S1550, the monitoring device 400 may monitor the interactive zone based on the information about the interactive zone that has been received from the electronic device 100. For example, the monitoring device 400 may determine the interactive zone corresponding to the 3D frame in the actual space photographed or captured by the monitoring device 400 (or another device) and monitor whether a preset event occurs in the interactive zone.

[0187] When (or based on) detecting the occurrence of the event in operation S1560 , the monitoring device 400 may transmit information indicating the occurrence of the event to the IoT device 500 in operation S1570 .

[0188] In operation S1580 , the IoT device 500 may perform an operation corresponding to the occurred (eg, detected) event.

[0189] Fig.18 The diagram illustrates an operating method of an interactive system according to an embodiment.

[0190] refer to Fig.18 , the interactive system according to the embodiment may include an electronic device 100 , a monitoring device 400 , and an IoT device 500 .

[0191] According to an embodiment, the electronic device 100 may control so as to set the first interaction zone 1610, the second interaction zone 1620, and the third interaction zone 1630 in the actual space. The electronic device 100 may obtain an image by photographing the actual space to be monitored (or from another device or accessory) and set the interaction zone based on the obtained image. Figure 6 , Figure 7 , Fig. 8A , Figure 8B , Fig.9A and 9B The method of setting the interactive zone by the electronic device 100 is described in detail, and thus a detailed description thereof is omitted below.

[0192] The monitoring device 400 may monitor whether a preset event occurs in each of the first to third interaction zones 1610 , 1620 , and 1630 .

[0193] For example, the electronic device 100 may set an event corresponding to each of the first to third interaction zones 1610, 1620, and 1630 as an event in which a certain object is detected. In addition, the electronic device 100 may set an on / off operation of the washing machine 511 as an operation corresponding to the first interaction zone 1610, set an on / off operation of the microwave oven 512 as an operation corresponding to the second interaction zone 1620, and set an on / off operation or a volume adjustment operation of the speaker 513 as an operation corresponding to the third interaction zone 1630, but it should be understood that the operations corresponding to the first to third interaction zones 1610, 1620, and 1630 are not limited thereto.

[0194] The monitoring device 400 may transmit information indicating the occurrence of an event to the washing machine 511 based on detecting an event (e.g., a first event) in the first interaction zone 1610, to the microwave oven 512 based on detecting an event (e.g., a first event or a second event) in the second interaction zone 1620, and to the speaker 513 based on detecting an event (e.g., a first event, a second event, or a third event) in the third interaction zone 1630. Alternatively, the monitoring device 400 may transmit a control command to the washing machine 511, the microwave oven 512, and the speaker 513 so that each of the washing machine 511, the microwave oven 512, and the speaker 513 performs a preset operation corresponding to the occurrence of the event. Each of the washing machine 511, the microwave oven 512, and the speaker 513 may perform an on / off operation or a volume adjustment operation in response to the occurrence of the event.

[0195] Therefore, the first to third interaction zones 1610 , 1620 , and 1630 may respectively operate as virtual switches for the IoT devices (washing machine 511 , microwave oven 512 , and speaker 513 ).

[0196] Fig.19 The diagram illustrates an operating method of an interactive system according to an embodiment.

[0197] refer to Fig.19 , the interactive system may include a monitoring device 400 and at least one of the IoT devices 520 , 530 , 540 , 550 , or 560 .

[0198] According to an embodiment, the monitoring device 400 may be a display device, but is not limited thereto. The monitoring device 400 may include a camera 410, a microphone 420, a human body detection sensor 430, an illumination sensor 440, a temperature sensor 450, and the like. The camera 410, the microphone 420, the human body detection sensor 430, the illumination sensor 440, the temperature sensor 450, and the like may be modules embedded in the monitoring device 400 or may be configured as separate modules and provided to the monitoring device 400.

[0199] The monitoring device 400 may monitor whether a preset event occurs, and when an event occurs (eg, based on detection of the event), transmit information indicating the occurrence of the event to an IoT device corresponding to the occurred (eg, detected) event.

[0200] For example, the monitoring device 400 can detect humans in the set interactive zone by using the camera 410 to identify the detected person or monitor the number of people. The monitoring device 400 can transmit information about the identified person or information about the number of detected people to the TV. Based on the received information, the TV can be turned on, changed from the current channel to the channel corresponding to the identified person, or changed from the current channel to the channel corresponding to the number of detected people.

[0201] The monitoring device 400 can detect a baby crying by using the microphone 420. The monitoring device 400 can transmit information indicating the occurrence of an event (a baby crying is detected) to the audio device 520 or the cellular phone 530. Based on the received information, the audio device 520 can play a lullaby, and / or the cellular phone 530 can send a text message or make a call to a preset external device.

[0202] The monitoring device 400 can identify the user by analyzing the voice input via the microphone 420. When a specific user is identified, the monitoring device 400 can transmit information indicating the occurrence of an event (specific user identification) to the corresponding IoT device, so that the corresponding IoT device performs operations corresponding to the identified specific user, respectively. The corresponding IoT devices can perform preset operations based on the received information, respectively. For example, the audio device 520 can play music corresponding to the identified specific user, the lighting device 550 can be turned on at a brightness corresponding to the identified specific user, and / or the air conditioner 560 can set the desired temperature to the temperature corresponding to the identified specific user.

[0203] The monitoring device 400 may be detected by using the human body detection sensor 430. When (or based on) the human body is detected, the monitoring device 400 may transmit information indicating the occurrence of an event to the lighting device 550, the TV, and / or the audio device 520. Based on the received information, the lighting device 550, the TV, and / or the audio device 520 may perform a turn-on operation.

[0204] Alternatively, when a human body is detected, the monitoring device 400 may transmit information indicating the occurrence of an event to the camera 410 and the cellular phone 530 , and the cellular phone 530 may transmit a text message or an image captured by the camera 410 to a preset external device.

[0205] When a human body is not detected for a preset time or longer, the monitoring device 400 may transmit information indicating the occurrence of an event to the lighting device 550 and the TV, and the lighting device 550 and the TV may perform a shut-down operation.

[0206] According to an embodiment, the monitoring device 400 may detect the illuminance by using the illuminance sensor 440. When it is detected that the illuminance is brighter than the preset illuminance, the monitoring device 400 may transmit information indicating the occurrence of an event to the lighting device 550 and / or the electric curtain. Based on the received information, the lighting device 550 may be turned off or the illuminance may be adjusted to be darker, and / or the electric curtain may perform a curtain closing operation.

[0207] In addition, when it is detected that the illuminance is darker than the preset illuminance (or a different preset illuminance value 0), the monitoring device 400 can transmit information indicating the occurrence of the event to the lighting device 550 and / or the electric curtain. Based on the received information, the lighting device 550 can be turned on or the illuminance can be adjusted to be brighter, and / or the electric curtain can perform a curtain opening operation.

[0208] According to an embodiment, the monitoring device 400 may detect the temperature by using the temperature sensor 450. When it is detected that the temperature is higher or lower than the preset temperature (or higher than the first preset temperature and lower than the second preset temperature), the monitoring device 400 may transmit information indicating the occurrence of the event to at least one of the air conditioner 560, the electric fan, the furnace, the heater, etc. Based on the received information, the air conditioner 560, the electric fan, the furnace, the heater, etc. may be turned on / off and / or set to an appropriate temperature.

[0209] According to an embodiment, the monitoring device 400 may collect current time information. When it becomes a preset time, the monitoring device 400 may transmit information indicating the occurrence of an event (e.g., reaching a preset time) to the corresponding IoT device, so that the corresponding IoT device performs operations corresponding to the preset time, respectively. The corresponding IoT devices may perform preset operations based on the received information, respectively. For example, the electric curtain may perform a curtain opening operation, the TV may be turned on, the robot cleaner 540 may perform cleaning work, and the audio device 520 may perform an operation of playing music.

[0210] According to an embodiment, the monitoring device 400 can detect the operation of the microwave oven. When the microwave oven is operating, the monitoring device 400 can transmit information indicating the occurrence of an event to the ventilator. The ventilator can operate based on the received information. In addition, when the operation of the microwave oven ends because cooking is completed, the ventilator can stop operating.

[0211] According to an embodiment, the interactive system is not limited to reference Fig.19 Described example. Reference Fig.19The examples described are merely illustrative, and the interactive system may be implemented in various modified or applied forms according to various embodiments. In addition, although in the embodiments described above, the monitoring device 400 transmits information (or control signals) indicating the occurrence of an event to the IoT device (multiple), it should be understood that one or more other embodiments are not limited thereto. For example, in one or more other embodiments, the monitoring device 400 may transmit information or control information or signals to an intermediate device such as a controller (e.g., a smart home controller, a ZigBee controller, a Z-wave controller, an IoT device controller, a third-party controller), a bridge, a gateway, a server, and the like.

[0212] Fig. 20 The diagram illustrates a method of setting an interactive zone performed by the display device 2000 according to an embodiment.

[0213] According to an embodiment, the display device 2000 may be used to perform Figure 6 , Figure 7 , Fig. 8A , Figure 8B , Fig.9A , Fig. 9B , Figures 10 to 13 , Fig.14A and Fig. 14B The operations of setting the interactive area, setting the content corresponding to the interactive area, setting tag information for the interactive area, and / or storing the interactive area are shown in and have been described with reference thereto.

[0214] refer to Fig. 20 , the display device 2000 may display an image 2110 on a display by photographing (e.g., capturing) (an image of) the actual space to be monitored. The image 2110 obtained by photographing the actual space to be monitored may be received from an external device or captured by the display device 2000. In addition, the image 2110 obtained by photographing the actual space may include at least one object and may include depth value information of the object.

[0215] The display device 2000 may generate a 3D frame based on at least one point or object included in the image. In this case, the display device 2000 may receive at least one of an input for selecting any one point included in the image, a drag input from any one point to another point on the image, an input for selecting an object included in the image, etc. from the control device 2200.

[0216] According to an embodiment, the control device 2200 may be implemented by various types of devices (such as a remote control or a cellular phone) for controlling the display device 2000. In addition, the control device 2200 may control the display device 2000 by using a short-range communication scheme including an infrared communication scheme, a Bluetooth communication scheme, a limited position (or distance) communication scheme, etc. The control device 2200 may control the function of the display device 2000 by using at least one of a key (including a hard button, a soft key, a scroll wheel, a rotatable dial, etc.), a touch pad, a microphone capable of receiving a user's voice, a sensor capable of recognizing a motion of the control device 2200, and the like.

[0217] In addition, the display device 2000 may display the generated 3D frame 2130. The display device 2000 may adjust at least one of the viewing angle of the displayed image 2110 and the size and position of the 3D frame 2130 based on the user input received from the control device 2200 (e.g., similar to the reference 2 Figure 7 , Fig. 8A , Figure 8B , Fig.9A as well as Fig. 9B described above).

[0218] The display device 2000 may be controlled so as to set an interactive area corresponding to the 3D frame 2130 in the real space. Fig.21 As shown in , an interaction area 2140 corresponding to the 3D frame 2130 may be set in the real space.

[0219] In addition, when the 3D frame 2130 is set, the display device 2000 may set an event corresponding to the 3D frame 2130, and an operation to be performed when (or based on) the event is detected. The display device 2000 may set an event in which a specific motion (gesture) is detected, an event in which an object is detected, an event in which an object is not detected, and the like as an event corresponding to the 3D frame 2130, although it should be understood that the event corresponding to the 3D frame 2130 is not limited thereto.

[0220] For example, Fig. 20 As shown in , the display device 2000 may set an event in which a specific motion is detected as an event corresponding to the 3D frame 2130. For example, the specific motion may include a first motion of opening and then clenching the hand to make a fist, a second motion of opening and then moving the hand downward and upward, and a third motion of opening and then shaking the hand to the left and right.

[0221] In addition, the display device 2000 may set a power on / off operation, a channel switch operation, a volume adjustment operation, a screen brightness adjustment operation, and the like as an operation to be performed when an event (e.g., a corresponding event for each operation or for any combination of operations) is detected in the interactive area corresponding to the 3D box 2130, although it should be understood that when (or based on) an event is detected in the interactive area corresponding to the 3D box 2130, the operation to be performed is not limited thereto. As an example, the display device 2000 may display the list 2150, and the user may set execution operations corresponding to multiple events respectively by performing a drag and drop input on the items to the areas 2161, 2162, and 2163 corresponding to the multiple events respectively. It should be understood that one or more other embodiments are not limited thereto, and any manner that allows a user to select or set execution operations corresponding to multiple events respectively (e.g., a drop-down menu, a manual input area, etc.) may be implemented.

[0222] like Fig. 20 As shown in , the display device 2000 may set a power on / off operation as a first motion, a channel switching operation as a second motion, and a volume adjustment operation as a third motion.

[0223] Fig.21 The diagram illustrates the operation of the display device 2000 according to the embodiment.

[0224] The display device 2000 may determine an interactive zone 2140 corresponding to the generated 3D frame in the actual space based on the coordinate value information of the 3D frame, and monitor whether an event occurs in the interactive zone 2140 based on the event information set for the 3D frame. According to an embodiment, the display device 2000 may include a motion sensor 2170 and detect a specific motion in the interactive zone 2140 by using the motion sensor 2170. When an event (e.g., a specific motion) is detected in the interactive zone 2140, the display device 2000 may perform a corresponding operation based on the execution operation information set for the 3D frame.

[0225] For example, when a first motion (e.g., a motion of opening and then closing the hand to make a fist) is detected in the interactive area 2140, the display device 2000 may perform a first operation (e.g., being turned on or off or changing a power state). In addition, when a second motion (e.g., a motion of opening and then moving the hand up and down) is detected in the interactive area 2140, the display device 2000 may perform a second operation (e.g., switching the current channel to another channel, performing a channel switching operation, a previous channel operation, a source input change operation, etc.).

[0226] Fig. 22 is a block diagram of a display device 2000 according to an embodiment.

[0227] refer to Fig. 22 According to an embodiment, the display device 2000 may include a tuner 2040, a controller 2010, a display 2020, a communication interface 2050, a sensor 2030, a video processor 2080, an audio processor 2015, an audio output interface 2025, a power supply 2060, an input / output (I / O) interface 2070, and a memory 2090.

[0228] The tuner 2040 can tune and select only the frequency of a channel that the display device 2000 (or a user of the display device 2000) desires to receive from among many frequency components by amplifying, mixing, resonating, etc. a broadcast signal received in a wired or wireless manner. The broadcast signal includes at least one of audio, video, and additional information (e.g., an electronic program guide (EPG)).

[0229] The tuner 2040 may receive a broadcast signal in a frequency band corresponding to a channel number according to a user input (e.g., a control signal received from the control device 2200, such as a channel number input, a channel up / down input, a channel input on an EPG screen image, etc.).

[0230] The tuner 2040 may receive broadcast signals from various sources such as terrestrial broadcast stations, cable broadcast stations, satellite broadcast stations, and Internet broadcast stations. The tuner 2040 may receive broadcast signals from sources such as analog broadcast stations and / or digital broadcast stations.

[0231] The communication interface 2050 may transmit and receive data or signals to and from an external device or server under the control of the controller 2010. The controller 2010 may transmit and receive content to and from an external device connected through the communication interface 2050, download an application from an external device, and / or perform web browsing through the communication interface 2050. The communication interface 2050 may transmit and receive data or signals through at least one scheme of WLAN (e.g., Wi-Fi), Bluetooth, and wired Ethernet consistent with the performance and structure of the display device 2000.

[0232] The video processor 2080 processes the video data received by the display device 2000. The video processor 2080 may perform various types of image processing, such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion, on the video data.

[0233] The audio processor 2015 performs processing on the audio data. The audio processor 2015 can perform various processing on the audio data, such as decoding, amplification and / or noise filtering. At the same time, the audio processor 2015 can include multiple audio processing modules to process audio corresponding to multiple contents.

[0234] The audio output interface 2025 outputs audio included in a broadcast signal received through the tuner 2040 according to the control of the controller 2010. The audio output interface 2025 may output audio (e.g., voice or sound) input through the communication interface 2050 or the input / output interface 2070. Also, the audio output interface 2025 may output audio stored in the memory 2090 according to the control of the controller 2010. The audio output interface 2025 may include at least one of a speaker 2026, an earphone output terminal 2027, and a Sony / Philips digital interface (S / PDIF) 2028. The audio output interface 2025 may include a combination of the speaker 2026, the earphone output terminal 2027, and the S / PDIF output terminal 2028.

[0235] The power supply 2060 supplies power received from an external power source to components in the display device 2000 according to the control of the controller 2010. Also, the power supply 2060 may supply power output from one or more internal power sources (e.g., batteries) provided inside the display device 2000 to the components according to the control of the controller 2010.

[0236] The sensor 2030 can receive an optical signal (e.g., a control signal) received from an external control device through an optical window of a frame in the display 2020, etc. The sensor 2030 can receive an optical signal corresponding to a user input (e.g., touch, push, touch gesture, voice, or motion) from the control device 2200. The controller 2010 can extract the control signal from the received optical signal. In addition, the sensor 2030 can detect the user's voice, the user's image, and / or the user's interaction. According to an embodiment, the sensor 2030 may include a microphone configured to receive a voice issued by the user. The microphone can convert the received voice into an electrical signal and output the electrical signal to the controller 2010.

[0237] In addition, according to an embodiment, the sensor 2030 may detect a user input and transmit the detected signal to the controller 2010. For example, the sensor 2030 may detect a user input such as channel up / down, power on / off, channel selection, and / or screen image configuration from the control device 2200. In addition, according to an embodiment, the sensor 2030 may detect an input of pressing a channel up key or a channel down key included in the control device 2200, an input of long pressing the channel up key or the channel down key, an input of releasing the channel up key or the channel down key, and the like.

[0238] The sensor 2030 detects the user's voice, the user's image, and / or the user's interaction (eg, a predetermined or specific movement or pattern of movement).

[0239] The microphone 2031 receives a voice issued by a user. The microphone 2031 can convert the received voice into an electrical signal and output the electrical signal to the controller 2010. The voice can include, for example, a voice input corresponding to a menu or function of the display device 2000. The microphone 2031 can be integrated with the display device 2000 or separated therefrom. When the microphone 2031 is separated from the display device 2000, the microphone 2031 can be electrically or wirelessly connected to the display device 2000 through the communication interface 2050 or the input / output interface 2070.

[0240] It should be understood that based on the performance and structure of the display device 2000, the microphone 2031 may be omitted.

[0241] The camera 2032 receives an image (e.g., continuous frames) corresponding to the motion of the user of the gesture within the camera recognition range. Examples of the motion include the motion of a part of the user's body such as a face, a face forming an expression, a hand, a hand clenched into a fist, a finger pointing, a finger, etc. The camera 2032 may convert the received image into an electrical signal and output the electrical signal to the controller 2010 according to the control of the controller 2010.

[0242] The controller 2010 may select a menu displayed in the display device 2000 or perform a result of the recognition motion, such as adjusting a channel, adjusting a volume, moving an indicator, and / or moving a cursor, by using the result of the recognition motion.

[0243] The camera 2032 may include a lens and an image sensor. The camera 2032 may support optical zoom or digital zoom by using multiple lenses and image processing. The recognition range of the camera 2032 may be set differently based on the angle of the camera and the surrounding conditions. When the camera 2032 includes multiple cameras, a three-dimensional (3D) still image or 3D motion may be received by using the multiple cameras.

[0244] The camera 2032 may be integrated with or separated from the display device 2000. A separate device including the camera 2032 separate from the display device 2000 may be electrically or wirelessly connected to the display device 2000 through the communication interface 2050 or the I / O interface 2070.

[0245] It should be understood that based on the performance and structure of the display device 2000, the camera 2032 may be omitted.

[0246] The optical receiver 2033 receives an optical signal (e.g., a control signal) from an external control device through an optical window of the frame of the display 2020. The optical receiver 2033 may receive an optical signal corresponding to a user input (e.g., touch, press, touch gesture, voice, or motion) from the external control device. The control signal may be extracted from the optical signal according to the control of the controller 2010.

[0247] The I / O interface 2070 receives video (e.g., moving images), audio (e.g., voice or music), and / or additional information (e.g., EPG) from an external source of the display device 2000 according to the control of the controller 2010. The I / O interface 2070 may include one of an HDMI port 2071, a component jack 2072, a personal computer (PC) port 2073, and a universal serial bus (USB) port 2074. The I / O interface 2070 may include a combination of the HDMI port 2071, the component jack 2072, the PC port 2073, and the USB port 2074.

[0248] It should be understood that based on the performance and structure of the display device 2000, the I / O interface 2070 may be omitted.

[0249] The controller 2010 controls the overall operation of the display device 2000 and the signal flow between the internal components 2010 to 2090 of the display device 2000 and processes data. When the user's input exists or satisfies the previously set and stored conditions, the controller 2010 can execute the operating system (OS) and various applications stored in the memory 2090.

[0250] The controller 2010 may include a random access memory 2081 for storing signals or data received from an external source of the display device 2000 or used as a storage area for various operations performed by the display device 2000, a read-only memory (ROM) 2072 in which a control program for controlling the display device 2000 is stored, and a processor 2083.

[0251] The processor 2083 may include a graphics processing unit (GPU) 2084 for performing graphic processing on a video. The processor 2083 may be implemented as a system on chip (SoC) in which a core and the GPU 2084 are integrated.

[0252] The GPU 2084 generates a screen including various objects such as icons, images, and texts by using an operator and a renderer. The operator calculates attribute values ​​such as coordinate values, shapes, sizes, and / or colors of each object according to the layout of the screen by using a user input detected by the sensor 2030. The renderer generates a screen having any one of various layouts including objects based on the attribute values ​​calculated by the operator. The screen generated by the renderer is displayed in the display area of ​​the display 2020.

[0253] The first to nth interfaces 2085-1 to 2085-n are connected to the respective components as described above. One of the first to nth interfaces 2085-1 to 2085-n may be a network interface connected to an external device through a network.

[0254] The RAM 2081 , the ROM 2082 , the processor 2083 , the GPU 2084 , and the first to nth interfaces 2085 - 1 to 2085 - n may be connected to each other through an internal bus 2086 .

[0255] The term “controller of the display device” in the current embodiment includes the processor 2083 , the ROM 2082 , and the RAM 2081 .

[0256] The display 2020 generates a driving signal by converting an image signal, a data signal, an OSD signal, a control signal, etc. processed by the controller 2010. The display 2020 may be implemented by a PDP, an LCD, an OLED, a flexible display, etc., and may also be implemented by a 3D display. In addition, by configuring the display 2020 with a touch screen, the display 2020 may be used as an input device in addition to being used as an output device.

[0257] Under the control of the controller 2010, the memory 2090 may store various data, programs and / or applications for operating and controlling the display device 2000. The memory 2090 may include a broadcast receiving module, a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, an optical receiving module, a display control module, an audio control module, an external input control module, a power control module, a module for controlling the power of an external device connected in a wireless manner (e.g., Bluetooth), a voice database (DB) and / or a motion DB. The modules and DB of the memory 2090 may be implemented in a software format to perform a broadcast receiving control function, a channel control function, a volume control function, a communication control function, a voice recognition function, a motion recognition function, an optical receiving control function, a display control function, an audio control function, an external input control function, a power control function and / or a function of controlling an external device connected in a wireless manner (e.g., Bluetooth). The controller 2010 may perform each function by using the software stored in the memory 2090.

[0258] Fig. 22 The block diagram of the display device 2000 shown in the figure is merely illustrative. Each component in the block diagram can be integrated, added, or omitted according to the specification or design of the display device 2000 actually implemented. That is, two or more components can be integrated into one component, one component can be separated into two or more components, and / or one or more components can be omitted, depending on the situation. In addition, the functions performed in each frame will describe the embodiment, and its detailed operation or device is not limited.

[0259] The operation method of the electronic device according to the embodiment can be implemented as a computer command that can be executed by various computer devices and recorded on a non-transitory computer-readable recording medium. The non-transitory computer-readable recording medium may include program commands, data files, and data structures adopted individually or in combination. The program commands recorded on the non-transitory computer-readable recording medium can be specially designed and constructed for the present disclosure or can be understood and used by those of ordinary skill in the art for the field of computer software. Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, or tapes, optical media such as compact disk read-only memories (CD-ROMs), or digital versatile disks (DVDs), magneto-optical media such as optically readable disks, and hardware devices such as ROMs, RAMs, or flash memories that are specially configured to store and execute program commands. Examples of program commands include high-level programming language codes that can be executed by a computer using an interpreter and machine language codes made by a compiler.

[0260] Furthermore, the electronic device and / or the operating method of the electronic device according to the embodiment may be provided by including it in a computer program product. The computer program product may be traded between a seller and a buyer.

[0261] The computer program product may include a software (S / W) program, a non-transitory computer-readable storage medium storing the S / W program. For example, the computer program product may include a S / W program form (e.g., a downloadable application) of a product electronically distributed by a manufacturer of an electronic device or an electronic market (e.g., GOOGLE PLAYSTORE or App store). For electronic distribution, at least a portion of the S / W program may be stored on a storage medium or at least a portion of the S / W program may be temporarily generated. In this case, the storage medium may be included in a server of the manufacturer, a server of the electronic market, or a relay station server configured to temporarily store the S / W program.

[0262] In a system including a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, when a third device (e.g., a smart phone) connected to the server or the client device by communication exists, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include an S / W program to be transmitted from the server to the client device or the third device or from the third device to the client device.

[0263] In this case, one of the server, the client device, and the third device can execute the computer program product and perform the method according to the embodiment. Alternatively, two or more of the server, the client device, and the third device can execute the computer program product and perform the method according to the embodiment in a distributed manner.

[0264] For example, a server (e.g., a cloud server, an artificial intelligence server, etc.) may execute a computer program product stored therein to control a client device connected to the server via communication so that the client device executes a method according to an embodiment.

[0265] The electronic device according to the embodiment can improve the use convenience of the user by providing a user interface through which the user can easily set the interactive zone.

[0266] The electronic device according to the embodiment can improve the use convenience of the user by providing a user interface through which the user can easily set or change the content corresponding to the interactive area.

[0267] It should be understood that the embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment should typically be considered as available for other similar features or aspects in other embodiments.

[0268] While embodiments have been described with reference to the drawings, those skilled in the art will understand that various changes in form and details may be made therein without departing from the spirit and scope as defined, at least in part, by the following claims.

Claims

1. An electronic device, include: monitor; User input interface; a memory configured to store one or more instructions; as well as at least one processor configured to execute one or more instructions stored in the memory to: receiving, from a display device, an image of the actual space to be monitored captured by the display device, controlling displaying of the captured image on the display, determining a three-dimensional 3D frame according to a shape and a size of at least one object included in the captured image based on a depth value of the at least one object, adjusting at least one of a size and a position of the 3D box based on a user input received through the user input interface, Setting an interactive area corresponding to the 3D frame in the actual space; as well as Content to be executed or output in the display device is set based on detection of an event in an interactive area corresponding to the 3D frame.

2. The electronic device according to claim 1, in: The user input interface is configured to receive an input for selecting at least one object included in the captured image displayed on the display.

3. The electronic device according to claim 1, in: The captured image displayed on the display includes an image obtained by three-dimensionally rendering a captured image of a real space; and The at least one processor is further configured to execute the one or more instructions to adjust a viewing angle of the captured image displayed on the display based on a dragging user input on the captured image displayed on the display.

4. The electronic device according to claim 1, in, The at least one processor is further configured to execute the one or more instructions to: Controlling display of a background image lock icon on the display; as well as A viewing angle of the captured image displayed on the display is fixed based on a user input for selecting the background image lock icon.

5. The electronic device according to claim 1, in, The at least one processor is further configured to execute the one or more instructions to control display of the 3D frame and display a horizontal / vertical size adjustment object for adjusting the size of the 3D frame and a rotation adjustment object for rotating the 3D frame relative to at least one surface of multiple surfaces of the displayed 3D frame.

6. The electronic device according to claim 1, in, The at least one processor is further configured to execute the one or more instructions to control setting of sub-interaction zones respectively corresponding to the plurality of contents in the actual space based on the contents set for the interaction zone including the plurality of contents.

7. The electronic device according to claim 1, in, The at least one processor is further configured to execute the one or more instructions to set at least one piece of tag information corresponding to the 3D frame, and control transmission of the at least one piece of tag information to an external device based on detecting an event in the interactive area.

8. A method for operating an electronic device, the operating method include: receiving, from a display device, an image of the actual space to be monitored captured by the display device, Controlling display of captured images on a display; determining a three-dimensional 3D frame according to a shape and a size of at least one object included in the captured image based on a depth value of the at least one object; adjusting at least one of a size and a position of the 3D box based on user input; Setting an interactive area corresponding to the 3D frame in the actual space; as well as Content to be executed or output in the display device is set based on detection of an event in an interactive area corresponding to the 3D frame. 9 . The operating method according to claim 8 , further comprising receiving an input for selecting at least one object included in the captured image displayed on the display.

10. The operating method according to claim 8, in: The captured image displayed on the display includes an image obtained by three-dimensionally rendering a captured image of a real space; and The operating method further includes adjusting a viewing angle of the captured image displayed on the display based on a user input of a drag on the captured image displayed on the display.

11. The operating method according to claim 8, further comprising: include: Controlling display of a background image lock icon on the display; as well as A viewing angle of the captured image displayed on the display is fixed based on a user input for selecting the background image lock icon.

12. The operating method according to claim 8, further comprising displaying the 3D frame, and displaying a horizontal / vertical size adjustment object for adjusting a size of the 3D frame and a rotation adjustment object for rotating the 3D frame with respect to at least one of a plurality of surfaces of the displayed 3D frame.