Electronic device and operating method thereof

Abstract

An electronic device and an operation method of the electronic device are provided. The electronic device includes a communication interface, a display, a memory storing one or more instructions, and at least one processor configured to execute the one or more instructions stored in the memory. The at least one processor may display content including the avatar on the display by executing the one or more instructions stored in the memory. The at least one processor may display a virtual object management user interface (UI) for managing a virtual object corresponding to an Internet of Things (IoT) device around the avatar by executing one or more instructions stored in the memory. The at least one processor may display an activated virtual object management UI based on a user's input to the virtual object management UI by executing the one or more instructions stored in the memory. When the virtual object management UI includes a notification UI, the at least one processor may obtain state information from the IoT device through the communication interface by executing one or more instructions stored in the memory.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and an operating method of the electronic device, and more particularly, to an electronic device capable of displaying a virtual object in a virtual space and an operating method of the electronic device. Background Art

[0002] Recently, there has been a growing interest in next-generation media environments that allow users to experience content in a virtual space similar to the real space. Specifically, the metaverse has attracted attention as a representative service that provides users with a virtual space. The metaverse, which is a compound word of meta (meaning fictional or abstract) and universe (meaning the real world), refers to a three-dimensional (3D) virtual world. The core technology of this metaverse is extended reality (XR) technology, which covers virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0003] There are various methods of implementing virtual space, but they all generally use virtual 3D images to interact with users in real space in real time.

[0004] Due to the development of information and communication technology, Internet of Things (IoT) technology is becoming more and more popular, and therefore, the number of IoT devices that need to be controlled is also increasing. According to conventional IoT device control methods, users generally use a user interface (UI) provided mainly through an application or a network.

[0005] As the metaverse environment expands, users will be able to perform various activities through avatars projected into virtual space. Therefore, technology that can control IoT devices in virtual space is required. Summary of the invention

[0006] Solution to the problem

[0007] According to one aspect of the present disclosure, an electronic device includes a communication interface, a display, a memory storing one or more instructions, and at least one processor configured to execute the one or more instructions stored in the memory. The at least one processor can display content including a virtual image on the display by executing one or more instructions stored in the memory. The at least one processor can display a virtual object management user interface (UI) for managing virtual objects corresponding to an Internet of Things (IoT) device around the virtual image by executing one or more instructions stored in the memory. The at least one processor can display an activated virtual object management UI based on user input to the virtual object management UI by executing one or more instructions stored in the memory. When the virtual object management UI includes a notification UI, the at least one processor can obtain status information from the IoT device through the communication interface by executing one or more instructions stored in the memory.

[0008] According to one aspect of the present disclosure, a server device includes a communication interface, a memory storing one or more instructions, and at least one processor configured to execute one or more instructions stored in the memory. The at least one processor can send content including a virtual image to an electronic device through the communication interface by executing one or more instructions. The at least one processor can generate a virtual object management UI for managing virtual objects corresponding to an IoT device around the virtual image by executing one or more instructions. The at least one processor can activate the virtual object management UI based on an input signal for the virtual object management UI by executing one or more instructions.

[0009] According to one aspect of the present disclosure, an operating method of an electronic device includes: displaying content including a virtual image on a display, displaying a virtual object management UI for managing virtual objects corresponding to an IoT device around the virtual image, displaying an activated virtual object management UI based on user input to the virtual object management UI, and when the virtual object management UI includes a notification UI, obtaining status information from the IoT device through a communication interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram showing a virtual space providing system reflecting a real space according to an embodiment.

[0011] Figure 2 is a flowchart illustrating an operation performed by an electronic device to provide a virtual space reflecting a real space according to an embodiment.

[0012] Figure 3 is a diagram illustrating an operation performed by an electronic device to provide a virtual space reflecting a real space according to an embodiment.

[0013] Figure 4 is a flowchart illustrating an operation of a notification user interface (UI) displaying status information about an Internet of Things (IoT) device, performed by a virtual space providing system according to an embodiment.

[0014] Figure 5 is a diagram illustrating an operation of displaying a notification UI of status information about an IoT device, performed by a virtual space providing system according to an embodiment.

[0015] Figure 6a is a diagram illustrating an operation of displaying a notification UI of status information about an IoT device, performed by a virtual space providing system according to an embodiment.

[0016] Figure 6b is a diagram illustrating an operation of displaying a notification UI of status information about an IoT device, performed by a virtual space providing system according to an embodiment.

[0017] Figure 7 is a flowchart of an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0018] Figure 8 is a flowchart of an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0019] Fig. 9 is a diagram illustrating an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0020] Fig.10 is a diagram illustrating an operation of displaying a virtual object list UI performed by a virtual space providing system according to an embodiment.

[0021] Fig.11 is a diagram illustrating an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0022] Fig.12 is a diagram illustrating an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0023] Fig.13 is a block diagram of a structure of an electronic device according to an embodiment.

[0024] Fig.14 is a detailed block diagram of a structure of an electronic device according to an embodiment.

[0025] Fig.15 is a block diagram of a structure of a server device according to an embodiment.

[0026] Fig.16 is a block diagram of a structure of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0027] Throughout this disclosure, the expression “at least one of a, b, or c” refers to only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0028] The embodiments of the present disclosure are described in detail with reference to the accompanying drawings so that the present disclosure can be easily performed by those skilled in the art. However, the present disclosure can be embodied in many different forms and should not be construed as being limited to the examples set forth herein.

[0029] Although general terms currently widely used are selected to describe the present disclosure in consideration of the functions of the present disclosure, these general terms may vary according to the intention of ordinary technicians in the field, case precedents, the emergence of new technologies, etc. Therefore, the terms must be defined based on their meanings and the contents of the entire specification rather than by simply declaring the terms.

[0030] The terms used in the present disclosure are for describing specific embodiments only and are not intended to limit the scope of the present disclosure.

[0031] When used in this specification, the terms "include" and / or "comprise" or "have" and / or "contain" indicate the presence of the elements, but do not exclude the presence or addition of one or more other elements. When used in this specification, the terms "unit", "...machine" and "module" refer to a unit that performs at least one function or operation, and can be implemented as hardware, software, or a combination of hardware and software.

[0032] The present disclosure will now be described more fully with reference to the accompanying drawings, in which example embodiments of the present disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. In the accompanying drawings, portions not relevant to the description are omitted for simplicity of illustration, and throughout the present invention, the same reference numerals represent the same elements.

[0033] The term "user" as used herein refers to a person who controls a system, function, or operation. Examples of users may include an inventor, an administrator, or an installation engineer.

[0034] Embodiments of the present disclosure will now be described more fully with reference to the accompanying drawings.

[0035] Figure 1 is a diagram showing a virtual space providing system reflecting a real space according to an embodiment.

[0036] Reference Figure 1 , a virtual space providing system according to an embodiment may include an electronic device 100 , a server device 200 , and an Internet of Things (IoT) device 300 .

[0037] The electronic device 100 may be an electronic device capable of outputting an image. According to an embodiment, the electronic device 100 may be implemented as any of various types of electronic devices including a display. The electronic device 100 may be fixed or movable, and may be, but is not limited to, a digital TV capable of receiving digital broadcasting.

[0038] The electronic device 100 may include at least one of the following: a desktop personal computer (PC), a smart phone, a tablet PC, a mobile phone, a video phone, an e-book reader, a laptop PC, a netbook computer, a digital camera, a personal digital assistant (PDA), a portable multimedia player (PMP), a video camera, a navigation wearable device, a smart watch, a home network system, a security system, or a medical device.

[0039] The electronic device 100 may be implemented not only as a flat display device but also as a curved display device having a curvature or a flexible display device having an adjustable curvature. The output resolution of the electronic device 100 may include any of various resolutions such as high definition (HD), full HD, ultra HD, or a resolution clearer than ultra HD.

[0040] According to an embodiment, the electronic device 100 may output various types of content. The electronic device 100 may receive and output virtual space content provided by the server device 200. For example, the server device 200 may generate virtual space content and send the virtual space content to the electronic device 100 through a communication network. For example, the server device 200 may generate a virtual image 20 corresponding to a user of the electronic device 100 in the virtual space content and send the virtual image 20 to the electronic device 100. The electronic device 100 may display the virtual space content including the virtual image 20 corresponding to the user on a display. Alternatively, the electronic device 100 according to the embodiment may generate and output virtual space content by executing an application installed in the electronic device 100.

[0041] The server device 200 generates virtual space content and provides the virtual space content so that users of various clients can access the virtual space content. The server device 200 can generate and provide virtual images reflecting users of various clients. The server device 200 can provide virtual space content to the electronic device 100 as an example of a client, and can manage the coordinates of objects (e.g., virtual images) in the virtual space in response to the interaction of the user of the electronic device 100. Therefore, users in the real space and objects in the virtual space can interact with each other.

[0042] The IoT device 300 may be a general-purpose device (or thing) that performs wired or wireless communication. For example, the IoT device 300 may include a refrigerator, an air conditioner, a CCTV, a TV, a washing machine, a vacuum cleaner, an oven, a dehumidifier, a lamp, a fire alarm, and a temperature or humidity sensor. However, the embodiment is not limited thereto. The IoT device 300 may include a plurality of IoT devices, including a first IoT device 310, a second IoT device 320, a third IoT device 330, and the like. In the following, the IoT device 300 is used to indicate that it includes at least one IoT device.

[0043] According to an embodiment, the IoT device 300 may be registered in the server device 200 by sending the device information of the IoT device 300 and the account information of the user to the server device 200. For example, the device information of the IoT device 300 may include a model name, a serial number, a control function, etc. of the IoT device 300. The model name and serial number of the IoT device 300 may be information for identifying one of the appearance, type, and control function of the IoT device 300.

[0044] According to an embodiment, the server device 200 may generate the IoT device 300 in the real space as a virtual object 10 in the virtual space. The server device 200 may send the generated virtual object 10 to the electronic device 100, and the electronic device 100 may display the virtual object 10 together with the virtual space content. For example, the server device 200 may send the virtual object 10 corresponding to the IoT device 300 to the electronic device 100, which is connected to the same account as the IoT device 300 registered in the server device 200.

[0045] In this case, the virtual object 10 may have the appearance and / or control functions of the IoT device 300. The virtual object 10 may have the appearance of replicating the IoT device 300 by considering the device information of the IoT device 300, such as the model name and the serial number, and may also have an appearance predetermined according to the type of the IoT device 300. Therefore, the virtual object 10 may be referred to as a replica or twin of the actual IoT device because the virtual object 10 has an appearance or function similar to that of the actual IoT device. For example, the virtual object 10 may have the appearance and / or control functions of a refrigerator, an air conditioner, a CCTV, a TV, a washing machine, a vacuum cleaner, an oven, a dehumidifier, a lamp, a fire alarm, a temperature or humidity sensor, and the like.

[0046] According to an embodiment, the electronic device 100 may display a virtual object 10 located in a virtual space. The virtual object 10 may include a plurality of virtual objects, including a first virtual object 11, a second virtual object 12, a third virtual object 13, etc. Hereinafter, the virtual object 10 is used to indicate that it includes at least one virtual object.

[0047] According to an embodiment, the electronic device 100 may display virtual object management user interfaces (UIs) 30 and 40 around the avatar 20. The virtual object management UIs 30 and 40 may include a notification UI 40 indicating notifications about status information of the IoT device 300 and a control UI 30 receiving input from a user who controls the IoT device 300.

[0048] According to an embodiment, the electronic device 100 may display a notification UI 40 about the status information of the IoT device 300 on the display. The user may monitor the status information of the IoT device 300 in the real space through the electronic device 100. For example, the IoT device 300 may send device information and status information to the electronic device 100. The IoT device 300 may be connected to the electronic device 100 in a wireless or wired manner, or may be connected through the server device 200. For example, the electronic device 100 may obtain device information and status information from the IoT device 300, and send the device information and status information to the server device 200. The server device 200 may receive the device information and status information of the IoT device 300, and generate a virtual object 10 having the appearance and / or control function of the IoT device 300. The server device 200 may generate the notification UI 40 as a virtual object management UI for managing the virtual object 10 corresponding to the IoT device 300. The server device 200 may generate the notification UI 40 around the virtual image 20. The server device 200 may send the virtual image 20 and the notification UI 40 around the virtual image 20 while sending the virtual space content to the electronic device 100. Therefore, the electronic device 100 may display the virtual space content, the avatar 20 , and the notification UI 40 around the avatar 20 on the display.

[0049] According to an embodiment, the electronic device 100 may display a control UI 30 for controlling the IoT device 300 on a display. The user may control the IoT device 300 in the real space through both the virtual object 10 displayed on the electronic device 100 and the UI related to the virtual object 10. For example, the electronic device 100 may display the control UI 30 around the virtual image 20. The user may input the content to be controlled by the IoT device 300 through the control UI 30 and how to set the operation of the IoT device 300, and the electronic device 100 may receive an input signal about the information of the IoT device 300 and the operation information of the IoT device 300. The electronic device 100 may send the information of the IoT device 300 and the setting operation of the IoT device 300 to the server device 200. The server device 200 may generate a control command capable of controlling the IoT device 300 according to the information of the IoT device 300 and the setting operation of the IoT device 300. The server device 200 may send the control command to the IoT device 300 via the electronic device 100, or may send the control command directly to the IoT device 300.

[0050] According to an embodiment, the server device 200 is shown as a device that manages the IoT device 300 while providing virtual space content to the electronic device 100. However, the embodiment is not limited thereto. For example, the server device 200 may include a first server device that provides virtual space content to the electronic device 100, and a second server device that manages the IoT device 300.

[0051] Figure 2 is a flowchart illustrating an operation performed by an electronic device to provide a virtual space reflecting a real space according to an embodiment.

[0052] Reference Figure 2 In operation S210, the electronic device 100 according to an embodiment may display content including a virtual image on a display. For example, the electronic device 100 may display virtual space content.

[0053] In operation S220, the electronic device 100 according to the embodiment may display a virtual object management UI for managing virtual objects corresponding to the IoT device 300 around the avatar. For example, the electronic device 100 may display virtual objects corresponding to the IoT device 300 on the display. For example, the virtual object management UI may include a notification UI for notifying status information of the IoT device 300, or may include a control UI for controlling the IoT device 300. The electronic device 100 may display the notification UI or the control UI around the avatar.

[0054] In operation S230, the electronic device 100 according to the embodiment may display an activated virtual object management UI based on the user's input to the virtual object management UI. For example, the inactivated virtual object management UI may have a first state, and the activated virtual object management UI may have a second state. For example, when the electronic device 100 obtains the user's input to the virtual object management UI, the electronic device 100 may display the virtual object management UI displayed in the first state as a second state. For example, the first state may be a translucent state, a black and white state, a dotted line state, a thin line state, etc., and the second state may be an opaque state, a color state, a solid line state, a thick line state, etc.

[0055] The electronic device 100 according to an embodiment may display virtual space content, and may change and display a virtual object management UI in real time through interaction with a user.

[0056] The electronic device 100 according to the embodiment may display the state information of the IoT device 300 by reflecting the state information of the IoT device 300 in the virtual object, or may control the IoT device 300 by reflecting the control input for the virtual object in the IoT device 300 .

[0057] The electronic device 100 according to the embodiment may conveniently manage virtual objects of a virtual space and IoT devices of a real space by manipulating a virtual object management UI displayed around an avatar corresponding to a user, without moving the avatar to a location where the virtual object is located.

[0058] Figure 3 is a diagram illustrating an operation performed by an electronic device to provide a virtual space reflecting a real space according to an embodiment.

[0059] Will combine Figure 2 and Figure 3 An example in which the electronic device 100 according to the embodiment provides a virtual space reflecting a real space is described. The IoT device 300 in the real space is shown as an oven.

[0060] The electronic device 100 according to the embodiment may display, on a display, virtual space content 301 including a virtual avatar 320. The virtual space content 301 may be, for example, a kitchen space.

[0061] For example, the electronic device 100 may display the virtual object 10 corresponding to the oven as the IoT device 300 on the display together with the virtual space content 301. The virtual object 10 may be an oven reflecting the appearance, type, and / or control function of the IoT device 300.

[0062] The server device 200 according to the embodiment may store a list of multiple IoT devices and device information (e.g., model name and serial number) of the IoT device 300 in a memory. For example, the server device 200 may generate virtual object data taking into account the appearance, type, and control function of the IoT device 300 based on the device information of the IoT device 300, and may store the virtual object data in the memory. For example, the server device 200 may include a list of multiple IoT devices registered using the same account as the account of the electronic device 100 and the device information of the IoT device 300, and may have virtual object data reflecting the appearance, type, and control function of the IoT device 300 for generating a virtual object.

[0063] The server device 200 according to the embodiment may generate a virtual object 10 corresponding to the IoT device 300 based on the device information and virtual object data of the IoT device 300. For example, the server device 200 may generate a virtual object 10 having the appearance, type, and control function of the IoT device 300 based on the device information and virtual object data of the IoT device 300.

[0064] The server device 200 according to the embodiment may generate a virtual object that replicates the IoT device 300 in consideration of the device information (e.g., model name and serial number) of the IoT device 300. For example, when the server device 200 has virtual object data that matches the device information of the IoT device 300, the server device 200 may generate a virtual object that reflects the appearance, type, and control function of the actual IoT device 300.

[0065] The server device 200 according to the embodiment may generate a virtual object having a predetermined representative appearance according to the type of the IoT device 300. For example, when the server device 200 does not have virtual object data matching the device information of the IoT device 300, the server device 200 may generate a virtual object reflecting a representative appearance and a representative control function according to the type of the IoT device 300.

[0066] For example, the electronic device 200 may have virtual object data matching the model name of an oven as the IoT device 300. The server device 200 may generate a virtual oven 10 reflecting the same appearance and cooking function as the oven as the IoT device 300.

[0067] The server apparatus 200 according to an embodiment may generate virtual object management UIs 330 and 340 for managing the virtual object 10 corresponding to the IoT device 300 .

[0068] The electronic device 100 according to an embodiment may display virtual object management UIs 330 and 340 for managing a virtual object 10 (e.g., an oven) around the avatar 320. For example, the electronic device 100 may display a notification UI 340 reflecting the virtual object 10 at the upper right of the avatar 320, and may display a control UI 330 for controlling the virtual object 10 at the upper left of the avatar 320. For example, the notification UI 340 may be a two-dimensional (2D) or three-dimensional (3D) object having the appearance of the virtual object 10. Figure 3 , the notification UI 340 is shown as a 2D icon having the appearance of the virtual object 10.

[0069] As described later, the electronic device 100 according to the embodiment may display an activated virtual object management UI based on a user input to the virtual object management UI. For example, the notification UI 340 and the control UI 330 may be displayed in a first state (e.g., a thin line state) until a user input is received, and then displayed in a second state (e.g., a thick line state) in response to the user input.

[0070] For example, the electronic device 100 may receive status information from the IoT device 300, and may display a notification UI 340 notifying the status information of the IoT device 300. The electronic device 100 may display a virtual object 10 reflecting the status information of the IoT device 300. For example, the electronic device 100 may receive a control input for the virtual object 10 through the control UI 330, and may send a control command to the IoT device 300.

[0071] Now refer to Figures 4 to 8 An operation of displaying a notification UI in a virtual object management UI, performed by the electronic device 100 according to an embodiment, is described.

[0072] Figure 4 is a flowchart illustrating an operation of displaying a notification UI of status information about an IoT device, performed by a virtual space providing system according to an embodiment.

[0073] Reference Figure 4 , in operation S410 , the IoT device 300 according to the embodiment may identify state information of the IoT device 300 .

[0074] According to an embodiment, the status information of the IoT device 300 may include the power-on / off status, operation mode, operation completion, operation start, pause, operation resumption, etc. of the IoT device 300. For example, when the IoT device 300 is an oven, the status information of the IoT device 300 may include cooking completion information.

[0075] In operation S420 , the electronic device 100 according to an embodiment may obtain device information and status information of the IoT device 300 .

[0076] According to an embodiment, the electronic device 100 may request status information from the IoT device 300. According to an embodiment, the electronic device 100 may obtain device information and status information of the IoT device 300 from the IoT device 300 based on the request to the IoT device 300. For example, the electronic device 100 may directly communicate with the IoT device 300 through a communication interface, or may indirectly communicate with the IoT device 300 through the server device 200.

[0077] According to an embodiment, the electronic device 100 may request the IoT device 300 for status information every few seconds, every few minutes, or at a preset time interval. According to an embodiment, the IoT device 300 may send status information to the electronic device 100.

[0078] According to an embodiment, the device information of the IoT device 300 may include information for identifying the appearance, type, control function, etc. of the IoT device 300. For example, the device information of the IoT device 300 may include the model name, serial number, control function, etc. of the IoT device 300. For example, when the IoT device 300 is an oven, the device information of the IoT device 300 may include the model name, serial number, and control function (e.g., cooking function) of the oven.

[0079] In operation S430 , the electronic device 100 according to an embodiment may transmit device information and state information of the IoT device 300 to the server device 200 .

[0080] According to an embodiment, when it is determined that the status information received from the IoT device 300 is information that needs to be notified, the electronic device 100 may send the status information to the server device 200. For example, when the status information received from the IoT device 300 is operation completion information, the electronic device 100 may send the operation completion information to the server device 200. For example, when the IoT device 300 is an oven, the electronic device 100 may send the cooking completion information of the IoT device 300 to the server device 200. For example, when the status information received from the IoT device 300 is operation progress information, the electronic device 100 may not send the operation progress information to the server device 200.

[0081] According to an embodiment, the electronic device 100 may transmit the device information of the IoT device 300 together with the status information thereof to the server device 200 .

[0082] In operation S440 , the server apparatus 200 according to an embodiment may generate a notification UI around the avatar.

[0083] According to an embodiment, the server apparatus 200 may receive device information and status information of the IoT device 300 from the electronic device 100. The server apparatus 200 may generate a notification UI based on the device information and status information of the IoT device 300, which is a virtual object management UI.

[0084] According to an embodiment, the server device 200 may store, for example, a list of IoT devices, device information of the IoT devices (e.g., model name and serial number), and virtual object data corresponding to the IoT devices. For example, the server device 200 may generate virtual object data taking into account the appearance, type, and control function of the IoT device 300 based on the device information of the IoT device 300, and may store the virtual object data in a memory.

[0085] According to an embodiment, the server device 200 may receive the device information of the IoT device 300 and match the virtual object data stored in the server device 200 with the device information. According to an embodiment, the server device 200 may generate a virtual object corresponding to the IoT device 300 based on the device information of the IoT device 300 and the virtual object data matched with the device information. For example, the server device 200 may generate a virtual object having the appearance, type, and control function of the IoT device 300 based on the device information and the virtual object data.

[0086] According to an embodiment, the server device 200 may receive status information of the IoT device 300 from the electronic device 100, and may generate a notification UI notifying the status information of the IoT device 300. According to an embodiment, the server device 200 may generate a notification UI through a virtual object based on the device information of the IoT device 300.

[0087] According to an embodiment, the notification UI may be generated in any of the following forms: a 2D object form reflecting the appearance of the IoT device 300, a 2D object form reflecting a representative appearance according to the type of the IoT device 300, and a 3D object form modeling the IoT device 300. For example, the server device 200 may generate a virtual object icon having the appearance of the IoT device 300 (see Figure 6b ), a virtual object icon having a representative appearance according to the type of the IoT device 300 (see Figure 5 ), or a virtual object icon modeling the IoT device 300 (see Figure 6a ).

[0088] According to an embodiment, the server device 200 may generate a notification UI around the avatar. According to an embodiment, the server device 200 may obtain, for example, the coordinates of the avatar of the virtual space and the coordinates of the background. According to an embodiment, the server device 200 may generate a notification UI by, for example, the coordinates of the avatar of the virtual space and the coordinates of the background so that the notification UI is located around the avatar.

[0089] According to an embodiment, the server device 200 may generate a notification UI around the avatar, the notification UI being a virtual object icon in the form of a 2D object (see Figure 5 and Figure 6b According to an embodiment, the server device 200 may generate a notification UI modeled as a 3D object around the avatar (see Figure 6a ).

[0090] According to an embodiment, the server device 200 may present a notification UI, which is a virtual object management UI. According to an embodiment, the server device 200 may send the presented notification UI to the electronic device 100. For example, the server device 200 may perform a calculation for generating a notification UI, and may send a result screen image including the presented notification UI to the electronic device 100 in a streaming manner.

[0091] In operation S450 , the electronic device 100 according to an embodiment may display a notification UI notifying status information of the IoT device 300 around the avatar.

[0092] According to an embodiment, the electronic device 100 may receive a presentation result screen including the generated notification UI from the server device 200 in a streaming manner. According to an embodiment, the electronic device 100 may display virtual space content including the notification UI received from the server device 200.

[0093] According to an embodiment, the electronic device 100 may display a notification UI in a first state around the avatar. For example, the first state may be an inactive state, and the notification UI may be in a semi-transparent state, a black and white state, a dotted line state, or a thin line state.

[0094] According to an embodiment, the electronic device 100 may display which IoT device 300 is notified, and may not display detailed status information of the IoT device 300. For example, according to a user input selecting the notification UI, the detailed status information may appear in the detailed notification UI. According to an embodiment, the electronic device 100 may use a virtual object based on the device information of the IoT device 300 to intuitively display which IoT device 300 is notified.

[0095] According to an embodiment, the electronic device 100 may display a notification UI through a virtual object based on the device information of the IoT device 300. For example, the electronic device 100 may display which IoT device 300 is notified through a virtual object reflecting the appearance, type, and / or control function of the IoT device 300. For example, the electronic device 100 may display a notification UI in which the appearance, type, and / or control function of the IoT device 300 has been reflected.

[0096] According to an embodiment, the electronic device 100 may explicitly notify the UI in any of the following forms: a 2D object form having the appearance of the IoT device 300, a 2D object form having a representative appearance according to the type of the IoT device 300, and a 3D object form modeling the IoT device 300. For example, the notification UI may be displayed as a virtual object icon having the appearance of the IoT device 300 (see Figure 6b), a virtual object icon having a representative appearance according to the type of the IoT device 300 (see Figure 5 ), or a virtual object icon modeling the IoT device 300 (see Figure 6a ).

[0097] In operation S460, the electronic device 100 according to the embodiment may receive a user input selecting the notification UI. The electronic device 100 according to the embodiment may transmit an input signal to the server device 200 according to the user input selecting the notification UI.

[0098] The electronic device 100 according to an embodiment may receive a user input through an input interface (eg, a remote controller or a touch screen).

[0099] In operation S470, the server apparatus 200 according to the embodiment may activate the notification UI based on an input signal with respect to the notification UI.

[0100] In the present disclosure, the notification UI being activated means that the notification UI changes from a first state to a second state. The second state may be an activated state, and the notification UI may be in an opaque state, a colored state, a solid line state, a thick line state, etc. The first state and the second state may be different from each other.

[0101] In operation S480, the server apparatus 100 according to the embodiment may generate a detailed notification UI based on the state information of the IoT device 300. The server apparatus 200 according to the embodiment may generate a detailed notification UI based on a user input for the notification UI.

[0102] According to an embodiment, the detailed notification UI may include a message about the status information of the IoT device 300. For example, the detailed notification UI may include a message about the cooking completion information of the oven, which is the IoT device 300.

[0103] According to an embodiment, the server device 200 may present an activated notification UI and a detailed notification UI. According to an embodiment, the server device 200 may transmit the presented notification UI and the presented detailed notification UI to the electronic device 100. For example, the server device 200 may perform calculations for generating an activated notification UI and a detailed notification UI, and may transmit virtual space content including the notification UI and the detailed notification UI to the electronic device 100.

[0104] In operation S490, the electronic device 100 according to an embodiment may display the activated notification UI and the detailed notification UI.

[0105] According to an embodiment, the electronic device 100 may display the notification UI in the second state around the avatar.

[0106] According to an embodiment, the electronic device 100 may display a detailed notification UI including a message about status information of the IoT device 300. For example, the electronic device 100 may display a message about cooking completion information of an oven, which is the IoT device 300.

[0107] According to an embodiment of the present disclosure, the electronic device 100 may display a notification UI notifying status information of the IoT device 300 around the avatar.

[0108] According to an embodiment of the present disclosure, a user can conveniently identify the status information of the IoT device 300 by selecting the notification UI displayed around the avatar. For example, when a user performs a content activity through the avatar in the virtual space content, the user can identify the status information of the IoT device 300 through the notification UI displayed around the avatar. For example, even when the user does not manipulate the avatar so that the avatar moves to the location where the virtual object corresponding to the IoT device 300 is located, the user can identify the status information of the IoT device 300 by interacting with the notification UI displayed around the avatar.

[0109] Figure 5 is a diagram illustrating an operation of displaying a notification UI of status information about an IoT device, performed by a virtual space providing system according to an embodiment.

[0110] Reference Figure 5 , the electronic device 100 according to the embodiment may receive the virtual space content 501: 501A, 501B, and 501C executed in the server device 200, and may display the virtual space content 501: 501A, 501B, and 501C on the display. The electronic device 100 according to the embodiment may display the virtual image 510 in the virtual space content 501. The server device 200 according to the embodiment may generate a virtual object in the virtual space content 501 corresponding to the IoT device 300 in the real space. Figure 5 The IoT device 300 is shown to be an oven.

[0111] exist Figure 5, the electronic device 100 may represent a state 501A in which the control UI 520 is displayed around the avatar 510, a state 501B in which the control UI 520 and the inactivated notification UI 530 are displayed around the avatar 510, and a state 501C in which the control UI 520 and the activated notification UI 531 are displayed around the avatar 510 and the detailed notification UI 540 is displayed. The state 501A is a general state in which the avatar 510 performs a content activity. The state 501B is a notification state for notifying the state information 502 of the IoT device 300. The state 501C is a detailed notification state for notifying the state information 502 of the IoT device 300 in detail.

[0112] The electronic device 100 according to the embodiment may periodically read the status information 502 of the IoT device 300, and when there is a completed task, a notification UI 530 notifying the status information 502 of the IoT device 300 may be displayed. The control UI 520 for controlling the IoT device 300 may always be displayed around the avatar 510. This will be described in the following Figure 7 Described in.

[0113] In state 501A, the electronic device 100 according to an embodiment may perform a content activity through the avatar 510 displayed in the virtual space content 501A.

[0114] The IoT device 300 according to the embodiment may identify the state information 502. For example, the IoT device 300 may identify the cooking completion information. The operation of the IoT device 300 may be preset by the user. For example, the user may set the cooking time of the oven and may perform a cooking action.

[0115] The electronic device 100 according to the embodiment may obtain device information and status information 502 of the IoT device 300 according to a request.

[0116] The electronic device 100 according to the embodiment may transmit the device information and the state information 502 of the IoT device 300 to the server device 200. For example, the electronic device 100 may transmit the device information and the cooking completion information of the IoT device 300 to the server device 200. For example, when the state information received from the IoT device 300 is the operation progress information, the electronic device 100 may not transmit the operation progress information to the server device 200.

[0117] The server device 200 according to the embodiment may generate a notification UI 530 around the avatar 510 based on the device information of the IoT device 300. For example, the notification UI 530 may be generated in the form of an icon representing a virtual object. For example, the notification UI 530 may be generated in the form of an icon having the appearance of an oven.

[0118] The server device 200 according to the embodiment may transmit the virtual space content 501B in which the notification UI 530 is generated to the electronic device 100 in a streaming manner.

[0119] In state 501B, the electronic device 100 according to the embodiment may display the notification UI 530 around the avatar according to the change of the state information of the IoT device 300. The electronic device 100 according to the embodiment may display the virtual space content 501B including the notification UI 530 received from the server device 200.

[0120] In state 501B, the electronic device 100 according to the embodiment may display a notification UI 530 having a representative appearance according to the type of the IoT device 300 based on the device information of the IoT device 300. For example, the notification UI 530 may be generated in the form of a virtual object icon having a representative appearance of an oven.

[0121] In state 501B, the electronic device 100 according to an embodiment may display the notification UI 530 in the first state around the avatar 510. For example, the notification UI 530 in the first state may be displayed as an inactive state in a semi-transparent state according to the absence of a user input.

[0122] The electronic device 100 according to the embodiment may receive a user input selecting the notification UI 530. The electronic device 100 according to the embodiment may transmit an input signal to the server device 200 according to the user input selecting the notification UI 530.

[0123] The server device 200 according to the embodiment may generate an activated notification UI 531 based on an input signal for the notification UI 530. The server device 200 according to the embodiment may generate a detailed notification UI 540 based on an input signal for the notification UI 530. The server device 200 according to the embodiment may transmit the virtual space content 501C in which the presented notification UI 531 and the detailed notification UI 540 are generated to the electronic device 100.

[0124] In state 501C, the electronic device 100 according to an embodiment may display an activated notification UI 531 according to a user input selecting the notification UI 530, and may display a detailed notification UI 540. The electronic device 100 according to an embodiment may display the notification UI 531 in the second state around the avatar 510. For example, the notification UI 531 in the second state may be displayed as an activated state in an opaque state according to the user input.

[0125] In state 501C, the electronic device 100 according to an embodiment may display a detailed notification UI 540 including a message regarding the state information 502 of the IoT device 300. For example, the electronic device 100 may display a detailed notification UI 540 including a message “Cooking is completed”.

[0126] Figure 6a is a diagram illustrating an operation of displaying a notification UI of status information about an IoT device, performed by a virtual space providing system according to an embodiment.

[0127] Reference Figure 6a , the electronic device 100 according to the embodiment may receive the virtual space content 601: 601A, 601B, and 601C executed in the server device 200, and may display the virtual space content 601: 601A, 601B, and 601C on the display. The electronic device 100 according to the embodiment may display the virtual image 610 within the virtual space content 601. Figure 6a , the IoT device 300 and the server device 200 are omitted, and only the electronic device 100 is shown.

[0128] Reference Figure 6a According to the embodiment, the server device 200 and Figure 5 The difference of the embodiment is that the server device 200 generates notification UIs 630 and 631, which represent virtual objects in the form of 3D objects that model the actual device. The electronic device 100 according to the embodiment can display notification UIs 630 and 631 in the form of 3D objects that model the actual IoT device 300 based on the device information of the IoT device 300.

[0129] The electronic device 100 may represent a state 601A in which the control UI 620 is displayed around the avatar 610, a state 601B in which the control UI 620 and the inactivated notification UI 630 are displayed around the avatar 610, and a state 601C in which the control UI 620 and the activated notification UI 631 are displayed around the avatar 610 and the detailed notification UI 640 is displayed. The state 601A is a general state in which the avatar 610 performs a content activity. The state 601B is a notification state for notifying the state information of the IoT device 300. The state 601C is a detailed notification state for notifying the state information of the IoT device 300 in detail.

[0130] In the state 601A, the electronic device 100 according to the embodiment may perform a content activity through the avatar 610 displayed in the virtual space content 601A.

[0131] The electronic device 100 according to the embodiment may receive status information of the IoT device 300, and may transmit the status information to the server device 200. The electronic device 100 may receive device information of the IoT device 300, and may transmit the device information to the server device 200.

[0132] The server device 200 according to the embodiment may generate a notification UI 630 around the avatar 630 based on the device information of the IoT device 300. The server device 200 according to the embodiment may generate a virtual object in the form of a 3D object that models the IoT device 300 as the notification UI 630. In state 601B, the server device 200 according to the embodiment may generate the notification UI 630 by considering the avatar 610 and the background around the avatar 610. For example, the server device 200 may generate an oven corresponding to the notification UI 630 so that the oven can be placed on the shelf object 660 around the avatar 610 by considering the coordinates of the shelf object 660.

[0133] The server device 200 according to the embodiment may generate the inquiry UI 640 together with the notification UI 630. The server device 200 may generate the inquiry UI 640 for asking the user whether to make a selection, and the location of the inquiry UI 640 is not limited to the surroundings of the avatar 610, and the inquiry UI 640 may be placed at any of various locations.

[0134] The server device 200 according to the embodiment may transmit the virtual space content 601B in which the notification UI 630 and the inquiry UI 640 are generated to the electronic device 100 .

[0135] In state 601B, the electronic device 100 according to the embodiment may display a notification UI 630 around the avatar according to the change of the state information of the IoT device 300, and may display an inquiry UI 640 in a partial area. The electronic device 100 according to the embodiment may display the virtual space content 601B including the notification UI 630 and the inquiry UI 640 received from the server device 200.

[0136] In state 601B, the electronic device 100 according to the embodiment may display a virtual object in the form of a 3D object modeling the IoT device 300 as a notification UI 630. The electronic device 100 according to the embodiment may display the notification UI 630 by considering the background around the avatar 610. For example, the electronic device 100 may display an oven corresponding to the notification UI 630 to be placed on a shelf object 660 around the avatar 610. The electronic device 100 according to the embodiment may display the notification UI 630 in the first state around the avatar 610. For example, the notification UI 630 of the first state may be displayed as an inactive state in a semi-transparent state according to the absence of a user input.

[0137] In state 601B, the electronic device 100 according to the embodiment may display an inquiry UI 640 for asking the user whether to make a selection. For example, the electronic device 100 may display an inquiry UI 640 with a message (such as "Do you want to check the oven? OK / Cancel") in a partial area of ​​the virtual space content 601B. However, the embodiment is not limited thereto. The electronic device 100 according to the embodiment may perform subsequent operations in response to a user input selecting the notification UI 630.

[0138] The electronic device 100 according to the embodiment may receive a user input selecting the notification UI 630. The electronic device 100 according to the embodiment may transmit an input signal to the server device 200 according to the user input selecting the notification UI 630.

[0139] The server device 200 according to the embodiment may generate an activated notification UI 631 based on an input signal for the notification UI 630. The server device 200 according to the embodiment may generate a detailed notification UI 650 based on an input signal for the notification UI 630. The server device 200 according to the embodiment may transmit the virtual space content 601C in which the activated notification UI 631 and the detailed notification UI 650 are generated to the electronic device 100.

[0140] In state 601C, the electronic device 100 according to an embodiment may display an activated notification UI 631 according to a user input selecting the notification UI 630, and may display a detailed notification UI 650. The electronic device 100 according to an embodiment may display the notification UI 631 of the second state around the avatar 610. For example, the notification UI 631 in the second state may be displayed as an activated state in a semi-transparent state according to the user input.

[0141] In state 601C, the electronic device 100 according to an embodiment may display a detailed notification UI 650 including a message about status information of the IoT device 300. For example, the electronic device 100 may display a detailed notification UI 650 including a message “Cooking is completed”.

[0142] Figure 6b is a diagram illustrating an operation of displaying a notification UI of status information about an IoT device, performed by a virtual space providing system according to an embodiment.

[0143] Reference Figure 6b , the electronic device 100 according to the embodiment may receive the virtual space content 601: 601D, 601E, and 601F executed in the server device 200, and may display the virtual space content 601: 601D, 601E, and 601F on the display. The electronic device 100 according to the embodiment may display the virtual image 610 within the virtual space content 601. Figure 6b , the IoT device 300 and the server device 200 are omitted, and only the electronic device 100 is shown.

[0144] Reference Figure 6b According to the embodiment, the server device 200 and Figure 5 The difference of the embodiment is that the server device 200 generates notification UIs 670 and 671, which represent virtual objects in the form of 2D objects having the appearance of the IoT device 300. The electronic device 100 according to the embodiment can display notification UIs 670 and 671 in the form of 2D objects having the appearance of the actual IoT device 300 based on the device information of the IoT device 300. For example, the electronic device 100 can display a virtual object icon having the appearance of an oven (which is the IoT device 300). The electronic device 100 according to the embodiment can display a detailed notification UI 651, which includes a message about the status information of the IoT device 300.

[0145] Now refer to Figures 7 to 12 An operation of displaying a control UI in a virtual object management UI performed by the electronic device 100 according to an embodiment is described.

[0146] Figure 7 is a flowchart of an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0147] Reference Figure 7 In operation S705, the electronic device 100 according to the embodiment may receive an input of a user selecting a control UI.

[0148] According to an embodiment, the electronic device 100 may display a control UI for controlling the IoT device 300 around the virtual image. The electronic device 100 according to the embodiment may receive virtual space content including the control UI from the server device 200, and may display the virtual space content. According to an embodiment, the electronic device 100 may always display the control UI around the virtual image. For example, the electronic device 100 may always display the control UI when receiving status information from the IoT device 300, instead of displaying the notification UI. However, the embodiment is not limited thereto, and the control UI may not be displayed according to the user's settings.

[0149] According to an embodiment, the electronic device 100 may display a control UI in a first state around the avatar. For example, the first state may be an inactive state, and the control UI may be in a semi-transparent state, a black and white state, a dotted line state, a thin line state, etc. According to an embodiment, the control UI may be displayed in the form of a 2D object, such as a setting icon.

[0150] The electronic device 100 according to the embodiment may receive a user input for selecting a control UI. The electronic device 100 according to the embodiment may transmit an input signal to the server device 200 according to the user input for selecting a control UI.

[0151] The electronic device 100 according to an embodiment may receive a user input through an input interface (eg, a remote controller or a touch screen).

[0152] In operation S710, the server apparatus 200 according to an embodiment may activate a control UI based on an input signal with respect to the control UI.

[0153] In the present disclosure, the control UI being activated means that the control UI changes from a first state to a second state. The second state may be an activated state, and the control UI may be in an opaque state, a colored state, a solid line state, a thick line state, etc. The first state and the second state may be different from each other.

[0154] In operation S715 , the server apparatus 200 according to an embodiment may generate a virtual object list UI based on an input signal with respect to the control UI.

[0155] The server apparatus 200 according to the embodiment may generate a virtual object list UI through a list of IoT devices 300 , device information of the IoT devices 300 , and virtual object data corresponding to the IoT devices 300 stored in a memory.

[0156] According to an embodiment, the server apparatus 200 may generate a list of IoT devices 300 registered using the same account as the electronic device 100 as a virtual object list UI. According to an embodiment, the virtual object list UI may include a list of virtual objects corresponding to the IoT devices.

[0157] According to an embodiment, the server device 200 may present an activated control UI and a virtual object list UI, and may transmit the presented activated control UI and the presented virtual object list UI in a streaming manner to the electronic device 100. For example, the server device 200 may perform calculations for generating an activated control UI and a virtual object list UI, and may transmit virtual space content including the activated control UI and the virtual object list UI to the electronic device 100.

[0158] In operation S720, the electronic device 100 according to an embodiment may display the activated control UI, and may display a virtual object list UI.

[0159] According to an embodiment, the electronic device 100 may display the control UI in the second state around the avatar. The second state may be an activated state, and the control UI may be in an opaque state or a colored state.

[0160] According to an embodiment, the electronic device 100 may display a virtual object list UI by a user selecting an input of a control UI. According to an embodiment, the electronic device 100 may display a virtual object list UI including a list of IoT devices 300 registered using the same account as the electronic device 100. According to an embodiment, the electronic device 100 may display at least one of the type of controllable IoT device 300, the location of the IoT device 300, and the status information of the IoT device 300 through the virtual object list UI. For example, the IoT device 300 may be controlled by the server device 200.

[0161] In operation S725, the electronic device 100 according to the embodiment may receive an input of a user selecting a virtual object displayed on the virtual object list UI. The electronic device 100 according to the embodiment may send an input signal to the server device 200 according to the input of the user selecting a virtual object from a plurality of virtual objects listed on the virtual object list UI.

[0162] In operation S730, the server apparatus 200 according to an embodiment may generate a detailed control UI including a control function of the selected virtual object. The server apparatus 200 according to an embodiment may generate a detailed control UI based on an input signal of a user who selects a virtual object.

[0163] According to an embodiment, the control UI may include a control function of the selected virtual object. For example, when the selected virtual object is an oven, the detailed control UI may include various icons, texts, etc. for setting the cooking function of the oven. For example, when the selected virtual object is an air conditioner, the detailed control UI may include various icons, texts, etc. for setting the driving function of the air conditioner.

[0164] In operation S735, the electronic device 100 according to an embodiment may display a detailed control UI including a control function of the selected virtual object, and may receive a user's control input for the virtual object.

[0165] According to an embodiment, the electronic device 100 may receive a control input from a user to set a control function of a virtual object displayed on a detailed control UI. For example, the electronic device 100 may receive a control input from a user to set a cooking temperature, a cooking mode, etc. of an oven. For example, the electronic device 100 may receive a control input from a user to set a driving temperature, a driving mode, etc. of an air conditioner.

[0166] The electronic device 100 according to the embodiment may transmit a control input for setting a control function of a virtual object by a user to the server device 200 .

[0167] In operation S740, the server device 100 according to the embodiment may generate a control command of the IoT device 300 corresponding to the control input of the user, and may transmit the control command. The server device 200 according to the embodiment may generate a control command of the IoT device 300 corresponding to the virtual object based on the control input of the user received from the electronic device 100. The server device 200 according to the embodiment may transmit the control command of the IoT device 300 to the electronic device 100.

[0168] In operation S745, the electronic device 100 according to the embodiment may transmit the control command of the IoT device 300 to the IoT device 300. The electronic device 100 according to the embodiment may receive the control command of the IoT device 300 from the server device 200.

[0169] In operation S750, the IoT device 300 according to the embodiment may perform the operation of the IoT device 300 in response to the control command. The IoT device 300 according to the embodiment may receive the control command from the electronic device 100, and may perform the operation corresponding to the control input of the user. For example, an oven as the IoT device 300 may set the cooking temperature and the cooking mode according to the control input of the user and perform the cooking operation. For example, an air conditioner as the IoT device 300 may set the driving temperature and the driving mode according to the control input of the user and perform the driving operation.

[0170] According to an embodiment of the present disclosure, the user can control the IoT device 300 by selecting the control UI displayed around the avatar. For example, even when the user does not manipulate the avatar so that the avatar moves to the location where the virtual object corresponding to the IoT device 300 is located, the user can control the IoT device 300 by interacting with the control UI displayed around the avatar.

[0171] Figure 8 is a flowchart of an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0172] Reference Figure 8 , operations S805 to S835 are performed with Figure 7 The operations in are the same, and thus their description is omitted.

[0173] In operation S840, the server apparatus 100 according to the embodiment may directly transmit a control command of the IoT device 300 corresponding to the control input of the user to the IoT device 300. The server apparatus 200 according to the embodiment may have an IoT device management module and may directly communicate with the IoT device 300.

[0174] In operation S845 , the IoT device 300 according to the embodiment may perform an operation corresponding to the control command received from the server apparatus 200 .

[0175] Fig. 9 is a diagram illustrating an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0176] Reference Fig. 9 , the electronic device 100 according to the embodiment may receive the virtual space content 901: 901A, 901B, and 901C executed in the server device 200, and may display the virtual space content 901: 901A, 901B, and 901C on the display. The electronic device 100 according to the embodiment may display the virtual image 910 in the virtual space content 901. The server device 200 according to the embodiment may generate a virtual object in the virtual space content 901 corresponding to the IoT device 300 in the real space. Fig. 9 It is shown that the IoT device 300 is an air conditioner.

[0177] exist Fig. 9, the electronic device 100 may display a state 901A in which an inactivated control UI 920 is displayed around the avatar 910, a state 901B in which an activated control UI 921 is displayed around the avatar 910 and a virtual object list UI 930 is displayed on a partial area, and a state 901C in which an activated control UI 921 is displayed around the avatar 910 and a detailed control UI 940 is displayed on a partial area. State 901A is a general state in which the avatar 910 performs a content activity. State 901B is a standby state that shows the IoT device 300 controllable by the user as a list of virtual objects. State 901C is a detailed control state that shows the control function of the IoT device 300 according to the selected virtual object.

[0178] In state 901A, the electronic device 100 according to an embodiment may perform a content activity through the avatar 910 displayed in the virtual space content 901A. The electronic device 100 according to an embodiment may display an inactivated control UI 920 around the avatar 910. For example, the electronic device 100 may always display an inactivated control UI 920 around the avatar 910 performing various content activities. However, the embodiment is not limited thereto.

[0179] The electronic device 100 according to the embodiment may receive an input of a user selecting the control UI 920. The electronic device 100 according to the embodiment may transmit an input signal to the server device 200 according to the input of the user selecting the control UI 920.

[0180] The server device 200 according to an embodiment may generate an activated notification UI 921 based on an input signal for the control UI 920. The server device 200 according to an embodiment may generate a virtual object list UI 930 based on an input signal for the control UI 920. The server device 200 according to an embodiment may transmit, to the electronic device 100, a virtual space content 901B in which the presented control UI 921 and the virtual object list UI 930 are generated.

[0181] In state 901B, the electronic device 100 according to an embodiment may display an activated control UI 921 according to a user input selecting the control UI 920, and may display a virtual object list UI 930. The electronic device 100 according to an embodiment may display the control UI 921 in the second state around the avatar 910. For example, the control UI 921 in the second state may be displayed as an activated state in an opaque state according to a user input.

[0182] In state 901B, the electronic device 100 according to the embodiment may display a list of IoT devices registered using the same account in the server device 200 through the virtual object list UI 930. For example, the electronic device 100 may display a list of virtual objects corresponding to the IoT devices through the virtual object list UI 930. For example, the electronic device 100 may display the corresponding positions of the IoT devices 300 and the virtual objects corresponding to the IoT devices 300 in the form of icons through the virtual object list UI 930. For example, the virtual object list UI 930 may include an air conditioner icon located in a small room, an air conditioner icon located in a living room, and an oven icon located in a kitchen.

[0183] The electronic device 100 according to the embodiment may receive a user input of selecting one virtual object 931 displayed on the virtual object list UI 930. The electronic device 100 according to the embodiment may send an input signal to the server device 200 according to the user input of selecting the one virtual object 931. For example, the electronic device 100 may send an input signal to the server device 200 according to the user input of selecting the air conditioner icon located in the living room.

[0184] The server apparatus 200 according to an embodiment may generate a detailed control UI including a control function of the selected virtual object 931. The server apparatus 200 according to an embodiment may generate a detailed control UI 940 based on an input signal of a user who selects the virtual object 931.

[0185] According to an embodiment, the detailed control UI 940 may include a control function of the selected virtual object 931. For example, the detailed control UI 940 may include a first field 941 for selecting an operation mode among the operation functions of the air conditioner and a second field 942 for setting an operation temperature.

[0186] The server device 200 according to the embodiment may transmit the virtual space content 901C in which the detailed control UI 940 is generated to the electronic device 100 .

[0187] In state 901C, the electronic device 100 according to an embodiment may display a detailed control UI 930 for setting a control function of the selected virtual object 931. The electronic device 100 according to an embodiment may display the detailed control UI 930 on a partial area.

[0188] In state 901C, the electronic device 100 according to the embodiment may receive a control input for the user to set a control function of a virtual object 931 displayed on the detailed control UI 930. For example, the electronic device 100 may receive a control input for the user to set the operating mode to "smart comfort" through the first field 941, and may receive a control input for the user to set the operating temperature to "28 degrees" through the second field 942.

[0189] In state 901C, the electronic device 100 according to the embodiment may display a list of IoT devices registered using the same account in the server device 200 through the virtual object list UI 930. For example, the electronic device 100 may display a list of virtual objects corresponding to the IoT devices through the virtual object list UI 930. For example, the electronic device 100 may display the corresponding positions of the IoT devices 300 and the virtual objects corresponding to the IoT devices 300 in the form of icons through the virtual object list UI 930. For example, the virtual object list UI 930 may include an air conditioner icon located in a small room, an air conditioner icon located in a living room, and an oven icon located in a kitchen.

[0190] The electronic device 100 according to the embodiment may send a user input signal received through the virtual object list UI 930 to the server device 200. The server device 200 according to the embodiment may generate a control command of the IoT device 300 corresponding to the user's control input, and may send the control command. For example, the server device 200 may send the control command of the IoT device 300 to the electronic device 100, and the electronic device 100 may send the control command to the IoT device 300. Alternatively, for example, the server device 200 may send the control command of the IoT device 300 directly to the IoT device 300.

[0191] The IoT device 300 according to the embodiment may receive a control command from the server device 200 or the electronic device 100. The IoT device 300 may perform an operation corresponding to the user's control input based on the received control command, as shown in reference numeral 902. For example, an air conditioner as the IoT device 300 may operate in the "smart comfort" mode and at a temperature of "28 degrees" according to the user's control input.

[0192] Now refer to Figures 10 to 12 Various operations are described for displaying various UIs for controlling the IoT device 300. The server apparatus 200 and the IoT device 300 are omitted in the drawing.

[0193] Fig.10 is a diagram illustrating an operation of displaying a virtual object list UI performed by a virtual space providing system according to an embodiment.

[0194] Reference Fig.10 , the electronic device 100 according to the embodiment may display information of IoT devices through the virtual object list UI 1030 .

[0195] In state 1001A, the electronic device 100 according to the embodiment may perform a content activity through the avatar 1010 displayed on the virtual space content 1001A. The electronic device 100 according to the embodiment may display an inactive control UI 1020 around the avatar 1010. State 1001A corresponds to Fig. 9 Status 901A.

[0196] In state 1001B, the electronic device 100 according to an embodiment may display an activated control UI 1021 according to a user input selecting the control UI 1020 , and may display a virtual object list UI 1030 .

[0197] The electronic device 100 according to the embodiment may display a list of IoT devices registered using the same account in the server device 200 through the virtual object list UI 1030. The electronic device 100 according to the embodiment may display at least one of the type of the IoT device 300, the location of the IoT device 300, and the status information of the IoT device 300 through the virtual object list UI 1030. For example, the electronic device 100 may display the type of the IoT device 300 using a virtual object icon.

[0198] For example, the virtual object list UI 1030 may include virtual objects 1031, 1032, and 1033 corresponding to the IoT device 300 and the corresponding location of the IoT device 300. For example, the electronic device 100 may display an air conditioner 1031 located in a small room, an air conditioner 1032 located in a living room, and an oven 1033 located in a kitchen through the virtual object list UI 1030.

[0199] The electronic device 100 according to the embodiment may display the power on / off state, operation mode, operation completion, operation start, pause, operation resumption, etc. of the IoT device 300 among the status information of the IoT device 100 .

[0200] According to an embodiment, the electronic device 100 may display the virtual object icon in different states depending on whether the IoT device 300 is in an on state or an off state. For example, when the IoT device 300 is in an on state, the electronic device 100 may display the virtual object icon in color, check, shadow, etc. For example, when the IoT device 300 is in an off state, the electronic device 100 may display the virtual object icon in black and white, or may display the virtual object icon by omitting the check or shadow. For example, the electronic device 100 may perform shadow display on the air conditioner 1031 and the oven 1033 in the on state, and may omit the shadow display for the air conditioner 1032 in the off state.

[0201] According to an embodiment, the electronic device 100 may display representative operation information in the state information of the IoT device 300 in the on state as text. For example, the operating temperature 1034 of the air conditioner 1031 in the on state may be displayed as text, and the cooking state 1035 of the oven 1033 in the on state may be displayed as text.

[0202] Meanwhile, the server device 200 according to the embodiment may generate a virtual object list UI 1030 including the type of the IoT device 300 , the location of the IoT device 300 , and status information of the IoT device 300 , and may provide the electronic device 100 with a virtual space content 1001B including the virtual object list UI 1030 .

[0203] Fig.11 is a diagram illustrating an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0204] refer to Fig.11 , the electronic device 100 according to the embodiment may display the virtual object 1150 in the form of a 3D object based on a user input of selecting one virtual object displayed on the virtual object list UI 1130 .

[0205] In state 1101A, the electronic device 100 according to the embodiment may display the avatar 1110 of the virtual space content 1101A and the control UI 1120. State 1101A corresponds to Fig. 9 Status 901A.

[0206] In state 1101B, the electronic device 100 may receive a user input of selecting a virtual object icon from a plurality of virtual objects listed on the virtual object list UI 1130. State 1101B corresponds to Fig. 9 Status 901B.

[0207] In state 1101C, the electronic device 100 according to the embodiment may display a virtual object 1150 around the avatar 1110 in a form obtained by modeling an actual device. For example, the virtual object 1150 may be a 3D object that models an oven as the IoT device 300.

[0208] In state 1101C, the electronic device 100 according to an embodiment may display a detailed control UI 1140 on which control functions of a virtual object are listed. For example, the detailed control UI 1140 may include a cooking function, temperature, content check, cooking time, etc. of the virtual object.

[0209] The server device 200 according to the embodiment may generate the virtual object 1150 around the avatar 1110 in a form obtained by modeling the actual device. The server device 200 according to the embodiment may generate the virtual object 1150 by considering the background around the avatar 1110. The manner in which the server device 200 generates the virtual object 1150 in a form obtained by modeling the actual IoT device may be similar to Figure 6a The server device 200 generates the notification UI 630 in the form of modeling the actual IoT device. The server device 200 according to the embodiment may provide the electronic device 100 with virtual space content 1101C including the virtual object 1150.

[0210] Fig.12 is a diagram illustrating an operation of displaying a control UI for controlling an IoT device, performed by a virtual space providing system according to an embodiment.

[0211] Reference Fig.12 , the electronic device 100 according to the embodiment may display virtual space contents 1201 and 1202, including detailed control UIs 1240: 1241 and 1242 and virtual objects 1250: 1251 and 1252. The virtual space contents 1201 and 1202 may be contents according to an input of a user selecting one of the virtual objects displayed on the virtual object list UI. The virtual space contents 1201 and 1202 may be screens for manipulating the selected virtual object from a first-person perspective. When the electronic device 100 displays the virtual space contents 1201 and 1202, the user may intuitively control them, just like using the actual IoT device 300.

[0212] For example, the electronic device 100 may display the virtual space content 1201 including the detailed control UI 1241 and the virtual object 1251 in the first-person perspective according to the user's input of selecting the oven. For example, the electronic device 100 may display the virtual image 1211 for controlling the oven in the first-person perspective. For example, the user may manipulate the operation of the virtual image 1211 and may set the control function of the oven by selecting the detailed control UI 1241.

[0213] For example, the electronic device 100 may display the virtual space content 1202 including the detailed control UI 1242 and the virtual object 1252 in the first-person perspective according to the user's input of selecting the air conditioner. For example, the electronic device 100 may display the virtual image 1212 for controlling the air conditioner in the first-person perspective. For example, the user may manipulate the operation of the virtual image 1212 and may set the control function of the air conditioner by selecting the detailed control UI 1242.

[0214] The server device 200 according to the embodiment may generate virtual space contents 1201 and 1202, including detailed control UIs 1240: 1241 and 1242 and virtual objects 1250: 1251 and 1252. The server device 200 according to the embodiment may generate virtual space contents 1201 and 1202 of a first-person perspective. The server device 200 according to the embodiment may provide the virtual space contents 1201 and 1202 of a first-person perspective to the electronic device 100.

[0215] Fig.13 is a block diagram of a structure of an electronic device according to an embodiment.

[0216] Reference Fig.13 , the electronic device 100 according to the embodiment may include a processor 110 , a communication interface 120 , a display 130 , an input interface 140 , and a memory 150 .

[0217] The communication interface 120 according to the embodiment can connect the electronic device 100 to an external device, such as an IoT device 300, a server device 200, or a mobile terminal, under the control of the processor 110. For example, the communication interface 120 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a local area network (LAN) module, an Ethernet module, a wired communication module, etc. Each of these communication modules can be implemented in the form of at least one hardware chip. The wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), LTE advanced (LTE-A), fourth generation (4G), fifth generation (5G).

[0218] The communication interface 120 according to an embodiment may receive content including an avatar from the server device 200 under the control of the processor 110 .

[0219] The communication interface 120 according to an embodiment may receive state information of the IoT device 300 from the IoT device 300 under the control of the processor 110 .

[0220] The communication interface 120 according to an embodiment may receive device information of the IoT device 300 from the IoT device 300 under the control of the processor 110 .

[0221] The communication interface 120 according to an embodiment may send a control command to the IoT device 300 under the control of the processor 110 .

[0222] The communication interface 120 according to an embodiment may receive content in which a virtual object management UI is generated around the avatar from the server device 200 under the control of the processor 110 .

[0223] The communication interface 120 according to an embodiment may receive content in which an activated virtual object management UI is generated from the server device 200 under the control of the processor 110 .

[0224] The display 130 according to an embodiment may generate a driving signal by converting an image signal, a data signal, an on screen display (OSD) signal, and a control signal processed by the processor 110 , and may display an image according to the driving signal.

[0225] The display 130 according to an embodiment may display contents including an avatar under the control of the processor 110 .

[0226] The display 130 according to an embodiment may display a virtual object management UI for managing virtual objects corresponding to the IoT device around the avatar under the control of the processor 110 .

[0227] The display 130 according to an embodiment may display an activated virtual object management UI based on a user input to the virtual object management UI under the control of the processor 110 .

[0228] The input interface 140 according to an embodiment may receive a user input for controlling the electronic device 100. The user interface 140 may include, for example, a touch panel for sensing a user touch, a button for receiving a user input, a scroll wheel, a keyboard, a dome switch, a microphone for voice recognition, and a motion detection sensor. The input interface 140 according to an embodiment may receive a user input via an external control device such as a remote controller.

[0229] The communication interface 140 according to an embodiment may receive an input of a user selecting a virtual object management UI under the control of the processor 110. For example, the input interface 140 may receive a user input for a notification UI, a control UI, a detailed notification UI, a virtual object list UI, a detailed control UI, an inquiry UI, etc. under the control of the processor 110.

[0230] The memory 150 according to an embodiment may store various data, programs or applications for driving and controlling the electronic device 100. The program stored in the memory 150 according to an embodiment may include one or more instructions. The program (one or more instructions) or application stored in the memory 150 may be executed by the processor 110.

[0231] The memory 150 according to an embodiment may include at least one type of storage medium selected from the following: a flash memory type, a hard disk type, a multimedia card micro, a card type memory (e.g., a secure digital (SD) or an extreme digital (XD) memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0232] The processor 110 according to an embodiment controls the overall operation of the electronic device 100 , controls signal transmission between internal components of the electronic device 100 , and processes data.

[0233] The processor 110 according to an embodiment may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a video processing unit (VPU). Alternatively, according to an embodiment, the processor 110 may be implemented in the form of a system on chip (SOC) integrating at least one of a CPU, a GPU, and a VPU. Alternatively, the processor 110 may also include a neural processing unit (NPU).

[0234] The processor 110 according to an embodiment may control operations to be performed by the electronic device 100 by executing one or more programs stored in the memory 150 .

[0235] The processor 110 according to an embodiment may display content including a virtual avatar on the display 130 by executing one or more instructions stored in the memory 150. The processor 110 according to an embodiment may display a virtual object management UI for managing virtual objects corresponding to the IoT device 300 around the virtual avatar. The processor 110 according to an embodiment may display an activated virtual object management UI based on a user input to the virtual object management UI. When the virtual object management UI includes a notification UI, the processor 110 according to an embodiment may obtain status information from the IoT device 300 through the communication interface 120.

[0236] The processor 110 according to an embodiment may obtain device information of the IoT device 300 from the IoT device 300 through the communication interface 120 by executing one or more instructions stored in the memory 150. The processor 110 according to an embodiment may display a notification UI through a virtual object based on the device information of the IoT device 300.

[0237] The processor 110 according to an embodiment may display a detailed notification UI having a message about the status information of the IoT device 300 based on a user input to the notification UI by executing one or more instructions stored in the memory 150 .

[0238] For example, the device information may include at least one of a model name, a serial number, and a control function of the IoT device 300 .

[0239] According to an embodiment, the processor 110 can explicitly notify the UI in any of the following forms: a 2D object form having the appearance of the IoT device 300, a 2D object form having a representative appearance according to the type of the IoT device 300, and a 3D object form modeling the IoT device 300 by executing one or more instructions stored in the memory 150.

[0240] When the virtual object management UI includes a control UI, the processor 110 according to an embodiment may obtain a user's control input to the IoT device 300 through the control UI by executing one or more instructions stored in the memory 150. The processor 110 according to an embodiment may send a control command to the IoT device 300 based on the user's control input.

[0241] The processor 110 according to an embodiment may display a virtual object list UI on which a plurality of virtual objects are listed based on a user input to the control UI by executing one or more instructions stored in the memory 150. Based on a user input to one virtual object among the plurality of virtual objects listed in the virtual object list UI, the processor 110 according to an embodiment may display a detailed control UI having a control function of the one virtual object.

[0242] The processor 110 according to an embodiment may display at least one of the type of the controllable IoT device 300 , the location of the IoT device 300 , and status information of the IoT device 300 through the control UI by executing one or more instructions stored in the memory 150 .

[0243] The processor 110 according to an embodiment may display the virtual object list UI in a first state by executing one or more instructions stored in the memory 150. The processor 110 according to an embodiment may display the activated virtual object management UI in a second state different from the first state.

[0244] The processor 110 according to an embodiment may receive content including a virtual object list UI from the server apparatus 200 by executing one or more instructions stored in the memory 150. The processor 110 according to an embodiment may transmit state information of the IoT device 300 to the server apparatus 200.

[0245] According to an embodiment of the present disclosure, the electronic device 100 may control a virtual object synchronized with the IoT device 300, or may display a virtual object list UI for providing status information of the IoT device 300 around the virtual image. The electronic device 100 according to the embodiment may conveniently manage virtual objects in the virtual space and IoT devices in the real space through user interaction with the virtual object management UI displayed around the virtual image. For example, the user may control the IoT device 300 in the real space without moving to the IoT device 300, and may control the virtual objects in the virtual space without moving the virtual image to the space where the virtual object is located. The user may conveniently control the IoT device 300 in the real space through the virtual object management UI displayed around the virtual image. In addition, the electronic device 100 may adaptively display the status information of the IoT device 300 in real time, and may interact with the user in real time.

[0246] Fig.14 is a detailed block diagram of a structure of an electronic device according to an embodiment.

[0247] Fig.14 The electronic device 1400 may be Fig.13 An example of an electronic device 100 is shown. Fig.14 Refer to the above Fig.13 The description given is the same as the description given.

[0248] Reference Fig.14 , the electronic device 1400 may include a processor 1401 and a memory 1402. The processor 1401 and the memory 1402 included in the electronic device 1400 may execute the same Fig.13The same operation as the processor 110 and the memory 150 included in the electronic device 100.

[0249] According to an embodiment, in addition to the processor 1401 and the memory 1402, the electronic device 1400 may further include a tuner 1410, a communication interface 1420, a sensor 1430, an input / output (I / O) interface 1440, a video processor 1450, a display 1460, an audio processor 1470, an audio output interface 1480, and an input interface 1490. The communication interface 1420 may correspond to Fig.13 The display 1460 may correspond to Fig.13 The input interface 1490 may correspond to Fig.13 An input interface 140 is provided.

[0250] Among many radio wave components obtained through amplification, mixing, resonance, etc. of wired or wireless broadcast content, the tuner 1410 can tune and select only the frequency of the channel that the electronic device 1400 wants to receive. The content received by the tuner 1410 is decoded and separated into audio, video and / or additional information. The audio, video and / or additional information can be stored in the memory 1402 under the control of the processor 1401.

[0251] According to an embodiment, the communication interface 1420 may connect the electronic device 1400 to, for example, a peripheral device, an external device, a server, or a mobile terminal under the control of the processor 1401. The communication interface 1420 may include at least one communication module capable of performing wireless communication. The communication interface 1420 may include at least one of a wireless LAN module 1421, a Bluetooth module 1422, and a wired Ethernet 1423 corresponding to the performance and structure of the electronic device 1400.

[0252] The wireless LAN module 1421 may send a Wi-Fi signal to a peripheral device or receive a Wi-Fi signal from a peripheral device according to the Wi-Fi communication standard. The Bluetooth module 1422 may receive a Bluetooth signal sent from a peripheral device according to the Bluetooth communication standard. The Bluetooth module 1422 may be a Bluetooth low energy (BLE) communication module and may receive a BLE signal. The Bluetooth module 1422 may continuously or temporarily scan the BLE signal to detect whether the BLE signal is received.

[0253] The sensor 1430 may sense a user's voice, a user's image, or a user's interaction, and may include a microphone, a camera, a light receiver, and a sensing unit.

[0254] The I / O interface 1440 may receive video (e.g., a moving image signal or a still image signal), audio (e.g., a voice signal or a music signal), and additional information from an external device under the control of the processor 1401. The I / O interface 1440 may include one of a High Definition Multimedia Interface (HDMI) port, a component jack, a PC port, and a USB port.

[0255] The video processor 1450 may process image data to be displayed on the display 1460 , and may perform various image processing such as decoding, rendering, scaling, noise filtering, frame rate conversion, and resolution conversion on the image data.

[0256] The display 1460 may output content received from a broadcasting station, content received from an external device such as an external server or an external storage medium, or content provided by various applications such as an OTT service provider or a metaverse content provider on a screen.

[0257] The audio processor 1470 processes audio data. The audio processor 1470 may perform various processes such as decoding, amplification, or noise filtering on the audio data.

[0258] The audio output interface 1480 may output audio included in content received via the tuner 1410, audio input via the communication interface 1420 or the I / O interface 1440, and audio stored in the memory 1402 under the control of the processor 1401. The audio output interface 1480 may include at least one of a speaker, a headphone, or a Sony / Philips Digital Interface (S / PDIF).

[0259] The input interface 1490 may receive a user input for controlling the electronic device 1400. The input interface 1490 may include, but is not limited to, various types of user input devices, including a touch pad for sensing a user's touch, a button for receiving a user's pressing operation, a scroll wheel for receiving a user's rotating operation, a keyboard, a dome switch, a microphone for voice recognition, and a motion detection sensor.

[0260] According to an embodiment, the input interface 1490 may receive a user input for controlling the avatar displayed on the display 1060. For example, the input interface 1490 may receive an input regarding a moving direction of the avatar under the control of the processor 1401. For example, when the input interface 1490 includes a remote controller having a direction key, the processor 1401 may control the moving direction of the avatar through the direction key.

[0261] According to an embodiment, the input interface 1490 may receive a user input for selecting a virtual object management UI displayed on the display 1060. For example, the input interface 1490 may receive an input for selecting a virtual object management UI under the control of the processor 1401.

[0262] Fig.15 is a block diagram of a structure of a server device according to an embodiment.

[0263] Reference Fig.15 , the server 200 according to the embodiment may include a processor 210 , a communication interface 220 , a memory 230 , and a virtual object management UI 240 .

[0264] The communication interface 220 according to the embodiment may send data or signals to the electronic device 100 and the IoT device 300, or receive data or signals from the electronic device 100 and the IoT device 300. For example, the communication interface 220 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a LAN module, an Ethernet module, a wired communication module, etc. Each of these communication modules may be implemented in the form of at least one hardware chip.

[0265] The communication interface 220 according to an embodiment may transmit content including a virtual image to the electronic device 100 under the control of the processor 210 .

[0266] The communication interface 220 according to an embodiment may receive status information of the IoT device 300 from the electronic device 100 under the control of the processor 210 .

[0267] The communication interface 220 according to the embodiment may receive an input signal for a control UI included in the virtual object management UI from the electronic device 100 under the control of the processor 210. The communication interface 220 according to the embodiment may transmit a control command for controlling the IoT device 300 corresponding to the virtual object to the IoT device 300 under the control of the processor 210.

[0268] The processor 210 according to an embodiment controls the overall operation of the server apparatus 200 , controls signal transmission between internal components of the server apparatus 200 , and processes data.

[0269] The processor 210 according to an embodiment may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a video processing unit (VPU). Alternatively, according to an embodiment, the processor 210 may be implemented in the form of a system on chip (SOC) integrating at least one of a CPU, a GPU, and a VPU. Alternatively, the processor 210 may also include a neural processing unit (NPU).

[0270] The memory 230 according to an embodiment may store various data, programs or applications for driving and controlling the server device 200. The program stored in the memory 230 may include at least one instruction. The program (one or more instructions) or application stored in the memory 230 may be executed by the processor 210.

[0271] The memory 230 according to an embodiment may store an IoT device management module 231 , a virtual object generation module 232 , a coordinate acquisition module 233 , and a presentation module 234 .

[0272] The memory 210 according to an embodiment may execute one or more instructions included in each of the IoT device management module 231 , the virtual object generation module 232 , the coordinate acquisition module 233 , and the presentation module 234 .

[0273] The processor 210 according to an embodiment may identify the IoT device 300 registered using the same account in the server apparatus 200 by executing one or more instructions included in the IoT device management module 231. The IoT device management module 231 according to an embodiment may store a list of a plurality of IoT devices and device information (e.g., model name and serial number) of the IoT device 300. The IoT device management module 231 according to an embodiment may store the appearance, type, and control function of the IoT device 300 based on the device information of the IoT device 300.

[0274] The processor 210 according to an embodiment may generate a control command for controlling the IoT device 300 by executing one or more instructions included in the IoT device management module 231 , and may transmit the control command to the IoT device 300 .

[0275] The processor 210 according to an embodiment may generate virtual object data based on the device information of the IoT device 300 by executing one or more instructions included in the virtual object generation module 232. The virtual object generation module 232 according to an embodiment may store virtual object data considering the appearance, type, and control function of the IoT device 300. The processor 210 according to an embodiment may generate a virtual object that has reflected the appearance, type, and control function of the IoT device 300 by executing one or more instructions included in the virtual object generation module 232.

[0276] The processor 210 according to the embodiment may obtain the coordinates of the object in the virtual space by executing one or more instructions included in the coordinate obtaining module 233. The processor 210 according to the embodiment may obtain the coordinates of the avatar, the coordinates of the background, etc. by executing one or more instructions included in the coordinate obtaining module 233, and generate a notification UI so that the notification UI can be located around the avatar.

[0277] The processor 210 according to an embodiment may present the virtual space content by executing one or more instructions included in the presentation module 234. The processor 210 according to an embodiment may transmit the presented virtual space content to the electronic device 100 through the communication interface 220.

[0278] The virtual object management UI 240 according to an embodiment may be a user interface (UI) for managing virtual objects in a virtual space. The virtual object management UI according to an embodiment may interact with a user in the form of a 2D object or a 3D object. The virtual object management UI according to an embodiment may include a notification UI 241, a detailed notification UI 242, a control UI 243, a virtual object list UI 244, and a detailed control UI 245. The virtual object management UI may also include an inquiry UI requesting a user selection.

[0279] The notification UI 241 according to an embodiment may indicate a notification of status information for the IoT device 300. The notification UI 241 according to an embodiment may be activated according to a user input selecting the notification UI 241.

[0280] The detailed notification UI 242 according to an embodiment may indicate a detailed message for status information of the IoT device 300 .

[0281] The control UI 243 according to an embodiment may control the IoT device 300. The control UI 243 according to an embodiment may receive a user input for controlling the IoT device 300. The control UI 243 may be activated according to the user input for selecting the control UI 243.

[0282] The virtual object list UI 244 according to an embodiment may include a list of virtual objects corresponding to IoT devices. The virtual object list UI 244 according to an embodiment may include at least one of the type of controllable IoT device 300, the location of the IoT device 300, and the status information of the IoT device 300.

[0283] The detailed control UI 245 according to an embodiment may include a control function of a virtual object selected by a user. A user input of selecting a control function of a virtual object may be obtained through the detailed control UI 245.

[0284] The virtual object list UI 240 according to an embodiment may be generated by the processor 210. The processor 210 according to an embodiment may generate the virtual object management UI 240 in the virtual space content. The processor 210 according to an embodiment may transmit the virtual space content including the virtual object management UI 240 to the electronic device 100. The processor 210 according to an embodiment may interact with the user through the virtual object management UI 240.

[0285] According to the embodiment, reference will now be made to Fig.16 It is described that the electronic device 100 executes virtual space content, generates a virtual object management UI, and displays the virtual object management UI.

[0286] Fig.16 is a block diagram of a structure of an electronic device according to an embodiment.

[0287] Reference Fig.16 According to an embodiment, the electronic device 1600 may include a processor 1610, a communication interface 1620, a display 1630, an input interface 1640, a memory 1650, and a virtual object management UI 1660. Fig.16 Refer to the above Fig.13 The description given is the same as the description given.

[0288] Fig.16 The communication interface 1620 may correspond to Fig.13 The communication interface 120, Fig.16 The display 1630 may correspond to Fig.13 a display 130, and Fig.16 The input interface 1640 may correspond to Fig.13 An input interface 140 is provided.

[0289] The electronic device 1600 according to this embodiment and Fig.13 The electronic device 100 according to the embodiment is different in that the memory 1650 may store an IoT device management module 1651 , a virtual object generation module 1652 , a coordinate acquisition module 1653 , and a presentation module 1654 .

[0290] The processor 1610 according to an embodiment may execute one or more instructions included in each of the IoT device management module 1651 , the virtual object generation module 1652 , the coordinate acquisition module 1653 , and the presentation module 1654 .

[0291] The processor 1610 according to an embodiment may identify the IoT device 300 registered using the same account in the server apparatus 200 by executing one or more instructions included in the IoT device management module 1651. The IoT device management module 1651 according to an embodiment may store a list of a plurality of IoT devices and device information (e.g., model name and serial number) of the IoT device 300. The IoT device management module 1651 according to an embodiment may store the appearance, type, and control function of the IoT device 300 based on the device information of the IoT device 300.

[0292] The processor 1610 according to an embodiment may generate a control command for controlling the IoT device 300 by executing one or more instructions included in the IoT device management module 1651 , and may transmit the control command to the IoT device 300 .

[0293] The processor 1610 according to an embodiment may generate virtual object data based on device information of the IoT device 300 by executing one or more instructions included in the virtual object generation module 1652. The virtual object generation module 1652 according to an embodiment may store virtual object data considering the appearance, type, and control function of the IoT device 300.

[0294] The processor 1610 according to an embodiment may obtain the coordinates of an object in the virtual space by executing one or more instructions included in the coordinate obtaining module 1653. The processor 1610 according to an embodiment may obtain the coordinates of the avatar, the coordinates of the background, etc. by executing one or more instructions included in the coordinate obtaining module 1653, and generate a notification UI so that the notification UI may be located around the avatar.

[0295] The processor 1610 according to an embodiment may generate virtual space content by executing one or more instructions included in the rendering module 1654 .

[0296] The processor 1610 according to an embodiment may generate a virtual object management UI 1660. The processor 1610 according to an embodiment may generate a notification UI 1661, a detailed notification UI 1662, a control UI 1663, a virtual object list UI 1664, and a detailed control UI 1665.

[0297] The processor 1610 according to an embodiment may receive an input of a user selecting the virtual object management UI 1660 through the input interface 1640, and may activate the virtual object management UI 1660. The processor 1610 according to an embodiment may display the activated virtual object management UI 1660 on the display 1630. The processor 1610 according to an embodiment may interact with the user through the virtual object management UI 1660.

[0298] The operation method of the display device according to the embodiment can be embodied as a program command that can be executed by various computer devices, and can be recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands to be recorded on the medium may be specially designed and configured for the present disclosure, or may be well known and available to ordinary technicians in the field of computer software. Examples of computer-readable recording media include: magnetic media such as hard disks, floppy disks, and tapes; optical media such as compact disk read-only memory (CD-ROM) or digital versatile disks (DVD); magneto-optical media such as optical magnetic floppy disks; and hardware devices such as ROM, random access memory (RAM), or flash memory that are specially configured to store and execute program commands. Examples of program commands are high-level language codes that can be executed by a computer by using an interpreter, etc., and machine language codes generated by a compiler.

[0299] According to an embodiment of the present disclosure, at least one operation method of a display device may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity.

[0300] A computer program product may include a software (S / W) program and a computer-readable storage medium on which the S / W program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by a manufacturer of an electronic device or an electronic market (e.g., Google Play Store, AppStore). For electronic distribution, at least a portion of the S / W program may be stored on a storage medium, or may be temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server, a server of an electronic market, or a relay server for temporarily storing SW programs.

[0301] In a system including a server and a client device, the computer program product may include the storage medium of the server or the storage medium of the client device. Alternatively, if there is a third device (e.g., a smart phone) that communicates with the server or the client device, the computer program product may include the storage medium of the third device. Alternatively, the computer program product may include the software program itself that is sent from the server to the client device or the third device or from the third device to the client device.

[0302] In this case, one of the server, the client device, and the third device may execute a computer program product to perform the method according to the disclosed embodiment. Alternatively, at least two of the server, the client device, and the third device may execute a computer program product to distribute and perform the method according to the disclosed embodiment.

[0303] For example, a server (e.g., a cloud server or an artificial intelligence server) may execute a computer program product stored on the server to control a client device communicating with the server to perform a method according to the disclosed embodiments.

[0304] Although one or more embodiments of the present disclosure have been described with reference to the drawings, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined in the following claims.

Claims

1. An electronic device (100), comprising: Communication interface (120); Display (130); A memory (150) storing one or more instructions; as well as at least one processor (110) configured to execute the one or more instructions stored in the memory (150), The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: Displaying content including a virtual image on the display (130), A virtual object management user interface (UI) is displayed around the virtual image, wherein the virtual object management UI is used to manage virtual objects corresponding to the Internet of Things (IoT) device (300). Based on user input to the virtual object management UI, displaying an activated virtual object management UI, and When the virtual object management UI includes a notification UI, status information is obtained from the IoT device (300) through the communication interface (120).

2. The electronic device (100) according to claim 1, wherein: The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: obtaining device information of the IoT device (300) from the IoT device (300) via the communication interface (120); and Based on the device information of the IoT device (300), the notification UI is displayed through the virtual object.

3. The electronic device (100) according to claim 2, wherein: The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: Based on a user input to the notification UI, a detailed notification UI is displayed, the detailed notification UI having a message regarding status information of the IoT device (300).

4. The electronic device (100) according to claim 2, wherein: The device information includes at least one of a model name, a serial number, and a control function of the IoT device (300).

5. The electronic device (100) according to any one of claims 1 to 4, wherein: The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: The notification UI is displayed in any one of the following forms: a two-dimensional 2D object form having the appearance of the IoT device (300), a 2D object form having a representative appearance according to the type of the IoT device 300, and a three-dimensional 3D object form modeling the IoT device (300).

6. The electronic device (100) according to any one of claims 1 to 5, wherein: The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: When the virtual object management UI includes a control UI, obtaining a user's control input to the IoT device (300) through the control UI, and A control command is sent to the IoT device (300) based on the control input of the user.

7. The electronic device (100) according to claim 6, wherein: The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: Based on a user input to the control UI, displaying a virtual object list UI on which a plurality of virtual objects are listed, and Based on a user input to one virtual object among the plurality of virtual objects listed in the virtual object list UI, a detailed control UI having a control function for the one virtual object is displayed.

8. The electronic device (100) according to claim 6, wherein: The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: At least one of the type of IoT device (300) controllable through the control UI, the location of the IoT device (300), and the status information of the IoT device 300 is displayed.

9. The electronic device (100) according to any one of claims 1 to 8, wherein: The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: displaying the virtual object management UI in a first state, and The activated virtual object management UI is displayed in a second state different from the first state.

10. The electronic device (100) according to any one of claims 1 to 9, wherein: The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: receiving content including the virtual object management UI from a server device (200); and The status information of the IoT device (300) is sent to the server equipment (200).

11. A server device (200), comprising: Communication interface (220); A memory (230) storing one or more instructions; as well as at least one processor (210) configured to execute the one or more instructions stored in the memory (150), The at least one processor (210) is configured to execute the one or more instructions to perform the following operations: sending content including the virtual image to the electronic device (100) via the communication interface (220); generating a virtual object management user interface (UI) around the virtual image, wherein the virtual object management UI is used to manage virtual objects corresponding to the Internet of Things (IoT) device (300); and Based on an input signal with respect to the virtual object management UI, the virtual object management UI is activated.

12. The server device (200) according to claim 11, wherein: The memory (230) comprises: One of a list of the IoT devices (300), device information of the IoT devices (300), and virtual object data corresponding to the IoT devices (300).

13. The server device (200) according to claim 11 or 12, wherein: The at least one processor (210) is configured to execute the one or more instructions to perform the following operations: generating a notification UI included in the virtual object management UI based on status information of the IoT device (300) received from the electronic device (100) through the communication interface (220); and Based on receiving an input signal for a control UI included in the virtual object management UI from the electronic device (100) through the communication interface (220), a control command for controlling an IoT device (300) corresponding to the virtual object is sent.

14. The server device (200) according to any one of claims 11 to 13, wherein: The at least one processor (110) is configured to execute the one or more instructions to perform the following operations: presenting the virtual object management UI, and The content including the presented virtual object management UI is sent to the electronic device (100) through the communication interface (220).

15. An operating method of an electronic device (100), the operating method comprising: Displaying content including the avatar on a display (130); Displaying a virtual object management user interface (UI) around the virtual image, wherein the virtual object management UI is used to manage virtual objects corresponding to the Internet of Things (IoT) device (300); Based on user input to the virtual object management UI, displaying an activated virtual object management UI; as well as When the virtual object management UI includes a notification UI, status information is obtained from the IoT device (300) through a communication interface (120).

Images