Electronic device and method for managing IoT devices in meta-cosmic environment

Abstract

A method for managing one or more Internet of Things (IoT) devices in a meta-universe environment is provided. The method comprises the following steps: detecting that a user participates in a meta universe environment; upon detecting that the user participates in the meta-universe environment, monitoring an operational state of each of the one or more IoT devices in the real-world environment; determining at least one IoT device requiring attention of the user from the one or more IoT devices based on the monitored operating state of each of the one or more IoT devices; and generating an interactive virtual image of the determined at least one IoT device requiring attention of the user within the meta-universe environment, where the interactive virtual image represents an operating state of the at least one IoT device.

Description

Technical Field

[0001] This disclosure relates to a metaverse environment. More specifically, this disclosure relates to an electronic device and method for managing one or more Internet of Things (IoT) devices in a metaverse environment. Background Technology

[0002] With technological advancements, users are seamlessly interacting with each other and digital objects using metaverse environments. A metaverse is a virtual world shared by users in real time. Furthermore, a metaverse represents the fusion of physical and virtual reality. For example, metaverse environments can have a wide range of applications, such as education and training, social interaction, environmental applications, sports and fitness, business applications, and more.

[0003] Traditionally, when a user is immersed in a metaverse session, they need to exit the metaverse session if they need to perform actions to change the operating state of one or more IoT devices (e.g., vacuum cleaners, air conditioners, etc.).

[0004] Figure 1A , Figure 1B and Figure 1C The illustration shows one or more existing scenarios associated with the metaverse environment based on relevant technologies.

[0005] exist Figure 1A In the scenario shown, at point 102, the user is shopping in the metaverse environment. At point 104, the IoT system determines that frost is forming inside the refrigerator and generates a notification associated with the frost. The generated notification is shared with the user on a smartphone to defrost the refrigerator. At point 106, the user exits the metaverse session. At point 108, the user defrosts the refrigerator.

[0006] exist Figure 1B In the scenario shown, at point 110, the user is reading a book in the metaverse environment. At point 112, the air conditioner turns off due to an error code. At point 114, the user exits the metaverse session due to discomfort. At point 116, the user begins to focus on the air conditioner.

[0007] exist Figure 1C In the scenario shown, at point 118, the user is playing a game in the metaverse environment. At point 120, a vacuum cleaner begins cleaning the house at a scheduled time. At point 122, the user is disturbed by the house cleaning and exits the metaverse session. Therefore, in the system of the relevant technology, when one or more IoT devices require the user's attention, the user needs to exit the metaverse session. Consequently, the user cannot complete one or more ongoing tasks in the metaverse session.

[0008] Therefore, a technology is needed to overcome the above problems.

[0009] The above information is provided as background information only to aid in understanding this disclosure. It is neither confirmed nor asserted that any of the above content can be used as prior art in relation to this disclosure. Summary of the Invention

[0010] Technical solutions to the technical problems

[0011] Various aspects of this disclosure are intended to at least address the aforementioned problems and / or disadvantages, and to at least provide the advantages described below. Therefore, an aspect of this disclosure is to provide a system and method for managing one or more Internet of Things (IoT) devices in a metaverse environment.

[0012] Additional aspects will be set forth in part in the description which follows, and will become clear in part from the description, or may be learned by practice of the embodiments presented.

[0013] According to aspects of this disclosure, a method for managing one or more Internet of Things (IoT) devices in a metaverse environment is provided. The method may include detecting user participation in the metaverse environment. The method may include: monitoring the operational state of each of the one or more IoT devices in a real-world environment when user participation is detected in the metaverse environment. The method may include: determining, based on the monitored operational state of each of the one or more IoT devices, at least one IoT device from the one or more IoT devices that requires user attention. The method may include: generating an interactive virtual avatar within the metaverse environment for the determined at least one IoT device requiring user attention, wherein the interactive virtual avatar represents the operational state of the at least one IoT device.

[0014] According to another aspect of this disclosure, an electronic device is provided for managing one or more Internet of Things (IoT) devices in a metaverse environment. The electronic device may include a memory storing instructions and one or more processors communicatively coupled to the memory. When executed by the one or more processors, the instructions cause the electronic device to perform operations. These operations may include detecting user participation in the metaverse environment. The operations may include: monitoring the operational state of each of the one or more IoT devices in a real-world environment when user participation is detected in the metaverse environment. The operations may include: determining, based on the monitored operational state of each of the one or more IoT devices, at least one IoT device from the one or more IoT devices that requires user attention. The operations may include: generating an interactive virtual avatar within the metaverse environment for the determined at least one IoT device requiring user attention, wherein the interactive virtual avatar represents the operational state of the at least one IoT device.

[0015] According to another aspect of this disclosure, one or more non-transitory computer-readable storage media are provided storing instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform operations. These operations may include detecting user participation in a metaverse environment. The operations may include: monitoring the operational state of each of one or more IoT devices in a real-world environment when user participation in the metaverse environment is detected. The operations may include: determining, based on the monitored operational state of each of the one or more IoT devices, at least one IoT device from the one or more IoT devices that requires user attention. The operations may include: generating an interactive virtual avatar within the metaverse environment for the determined at least one IoT device requiring user attention, wherein the interactive virtual avatar represents the operational state of the at least one IoT device.

[0016] Other aspects, advantages, and salient features of this disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments disclosed in conjunction with the accompanying drawings. Attached Figure Description

[0017] The above and other aspects, features, and advantages of some embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1A , Figure 1B and Figure 1C Illustrations are shown of one or more existing scenarios associated with the metaverse environment based on relevant technologies;

[0019] Figure 2 A block diagram of a system for managing one or more Internet of Things (IoT) devices in a metaverse environment is shown according to embodiments of the present disclosure;

[0020] Figure 3 A block diagram of multiple modules of a system for managing one or more IoT devices in a metaverse environment, according to embodiments of the present disclosure, is shown.

[0021] Figure 4A and Figure 4B A block diagram illustrating the operation of a system for managing one or more IoT devices in a metaverse environment according to various embodiments of the present disclosure is shown;

[0022] Figure 5A and Figure 5B Illustrations show various use case scenarios for managing one or more IoT devices in a metaverse environment according to different embodiments of this disclosure;

[0023] Figure 6A process flowchart depicting a method for managing one or more IoT devices in a metaverse environment according to embodiments of the present disclosure is shown.

[0024] Throughout the accompanying drawings, similar reference numerals will be understood to refer to similar parts, components, and structures. Detailed Implementation

[0025] The following description, provided with reference to the accompanying drawings, is intended to aid in a comprehensive understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. The following description includes various specific details to aid understanding, but these details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0026] The terms and words used in the following description and claims are not limited to their literal meaning, but are intended solely by the inventors to achieve a clear and consistent understanding of this disclosure. Therefore, it will be clearly understood by those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the purpose of this disclosure as defined by the appended claims and their equivalents.

[0027] It should be understood that the singular forms “a,” “an,” and “the” include plural indicators unless the context explicitly indicates otherwise. Thus, for example, a reference to “component surface” includes a reference to one or more such surfaces.

[0028] Throughout this specification, references to "one aspect," "another aspect," or similar language indicate that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Therefore, throughout this specification, the phrases "in one embodiment," "in another embodiment," and similar language may (but not necessarily) refer to the same embodiment.

[0029] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that includes a list of steps may include not only those steps but also other steps not expressly listed or inherent to such a process or method. Similarly, without further constraints, “comprising…” followed by one or more devices, subsystems, elements, structures, or components does not exclude the presence of other devices, subsystems, elements, structures, or components, or additional devices, subsystems, elements, structures, or components.

[0030] It should be understood that each box in a flowchart, as well as combinations of flowcharts, can be executed by one or more computer programs, including instructions. One or more computer programs can be stored entirely in a single storage device, or they can be divided into different parts stored in multiple different storage devices.

[0031] Any function or operation described herein can be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing and includes circuits such as application processors (APs, such as central processing units (CPUs)), communication processors (CPs, such as modems), graphics processing units (GPUs), neural processing units (NPUs) (such as artificial intelligence (AI) chips), Wi-Fi chips, Bluetooth® chips, global positioning system (GPS) chips, near field communication (NFC) chips, connectivity chips, sensor controllers, touch controllers, fingerprint sensor controllers, display driver integrated circuits (ICs), audio CODEC chips, universal serial bus (USB) controllers, camera controllers, image processing ICs, microprocessor units (MPUs), system-on-a-chip (SoCs), integrated circuits (ICs), and the like.

[0032] Figure 2 A block diagram of a system for managing one or more Internet of Things (IoT) devices in a metaverse environment is shown according to embodiments of the present disclosure.

[0033] refer to Figure 2 One or more IoT devices 201 are physical objects embedded with sensors and connectivity features, allowing them to collect data and exchange data with other devices and systems via the Internet. For example, one or more IoT devices 201 may include an IoT-enabled refrigerator, an IoT-enabled vacuum cleaner, an IoT-enabled air conditioner, etc. According to embodiments of this disclosure, system 200 is implemented in electronic device 202. Electronic device 202 may be a device that does not support IoT or a device that supports IoT. IoT-enabled electronic devices are those with built-in Internet connectivity and the ability to acquire, exchange, and process data via the Internet or other networks. These electronic devices are designed to be interconnected and can communicate with other IoT devices, cloud platforms, and applications to provide a wide range of functionalities. Examples of electronic devices 202 include, but are not limited to, smartphones, laptops, camera devices, smartwatches, etc.

[0034] According to another embodiment of this disclosure, the system can be implemented in a server (e.g., a cloud server or a remote server). In another embodiment of this disclosure, the system can be implemented in a smart hub device. The system can be partially implemented in one or more electronic devices, servers, smart hub devices, or combinations thereof.

[0035] A system implemented in one or more electronic devices, servers, smart hub devices, or combinations thereof is connected to one or more IoT devices via network 203. Network 203 can be any wireless or wired network. Wireless networks can be Wi-Fi, cellular networks, Bluetooth, Zigbee, Near Field Communication (NFC), etc. Network 203 can be the Internet. Network 203 facilitates data exchange between system 200 and one or more IoT devices 201. For example, system 200 can send one or more IoT commands to one or more IoT devices 201, causing one or more IoT devices 201 to execute one or more IoT commands to update their operational status. (Already referenced in at least another paragraph) Figure 3 This describes the details of communication between IoT devices 201.

[0036] System 200 may include one or more processors / controllers 204, input / output (I / O) interfaces 206, multiple modules 208, and memory 210.

[0037] One or more processors / controllers 204 may be operatively coupled to each of the corresponding I / O interface 206, the plurality of modules 208, and the memory 210. One or more processors / controllers 204 may include at least one data processor for performing processes in a Virtual Storage Area Network (VSAN). One or more processors / controllers 204 may include dedicated processing units, such as integrated system (bus) controllers, memory management control units, floating-point units, graphics processing units, digital signal processing units, etc. In one embodiment, one or more processors / controllers 204 may include a central processing unit (CPU), a graphics processing unit (GPU), or both. One or more processors / controllers 204 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), servers, networks, digital circuits, analog circuits, combinations thereof, or other devices now known or to be developed in the future for analyzing and processing data. One or more processors / controllers 204 may execute software programs (e.g., manually generated (i.e., programmed) code) to perform desired operations. The processor / controller 204 may be a general-purpose processor (e.g., CPU, application processor (AP), etc.), a graphics-specific processing unit (e.g., GPU, visual processing unit (VPU)), and / or an artificial intelligence (AI)-specific processor (e.g., neural processing unit (NPU)).

[0038] One or more processors / controllers 204 control the processing of input data based on predefined operating rules or machine learning (ML) models stored in non-volatile memory and volatile memory. The predefined operating rules or ML models are provided through training or learning.

[0039] Providing through learning means: developing predefined operational rules or ML models for desired characteristics by applying learning techniques to multiple learning datasets. This learning can be performed within the device itself that performs the ML according to the embodiment, and / or can be implemented via a separate server / system.

[0040] ML models can consist of multiple neural network layers. Each layer has multiple weight values, and layer operations are performed by computing the previous layer and operating on the multiple weights. Examples of neural networks include, but are not limited to, Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Generative Adversarial Networks (GANs), and Deep Q-Networks.

[0041] Learning techniques are methods for training a predetermined target device (e.g., a robot) using multiple learning datasets to enable, allow, or control the target device to make determinations or predictions. Examples of learning techniques include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0042] One or more processors / controllers 204 may be configured to communicate with one or more input / output (I / O) devices via corresponding I / O interfaces 206. The I / O interfaces 206 may employ communication methods such as Code Division Multiple Access (CDMA), High-Speed ​​Packet Access (HSPA+), Global System for Motion Communications (GSM), Long Term Evolution (LTE), and Global Microwave Access Interoperability (WiMax).

[0043] One or more processors / controllers 204 may be configured to communicate with a communication network via a network interface. In an embodiment, the network interface may be an I / O interface 206. The network interface may be connected to the communication network to enable the electronic device 202 to connect to other electronic devices. The network interface may employ connectivity protocols, including but not limited to direct connection, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 802.11a / b / g / n / x, etc. The communication network may include, but is not limited to, direct interconnect, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), the Internet, etc.

[0044] One or more processors 204 may be configured to detect user immersion (or participation) in a metaverse environment. One or more processors 204 may be configured to: when user immersion in a metaverse environment is detected, monitor the operational state of each of the user's one or more IoT devices 201 in the user's real-world environment. One or more processors 204 are configured to: based on the monitored operational states of the one or more IoT devices 201, determine at least one IoT device from the one or more IoT devices 201 that requires user attention. Additionally, one or more processors 204 are configured to: generate an interactive avatar within the metaverse environment for the determined at least one IoT device requiring user attention, wherein the interactive avatar represents the operational state of the at least one IoT device.

[0045] Memory 210 may be communicatively coupled to one or more processors / controllers 204. Memory 210 may be configured to store data and instructions executable by one or more processors / controllers 204. Memory 210 may include, but is not limited to, non-volatile computer-readable storage media, such as various types of volatile and non-volatile storage media, including (but not limited to) random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media, etc. In one example, memory 210 may include a cache or random access memory of one or more processors / controllers 204. In alternative examples, memory 210 may be part of one or more processors / controllers 204, such as a processor's cache memory, system memory, or other memory. In some embodiments, memory may be an external storage device or database for storing data. Memory 210 may be operable to store instructions executable by one or more processors / controllers 204. The functions, actions, or tasks shown or described in the figures can be performed by executing the instructions stored in memory 210 by a programmed processor / controller. Functions, actions, or tasks are independent of a specific type of instruction set, storage medium, processor, or processing strategy, and can be executed by software, hardware, integrated circuits, firmware, microcode, etc., operating individually or in combination. Similarly, processing strategies can include multiprocessing, multitasking, parallel processing, etc.

[0046] Multiple modules 208 may be included in memory 210. Memory 210 may also include a system database 212 for storing data. Multiple modules 208 may include a set of instructions that can be executed to cause system 200 to perform one or more methods / procedures disclosed herein. Multiple modules 208 may be configured to use data stored in system database 212 to perform steps of this disclosure for managing one or more IoT devices 201 in a metaverse environment, as discussed herein. In embodiments, each of the multiple modules 208 may be a hardware unit that may be located outside of memory 210. Furthermore, memory 210 may include an operating system 214 for performing one or more tasks of system 200, such as tasks performed by a general-purpose operating system 214 in a communications domain. In one embodiment, system database 212 may be configured to store information required by the multiple modules 208 and one or more processors / controllers 204 for managing one or more IoT devices 201 in a metaverse environment.

[0047] At least one of the multiple modules 208 can be implemented using an ML model. Functionality associated with ML can be executed using non-volatile memory, volatile memory, and one or more processors 204.

[0048] I / O interface 206 can use appropriate devices (such as, but not limited to, a monitor, keyboard, mouse, touch screen, microphone, speaker, etc.) to implement input and output to and from the system.

[0049] This disclosure also contemplates a computer-readable medium that includes instructions or receives and executes instructions in response to a propagated signal. Instructions can be sent or received over a network via a communication port or interface or using a bus (not shown). The communication port or interface can be part of one or more processors / controllers 204 or can be a separate component. The communication port can be created in software or can be a physical connection in hardware. The communication port can be configured to connect to a network, external media, a display, or any other component or combination thereof in electronic device 202. The connection to the network can be a physical connection such as a wired Ethernet connection or can be established wirelessly. Similarly, additional connections to other components of electronic device 202 can be physical or can be established wirelessly. The network can alternatively be directly connected to the bus. For the sake of brevity, the architecture and standard operation of operating system 214, memory 210, system database 212, one or more processors / controllers 204, and I / O interface 206 are not discussed in detail.

[0050] Figure 3 A block diagram of multiple modules of a system for managing one or more IoT devices in a metaverse environment, according to embodiments of the present disclosure, is shown.

[0051] refer to Figure 3 The multiple modules 208 may include, but are not limited to, a monitoring module 302, a determination module 304, a generation module 306, an update module 308, an initiating module 310, and a display module 312. The multiple modules 208 may be implemented through appropriate hardware and / or software applications.

[0052] The detection module 304 can be configured to detect user immersion in a metaverse environment. The metaverse environment can correspond to a virtual or digital environment containing an immersive and interconnected digital space where users can interact with each other and digital objects. The determination module 304 can detect user immersion in the metaverse environment based on whether the user is currently using a metaverse device (headset), sensors, and user profile information. The virtual reality (VR) headset (including sensors) may be sending events, and the metaverse session may confirm the user profile connected to the metaverse world. When user immersion in the metaverse environment is detected, the detection module can be configured to determine whether a user action is required to change the operating state of one or more IoT devices 201 based on the determined operating state. For example, when frost is forming inside a refrigerator, a user action is required to defrost the refrigerator. The detection module can be configured to detect user immersion in the metaverse environment when it is determined that a user action is required.

[0053] The monitoring module 302 can be configured to monitor the operational status of each of the user's one or more IoT devices 201 in the user's real-world environment when it detects that the user is immersed in the metaverse environment. In embodiments of this disclosure, the system distinguishes between the one or more IoT devices 201 in the real-world environment and the IoT devices in the metaverse environment to improve the accuracy and precision of the system 200 in managing the operational status of the one or more IoT devices 201. The operational status can correspond to the current conditions or modes in which the one or more IoT devices 201 are operating. For example, the operational status can be idle / standby mode, sleep mode, maintenance mode, security mode, on, off, etc. The real-world environment can correspond to the physical, tangible environment and context in which the user exists and interacts.

[0054] The determination module 304 is configured to: determine at least one IoT device that requires user attention from one or more IoT devices 201 based on the operating status of the monitored one or more IoT devices 201.

[0055] The generation module 306 can be configured to generate, within a metaverse environment, an interactive avatar of at least one IoT device that requires user attention. The interactive avatar represents the operational state of at least one IoT device. For example, an interactive avatar of a refrigerator is generated when frost is forming inside. The interactive avatar of the refrigerator may be vibrating to indicate that frost is forming inside. In generating the interactive avatar of the at least one IoT device requiring user attention, one or more processors 204 can be configured to determine one or more device parameters associated with each of the at least one IoT device. The one or more device parameters may include temperature, speed, cooler, etc. Furthermore, the generation module 306 can be configured to generate the interactive avatar of at least one IoT device using a virtual-generation-based artificial intelligence (AI) model, based on the determined one or more device parameters, one or more required actions, a predefined 3D model library, and a predefined gesture library. The predefined 3D model library may correspond to a repository of 3D objects, characters, environments, and assets used to generate the interactive avatar. A predefined gesture library can be a collection of pre-designed or predefined body movements or gestures that can be used to generate interactive virtual avatars.

[0056] The update module 308 can be configured to receive one or more inputs from the user in the form of real-time interaction with the generated interactive virtual avatar within the metaverse environment. While receiving one or more inputs from the user in the form of real-time interaction with the generated interactive virtual avatar, the update module 308 can be configured to monitor one or more virtual avatar movements and one or more virtual avatar actions within the metaverse environment. For example, one or more virtual avatar movements and one or more virtual avatar actions could be a refrigerator vibrating to indicate that frost is forming inside the refrigerator.

[0057] The update module 308 can be configured to monitor one or more user movements and one or more user actions in the metaverse environment. In embodiments of this disclosure, one or more user actions correspond to multimodal actions. Multimodal actions can correspond to actions or behaviors that simultaneously involve multiple sensory modalities or input channels, such as touch and voice input, voice commands with gestures, multi-touch gestures, etc. System 200 can scan the metaverse environment to find one or more forms of interactive virtual avatars and one or more user actions that can originate from other participating users or central entities. System 200 also analyzes activities performed by at least one object located near a user in the metaverse environment (e.g., the vibration of a refrigerator to indicate frost inside the refrigerator). For example, when the interactive virtual avatar of the refrigerator can vibrate to indicate that frost is forming inside the refrigerator, one or more user movements and one or more user actions could be placing objects on the refrigerator in the metaverse environment to initiate a defrost mode.

[0058] Upon receiving one or more inputs from the user in the form of a real-time interaction with the generated interactive virtual avatar, the update module 308 can be configured to: recognize one or more user gestures by using a gesture detection-based AI model, correlated with one or more virtual avatar movements, one or more virtual avatar actions, one or more user movements, and one or more user actions. The one or more gestures may correspond to user intentions for changing operational states. Recognizing one or more gestures avoids misunderstanding of user intentions. For example, in a metaverse environment, the action of placing something on a refrigerator is recognized as a gesture for activating the refrigerator's defrost mode. The update module 308 can also be configured to detect one or more IoT commands by correlating the recognized one or more gestures with a predefined IoT command library.

[0059] The update module 308 can be configured to update the operational state of at least one IoT device in a real-world environment based on one or more received inputs. For example, when a user places something on a refrigerator in a metaverse environment, the refrigerator's defrost mode is automatically activated. In updating the operational state of each of the at least one IoT device, the update module 308 can be configured to send one or more detected IoT commands to each of the at least one IoT device. The at least one IoT device can execute one or more IoT commands to update its operational state.

[0060] System 200 can decompose the user's metaverse environment into device capabilities that serve as input in another phase. Skeleton-based motion recognition is a computer vision task that involves recognizing human actions from a sequence of 3D skeletal joint data captured as a template. By associating human behavior with this template, the recognition of corresponding actions can be facilitated. Based on actions (e.g., a handshake in the case of a refrigerator avatar), deforestation IoT commands can be mapped and sent to electronic devices to take appropriate actions. System 200 can use an AI model to treat device capabilities as input and determine whether to forward one or more device parameters as metaverse data to identify the best possible scenarios to maximize user comfort with appropriate interactions and interactive avatars. In embodiments of this disclosure, the AI ​​model can prioritize device events based on whether the event requires immediate user attention. The AI ​​model can be trained using relevant data based on compatibility scores. The AI ​​model can then decide whether the event needs to be propagated to the metaverse world.

[0061] The initiating module 310 can be configured to, upon detecting that a user is immersed in the metaverse environment, determine whether the current task the user is performing in the metaverse environment is a critical task or a non-critical task by analyzing one or more metaverse tasks. For example, one or more metaverse tasks may include the user being in focused mode, attending a meeting, making a call, or using one or more applications (e.g., a game application).

[0062] Initiation module 310 can be configured to: determine one or more required user actions to change the operational state of at least one IoT device when it is determined that the current task is a non-critical task. For example, when an interactive virtual avatar of a refrigerator can vibrate to indicate that frost is forming inside the refrigerator, one or more required user actions could be to activate defrost mode. Initiation module 310 can be configured to generate a compatibility score associated with each of the one or more required user actions based on a predefined priority list and the monitored operational state. The predefined list can include the weight of each task to be performed by the user in the metaverse environment. The compatibility score can represent the priority of performing one or more user-required actions relative to the criticality of the current task. Initiation module 310 can be configured to determine whether the generated compatibility score is greater than a predefined threshold score. Initiation module 310 can also be configured to: initiate a metaverse session to change the operational state of at least one IoT device when it is determined that the generated compatibility score is greater than the predefined threshold score.

[0063] Display module 312 can be configured to generate a metaverse representation of an operational state in a metaverse environment by using a virtual-generated AI model, based on one or more determined device parameters, one or more required actions, a predefined 3D model library, and a predefined gesture library. The operational state can represent one or more actions to be performed by each of at least one IoT device in the metaverse environment. Display module 312 can be configured to correlate the generated interactive virtual avatar with the generated metaverse representation of the operational state. Display module 312 can be configured to display the generated interactive virtual avatar and the metaverse representation of the operational state after correlation. For example, when frost is forming inside a refrigerator, the metaverse representation of the frost-forming state of the refrigerator in the metaverse environment is the refrigerator's vibration. (See below for at least...) Figure 4A , Figure 4B , Figure 5A and Figure 5B This describes details about other modules of the system (e.g., the character selection engine, the real-world event router, etc.) and the operation of the system 200 for managing one or more IoT devices 201 in the metaverse environment.

[0064] Figure 4A and Figure 4B A block diagram illustrating the operation of a system for managing one or more IoT devices in a metaverse environment according to various embodiments of the present disclosure is shown. For brevity, they are described together. Figure 4A and Figure 4B For example, at least refer to Figure 2 and Figure 3 As illustrated, the multiple modules 208 of system 200 help manage one or more IoT devices 201 in the metaverse environment.

[0065] Figure 4A A block diagram is shown for generating an interactive virtual avatar for at least one IoT device.

[0066] refer to Figure 4A At operation 402, system 200 acquires the device capabilities of at least one IoT device. Device capabilities may include the type of at least one IoT device, the current operating state of at least one IoT device, and any changes required by at least one IoT device. Furthermore, system 200 selects an interactive virtual avatar for at least one IoT device based on the acquired IoT device capabilities.

[0067] At operation 404, one or more desired actions of at least one IoT device are determined based on the acquired device capabilities. System 200 identifies relevant attributes associated with the identified one or more desired actions to be attached to the interactive virtual avatar. At operation 406, system 200 determines relevant behaviors (e.g., trembling, suffocation, etc.) of at least one IoT device to be attached to the interactive virtual avatar based on the acquired device capabilities. The relevant attributes and behaviors may correspond to a metaverse representation of the operational state in the metaverse environment. Furthermore, the identified relevant attributes and determined relevant behaviors of at least one IoT device are stored in content database 408.

[0068] At operation 410, the virtual avatar and device capability mapper of system 200 maps device capabilities to the relevant attributes and behaviors of at least one IoT device. Furthermore, at operation 412, the role selection engine of system 200 selects an interactive virtual avatar based on the result of operation 410. At operation 414, the inference / generation module 306 uses a predefined 3D model library 416, a predefined gesture library 418, and a predefined behavior library 420 to generate 3D model features associated with the interactive virtual avatar. At operation 422, the interactive virtual avatar is rendered based on the generated 3D model features.

[0069] Figure 4B A block diagram is shown for facilitating interactions between a user and one or more IoT devices 201 in a real-world environment during a metaverse session.

[0070] refer to Figure 4B At operation 424, the user interacts with a rendered interactive avatar in the metaverse environment. At operation 426, the Avatar Motion Control System (AACS) of System 200 monitors one or more avatar actions and one or more avatar movements of the interactive avatar. At operation 428, the User Motion Control System (UACS) of System 200 monitors one or more user actions and one or more user movements in the metaverse environment. At operation 430, a predefined gesture library is stored in a database along with potential overlapping assets. A list of possible body parts extracted from different available characters can be provided to the user. When a selection is made, the avatar skeleton is updated by loading that asset.

[0071] Database 432 can store the physical characteristics of a knowledge base for interactive virtual avatars. The system can combine virtual avatars (graphical representations of users or characters) with a knowledge base (information repository). The knowledge base is used to provide users with a more interactive and visually appealing way to access information. Skeletal template 434 can be used to recognize gestures of the interactive virtual avatar based on interactions between the avatar and at least one IoT device. In embodiments of this disclosure, the recognized gestures can be mapped to corresponding IoT commands. System 200 can recognize user gestures by using one or more actions performed by the interactive virtual avatar, with the aid of motion library 436. In embodiments of this disclosure, the creation of reusable virtual avatars focuses on implementing defined motion sequences in each case, allowing the utilization of commonalities between different characters. For example, any virtual avatar with arms and hands can perform a general waving gesture, while the head movements of the virtual avatars may differ in speed and distance, but still maintain a common direction.

[0072] At operation 438, the user avatar interaction inference module / update module 308 identifies one or more user gestures by associating one or more avatar movements, one or more avatar actions, one or more user movements, and one or more user actions. At operation 440, the IoT command mapping engine / update module 308 is configured to detect one or more IoT commands by associating the identified one or more gestures with a predefined IoT command library.

[0073] Figure 5A and Figure 5B Illustrations show use case scenarios for managing one or more IoT devices 201 in a metaverse environment, according to various embodiments of this disclosure. (See at least the references provided.) Figure 2 and Figure 3 As illustrated, the multiple modules 208 of system 200 help manage one or more IoT devices 201 in the metaverse environment.

[0074] Figure 5A A use case scenario for a system 200 for facilitating interaction between a user in a metaverse session and a refrigerator in a real-world environment is shown.

[0075] refer to Figure 5AAt operation 502, frost begins to form inside the refrigerator / freezer. At operation 504, the capability receiver / update module 308 can receive device capabilities. Device capabilities are shown in Table 1. At operation 506, the initiation module 210 can determine whether the current task the user is performing in the metaverse environment is a critical or non-critical task by analyzing one or more metaverse tasks. At operation 508, the metaverse session initiator of system 200 determines that the user is in the metaverse environment and initiates a metaverse session. System 200 can generate a compatibility score associated with each of one or more desired user actions for changing the operational state of one or more IoT devices 201, as shown in Table 2.

[0076] At operation 510, generation module 306 generates an interactive virtual avatar of at least one IoT device based on the received device capabilities. For example, the interactive virtual avatar could be a three-dimensional (3D) virtual avatar 511. At operation 512, update module 308 monitors one or more user movements and one or more user actions in the metaverse environment. In the current scenario, one or more user actions involve placing an object on the interactive virtual avatar of the refrigerator to initiate the refrigerator's defrost mode. At operation 514, update module 308 detects one or more IoT commands based on the monitoring results of one or more user movements and one or more user actions. In the current scenario, one or more user actions involve placing an object on the interactive virtual avatar of the refrigerator to initiate the refrigerator's defrost mode, as shown at 515. The real-world event router 516 of system 200 sends one or more IoT commands to the IoT device command mapper 518 of system 200. The IoT device command mapper (capability receiver) 518 sends one or more IoT commands to at least one IoT device, causing at least one IoT device to update its operational state. In the current scenario, at operation 520, the refrigerator automatically starts the defrost mode based on one or more IoT commands received.

[0077] In another use case scenario, a user is shopping in a metaverse world while the refrigerator is frosting up inside. System 200 acquires the refrigerator's device capabilities and identifies the priority of the ongoing tasks the user is performing in the metaverse session. System 200 generates an interactive virtual avatar of the refrigerator and notifies the user. System 200 then translates the user's actions into commands that the refrigerator can execute. In the current scenario, the user picks up a cloth and covers the refrigerator in the existing metaverse session to activate the refrigerator's defrost mode. The refrigerator automatically activates defrost mode based on the user's action.

[0078] In another use case, a user is reading a book in a metaverse session when the air conditioner shuts off due to an error code (automatic cleaning required). System 200 acquires the air conditioner's device capabilities and identifies the priority of the ongoing task the user is performing in the metaverse session. System 200 generates an interactive virtual avatar of the air conditioner (i.e., a person feeling cold) and notifies the user. System 200 then translates the user's action into commands that the air conditioner can execute. In the current scenario, the user picks up a cloth and wipes the face of the interactive virtual avatar in the existing metaverse session to activate the air conditioner's automatic cleaning mode. The air conditioner automatically starts its automatic cleaning mode.

[0079] In another use case scenario, a user is busy in a meeting, but a robotic cleaner (vacuum cleaner) is about to start, which may disrupt the meeting. System 200 acquires the robotic cleaner's device capabilities and identifies the priority of the ongoing tasks the user is performing in the metaverse session. System 200 generates an interactive virtual avatar of the robotic cleaner and notifies the user. System 200 then translates the user's actions into commands that the robotic cleaner can execute. In the current scenario, the user performs a stop action in the existing metaverse session to turn off the robotic cleaner. Therefore, the robotic cleaner automatically shuts down and stops cleaning.

[0080] Figure 5B An illustration is shown to facilitate interaction between a user and a refrigerator in a real-world environment during a metaverse session.

[0081] refer to Figure 5BAt operation 522, frost begins to form inside the refrigerator / freezer. At operation 524, the capability receiver / update module 308 receives device capabilities. At operation 526, the initiation module 310 determines whether the event (i.e., the refrigerator / freezer begins to frost) requires user action. At operation 528, the metaverse event router of system 200 connects the real-world environment and the metaverse environment and initiates a metaverse session. At operation 530, the session manager interface of system 200 manages the metaverse session based on device capabilities. At operation 532, the generation module 306 generates an interactive virtual avatar of the refrigerator based on the received device capabilities. At operation 534, the update module 308 monitors one or more user movements and one or more user actions in the metaverse environment. In the current scenario, one or more user actions are to place them on the interactive virtual avatar of the refrigerator to initiate the refrigerator's defrost mode, as shown at operation 536. At operation 538, the update module 308 detects one or more IoT commands based on the results of monitoring one or more user movements and one or more user actions. The IoT device command mapper 540 of system 200 sends one or more IoT commands to at least one IoT device, causing the at least one IoT device to update its operating status. In the current scenario, at operation 542, the refrigerator automatically starts the defrost mode based on the received one or more IoT commands.

[0082] Figure 6 A process flowchart depicting a method for managing one or more IoT devices in a metaverse environment according to embodiments of the present disclosure is shown.

[0083] refer to Figure 6 Method 600 can be executed by system 200 implemented in electronic device 202, such as Figure 2 and Figure 3 As shown.

[0084] At operation 602, method 600 includes detecting user immersion in the metaverse environment.

[0085] At operation 604, method 600 includes: when it is detected that a user is immersed in a metaverse environment, monitoring the operational state of each of one or more IoT devices 201 of the user in the user's real-world environment.

[0086] At operation 606, method 600 includes: determining, based on the operational status of one or more IoT devices 201 monitored, at least one IoT device from the one or more IoT devices 201 that requires user attention.

[0087] At step 608, method 600 includes: generating an interactive avatar within a metaverse environment for at least one determined IoT device that requires user attention. In embodiments of this disclosure, the interactive avatar represents the operational state of at least one IoT device.

[0088] At operation 610, method 600 includes: receiving one or more inputs from a user in a real-time interactive form with a generated interactive virtual avatar within a metaverse environment. While receiving one or more inputs from the user, method 600 includes monitoring one or more virtual avatar movements and one or more virtual avatar actions of the generated interactive virtual avatar within the metaverse environment. Method 600 may include monitoring one or more user movements and one or more user actions within the metaverse environment, wherein the one or more user actions correspond to multimodal actions. Method 600 may include: recognizing one or more user gestures by using a gesture detection-based AI model, by associating one or more virtual avatar movements, one or more virtual avatar actions, one or more user movements, and one or more user actions. Method 600 may also include detecting one or more IoT commands by associating the recognized one or more gestures with a predefined IoT command library.

[0089] At operation 612, the operational state of at least one IoT device in a real-world environment is updated based on one or more received inputs. Updating the operational state of the at least one IoT device may include sending one or more detected IoT commands to each of the at least one IoT device. The at least one IoT device may execute one or more IoT commands to update its operational state.

[0090] Although described in a specific order Figure 6 The operations described above are illustrated, but according to various embodiments of this disclosure, these operations may occur in a varying order. Furthermore, for the sake of brevity, details regarding [the specific operations described above] will not be discussed further here. Figure 6 The details of various operations are already discussed in the article. Figures 1A to 1C , Figure 2 , Figure 3 , Figure 4A , Figure 4B , Figure 5A and Figure 5B It is covered in the description.

[0091] This disclosure provides various technological advancements based on the aforementioned key features. This disclosure generates an interactive virtual avatar of at least one IoT device requiring user attention within a metaverse environment. This disclosure detects one or more IoT commands to be executed by one or more IoT devices 201 to update the operational state of each of the at least one IoT device. Because this disclosure allows users to change the operational state of each of the at least one IoT device without exiting the current metaverse session, users are able to complete one or more metaverse tasks within the current metaverse session (e.g., playing games, attending meetings, etc.).

[0092] Multiple modules 208 can be implemented using any suitable hardware and / or instruction set. Furthermore, Figure 2 The illustrated sequence is exemplary in nature, and embodiments may include steps that can be added or omitted as needed. In embodiments, the processor / controller may perform one or more operations performed by the plurality of modules 208 as needed.

[0093] It should be recognized that the various embodiments of this disclosure described in the claims and specification can be implemented in hardware, software, or a combination of hardware and software.

[0094] Any such software may be stored in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules) including computer-executable instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the methods of this disclosure.

[0095] Any such software may be stored in the form of volatile or non-volatile memory (e.g., a storage device such as read-only memory (ROM), whether erasable or rewritable), or in the form of memory (e.g., random access memory (RAM), memory chips, devices, or integrated circuits), or on optical or magnetic readable media such as CDs, DVDs, magnetic disks, or magnetic tapes. It should be understood that storage devices and storage media are various embodiments of non-transitory machine-readable storage suitable for storing one or more computer programs including instructions that, when executed, implement various embodiments of this disclosure. Therefore, various embodiments are provided in this specification including programs including code for implementing the means or methods claimed by any one of the claims, and non-transitory machine-readable storage of such programs.

[0096] Although this disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this disclosure as defined by the appended claims and their equivalents.

Claims

1. A method for managing one or more Internet of Things (IoT) devices (201) via electronic devices (202) in a metaverse environment, the method comprising: Detection of user participation in the metaverse environment (602); When the user is detected to be participating in the metaverse environment, the operational status of each of the one or more IoT devices (201) in the real-world environment is monitored (604); Based on the operational status of each of the monitored one or more IoT devices (201), determine (606) at least one IoT device from the one or more IoT devices (201) that requires user attention; as well as Within the metaverse environment, an interactive virtual avatar of at least one IoT device that requires user attention is generated (608), wherein the interactive virtual avatar represents the operating state of the at least one IoT device.

2. The method according to claim 1, further comprising: Within the metaverse environment, the user receives (610) one or more inputs in the form of real-time interaction with the generated interactive virtual avatar; as well as (612) Update (612) the operational state of the at least one IoT device in the real-world environment based on one or more received inputs.

3. The method according to claim 2, wherein, The user receives one or more inputs in the form of real-time interaction with the generated interactive virtual avatar, including: Monitoring the movement and actions of one or more interactive virtual avatars generated within the metaverse environment; and The metaverse environment monitors one or more user movements and one or more user actions, wherein the one or more user actions correspond to multimodal actions.

4. The method according to claim 3, wherein, The user receives one or more inputs in the form of real-time interaction with the generated interactive virtual avatar, including: By using a gesture detection-based artificial intelligence (AI) model, one or more user gestures are identified by correlating the movements of one or more virtual avatars, the actions of one or more virtual avatars, the movements of one or more user individuals, and the actions of one or more user individuals; and One or more IoT commands are detected by associating one or more recognized gestures with a predefined IoT command library.

5. The method according to claim 4, wherein, Updating the operational status of the at least one IoT device includes: Send one or more detected IoT commands to each of the at least one IoT device, wherein the one or more IoT commands are executed by the at least one IoT device to update the operational state.

6. The method according to claim 1, wherein, The detection of user participation in the metaverse environment includes: Based on the determined operating state, determine whether user action is required to change the operating state of the one or more IoT devices; and When it is determined that the user action is required, the user's participation in the metaverse environment is detected.

7. The method according to claim 1, further comprising: When the user is detected to be participating in the metaverse environment, one or more metaverse tasks are analyzed to determine whether the current task being performed by the user in the metaverse environment is a critical task or a non-critical task. as well as When it is determined that the current task is a non-critical task, one or more required user actions are determined to change the operating state of the at least one IoT device.

8. The method according to claim 7, further comprising: A compatibility score is generated based on a predefined priority list and the monitored operation status, which is associated with each of the one or more required user actions, wherein the compatibility score represents the priority of performing the one or more required user actions relative to the criticality of the current task; Determine whether the generated compatibility score is greater than a predefined threshold score; and When it is determined that the generated compatibility score is greater than the predefined threshold score, a metaverse session is initiated to change the operating state of the at least one IoT device.

9. The method according to claim 1, wherein, The method for generating an interactive virtual avatar of at least one determined IoT device that requires user attention includes: Determine one or more device parameters associated with each of the at least one IoT device; and By using a virtual-generative artificial intelligence (AI) model, an interactive virtual avatar of at least one IoT device is generated based on one or more determined device parameters, one or more required actions, a predefined 3D model library, and a predefined gesture library.

10. The method of claim 9, further comprising: By using the virtual-generated AI model, a metaverse representation of the operation state is generated in the metaverse environment based on the determined one or more device parameters, the one or more required actions, the predefined 3D model library, and the predefined gesture library, wherein the operation state represents one or more actions to be performed by each of the at least one IoT device in the metaverse environment; The generated interactive virtual avatar is correlated with the metaverse representation of the generated operational state; and Following this, a metaverse representation of the generated interactive virtual avatar and the operational state is displayed.

11. An electronic device (202) for managing one or more Internet of Things (IoT) devices (201) in a metaverse environment, said electronic device comprising: Memory (210), stores instructions; One or more processors (204) are communicatively coupled to the memory (210). When the instruction is executed by the one or more processors (204), it causes the electronic device (202) to perform an operation, the operation including: Detection is performed on user participation in the metaverse environment; When the user is detected to be participating in the metaverse environment, the operational status of each of the one or more IoT devices (201) in the real-world environment is monitored; Based on the operational status of each of the monitored one or more IoT devices (201), at least one IoT device that requires user attention is identified from the one or more IoT devices (201); and Within the metaverse environment, an interactive virtual avatar of at least one IoT device that requires user attention is generated, wherein the interactive virtual avatar represents the operational status of the at least one IoT device.

12. The electronic device according to claim 11, wherein, The operation further includes at least one operation of the method according to any one of claims 2 to 10.

13. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors (204) of an electronic device (202), cause the electronic device (202) to perform operations, the operations including: Detection of user participation in the metaverse environment; When the user is detected to be participating in the metaverse environment, the operational status of each of the one or more Internet of Things (IoT) devices (201) in the real-world environment is monitored; Based on the operational status of each of the monitored one or more IoT devices (201), at least one IoT device that requires user attention is identified from the one or more IoT devices (201); as well as Within the metaverse environment, an interactive virtual avatar of at least one IoT device that requires user attention is generated, wherein the interactive virtual avatar represents the operational status of the at least one IoT device.

14. One or more non-transitory computer-readable storage media according to claim 13, wherein, The operation further includes at least one operation of the method according to any one of claims 2 to 10.

Images