Methods and systems for controlling operation of devices in an internet of things (IOT) environment

Abstract

The method disclosed herein includes identifying a first device pointed at by a user for control. The method includes determining at least one second device related to a context on the first device. The method includes recommending at least one suggestion for controlling at least one operation of at least one of the first device and the at least one second device based on a correlation between a capability of the first device and a current operating context of the at least one second device.

Description

Technical Field

[0001] This disclosure relates to the field of Internet of Things (IoT) environments, and more specifically to controlling the operation of devices in an IoT environment based on the correlation between devices. Background Technology

[0002] Typically, an Internet of Things (IoT) environment includes multiple devices with which a user can interact and control their operation. In a conventional approach, a user can point to a device for control using ultra-wideband (UWB) technology supported by the user device. When pointing at a device, the user device controls that device. In examples, normal control may include at least one of turning the pointed device on / off, setting the pointed device, etc. However, the context of other devices that depend on the pointed device may affect the operation of the pointed device. Summary of the Invention

[0003] Technical issues

[0004] Consider an example scenario where a user can control a washing machine (an example device deployed in an IoT environment) by pointing their user device at it using UWB technology supported by the user device. In this scenario, the user device provides normal controls for operating the washing machine (such as turning it on, modifying washing machine settings, etc.). However, a low water level in the tank may affect the operation of the washing machine.

[0005] Therefore, conventional methods do not involve providing users with suggestions for controlling the pointed device based on the current context of one or more other devices in an IoT environment that depend on the pointed device.

[0006] Technical solutions

[0007] The primary objective of the embodiments herein is to disclose methods and systems for controlling the operation of multiple devices in an Internet of Things (IoT) environment.

[0008] Another objective of embodiments of this document is to disclose a method and system for identifying a first device among a plurality of devices for pointing to, in order to control and determine at least one second device in relation to the context of the first device.

[0009] Another objective of embodiments herein is to disclose a method and system for recommending at least one suggestion for controlling at least one operation of at least one of the at least one second device and at least one of the first devices, based on the correlation between the capabilities of a first device and the current operating context of at least one second device.

[0010] Therefore, embodiments of this document provide methods and systems for controlling the operation of devices in an Internet of Things (IoT) environment. The method disclosed herein includes identifying a first device in response to a user pointing an electronic device at the first device. The method includes determining at least one second device in relation to the context of the first device. The method includes recommending at least one suggestion for controlling at least one operation of at least one of the at least one second device and at least one of the first device based on the correlation between the capabilities of the first device and the current operating context of the at least one second device.

[0011] The first device can be pointed to by the user using ultra-wideband (UWB) supported by electronic devices.

[0012] Determining the at least one second device may include: identifying at least one first group of related devices for the first device using a capability-based device clustering model; identifying at least one second group of related devices for the first device using a device interaction prediction model; creating a union of the at least one first group of related devices and the at least one second group of related devices, wherein the created union is a set including at least one device present in either the at least one first group of related devices or the at least one second group of related devices; and determining the at least one device present in either the at least one first group of related devices or the at least one second group of related devices as the at least one second device related to the context of the first device.

[0013] Identifying the at least one first group of related devices for the first device may include: obtaining a list of registered devices in an IoT environment from a device database, wherein the device database includes a list of devices registered by a user when deploying each device in the IoT environment and associated capabilities; using the device database to identify the capabilities of the first device and each registered device; and forming a cluster of devices for the first device based on the identified capabilities of the first device and each registered device, wherein the formed cluster of devices includes the at least one first group of related devices from the list of registered devices that have similar capabilities to the first device.

[0014] Identifying the at least one second group of related devices for the first device may include: obtaining a list of registered devices from a device database; identifying the capabilities of the first device and each registered device using the device database; calculating and assigning a confidence score for each registered device to the first device by matching the capabilities of each registered device with the capabilities of the first device; discarding at least one device from the list of registered devices that is associated with a confidence score less than a device interaction threshold; and identifying at least one group of devices from the list of registered devices that are associated with a confidence score greater than or equal to the device interaction threshold as the at least one second group of related devices for the first device.

[0015] The at least one suggestion recommended by the electronic device for controlling at least one operation of at least one of the at least one second device and at least one first device may include: identifying the type of data collected from the first device and the at least one second device, wherein the data for each device corresponds to at least one of the following: the state, content, location, and current operating context of the corresponding device; collecting the identified type of data from the first device and the at least one second device in online or offline mode; determining the correlation between the first device and the at least one second device based on the data collected from the first device and the at least one second device and the capabilities of the first device and the at least one second device; and recommending the at least one suggestion for controlling at least one operation of at least one of the at least one second device based on the determined correlation between the first device and the at least one second device.

[0016] Identifying the types of data to be collected from the first device and the at least one second device may include: creating at least one device-capability pair, wherein each device-capability pair includes a capability and at least one of the at least one second device and the first device that interacts through the capability; querying a device capability mapping database based on each device-capability pair, wherein the device capability mapping database includes a mapping of each capability to type of data; and receiving from the device capability mapping database data on the types of data to be collected for each capability corresponding to at least one of the at least one second device and the first device.

[0017] Collecting identified type data from the first device and the at least one second device in online mode may include: establishing a session with an IoT cloud server, wherein the IoT cloud server collects and stores data from the first device and the at least one second device; sending a data request indicating the type of data required by the first device and the at least one second device to the IoT cloud server; and receiving the type of data indicated by the first device and the at least one second device from the IoT cloud server.

[0018] Collecting data of the identified type from the first device and the at least one second device in offline mode may include: establishing a session with the first device and the at least one second device using a communication network supported by the electronic device, the first device and the at least one second device; sending a data request indicating the type of data to the first device and the at least one second device; and receiving the data indicating the type of data from the first device and the at least one second device in response to sending the data request.

[0019] Determining the correlation between the first device and the at least one second device may include: determining a plurality of correlation features based on the capabilities of the first device and the at least one second device and at least one data collected from the first device and the at least one second device, wherein the plurality of correlation features include at least one of the following: action continuity features between the first device and the at least one second device, conflict resolution features, fault resolution features, and screen intelligence features.

[0020] Determining the action continuity features between the first device and the at least one second device may include: training an action continuity model based on a labeled dataset of candidate associated devices and associated device context and capabilities; and using the trained action continuity model to process the capabilities of the first device and the at least one second device and the collected data to determine the action continuity features between the first device and each of the at least one second device.

[0021] The method may include determining an action continuity relevance index for each of the at least one of the at least two second devices (104b-104n) with respect to the first device (104a) by using a trained action continuity model to process the capabilities of the first device (104a) and each of the second devices (104b-104n) and collected data, wherein the action continuity relevance index for each of the second devices (104b-104n) indicates the probability of continuing at least one action of each of the second devices (104b-104n) on the first device.

[0022] Determining the conflict resolution characteristics between the first device and the at least one second device may include: training a first reinforcement model based on at least one predefined conflict strategy to detect at least one conflict between the operation of the first device and the at least one second device; and processing data collected from each of the first device and the at least one second device using the trained first reinforcement model to determine the conflict resolution characteristics between the first device and each of the at least one second device, wherein the conflict resolution characteristics depict at least one conflict between the operation of the first device and each of the at least one second device.

[0023] The method may include processing data collected from a first device (104a) and each of the second devices (104b-104n) using a trained first reinforcement model to determine a conflict resolution relevance index for each of the second devices (104b-104n) with respect to the first device (104a), wherein the conflict resolution relevance index determined for each of the second devices (104b-104n) depicts the probability of conflict between the operation of the first device (104a) and each of the second devices.

[0024] Determining the fault resolution characteristics between the first device and the at least one second device may include: training a second reinforcement model based on at least one predefined fault strategy to detect faults in at least one operation of the first device with respect to the current operating context of the at least one second device; and processing data collected from the first device and each of the at least one second device using the trained second reinforcement model to determine the fault resolution characteristics between the first device and each of the at least one second device, wherein the fault resolution characteristics between the first device and each second device describe at least one operation of the first device that is expected to fail with respect to the current operating context of each second device.

[0025] The method may include processing data collected from a first device and each second device using a trained second reinforcement model to determine a fault resolution relevance index for each second device regarding the first device, wherein the fault resolution relevance index determined for each second device depicts the probability of a fault in at least one operation of the first device with respect to the current operating context of each second device.

[0026] The recommendation of at least one suggestion may include: ranking each of a plurality of relevance features based on a relevance index associated with the corresponding relevance feature, wherein the relevance index includes at least one of an action continuity relevance index, a conflict resolution relevance index, and a fault resolution relevance index; creating at least one semantic action based on the ranking of the plurality of relevance features and at least one of data collected from the first device and the at least one second device; and recommending the created at least one semantic action as the at least one suggestion for controlling the at least one operation of the at least one second device and the at least one first device.

[0027] The recommendation of at least one suggestion may include rendering the at least one suggestion as content covered by a keyboard on the control interface of a first device on the electronic device.

[0028] The method may include: recommending a configuration for a first device (104a) by an electronic device (106) based on at least one type of content rendered on at least one second device (104b-104n); and recommending a configuration for the first device (104a) by the electronic device (106) based on data collected from the at least one second device (104b-104n) and the correlation between the first device (104a) and the at least one second device.

[0029] Therefore, embodiments of this document provide a method for multi-device interaction based on ultra-wideband in an Internet of Things (IoT) environment, the method comprising: providing a user interface on a user device belonging to a first IoT device; performing an operational capability of the first IoT device and the operational context of a plurality of second IoT devices operatively connected to the first IoT device; and providing recommendations on the user interface of the first IoT device to change the operation of at least one of the first IoT device and the second IoT devices as a result of the correlation.

[0030] Therefore, embodiments of this document provide a method for multi-device interaction in an Internet of Things (IoT) environment, the method comprising: identifying a first IoT device in response to a user pointing a user device to a first IoT device in the IoT environment; determining at least one second IoT device associated with the context of the first IoT device based on the correlation between the device capabilities of the first IoT device and the current operating context of a second IoT device; and recommending one or more semantic actions associated with one of the first IoT device and the at least one second IoT device to a user of the user device based on the correlation.

[0031] One or more semantic actions can be recommended through the control interface of the first IoT device on the user device.

[0032] The correlation can determine at least one of conflict resolution, operation continuity and fault resolution between the first IoT device and the at least one second IoT device.

[0033] Therefore, embodiments of this document provide an electronic device for controlling the operation of devices in an Internet of Things (IoT) environment, wherein the electronic device includes a memory and a controller coupled to the memory. The controller is configured to identify a first device in response to a user pointing the electronic device at the first device. The controller is also configured to determine at least one second device associated with the context of the first device. The controller is further configured to recommend at least one suggestion for controlling at least one operation of at least one of the at least one second device and at least one of the first devices, based on the correlation between the capabilities of the first device and the current operating context of the at least one second device.

[0034] The controller can be configured to: use a capability-based device clustering model to identify at least one first group of related devices for a first device; use a device interaction prediction model to identify at least one second group of related devices for the first device; create a union of the at least one first group of related devices and the at least one second group of related devices, wherein the created union is a set including at least one device present in either the at least one first group of related devices or the at least one second group of related devices; and determine at least one device present in either the at least one first group of related devices or the at least one second group of related devices as at least one second device related to the context of the first device.

[0035] The controller can be configured to: obtain a list of registered devices in an IoT environment from a device database, wherein the device database includes a list of devices and associated capabilities registered by a user when deploying each device in the IoT environment; identify a first device (104a) and the capabilities of each registered device using the device database; and form a cluster of devices for the first device (104a) based on the identified first device (104a) and the capabilities of each registered device present in the list of devices, wherein the formed cluster of devices includes at least one first group of related devices from the list of registered devices that have similar capabilities to the first device.

[0036] The controller may be configured to: obtain a list of registered devices from a device database; identify the capabilities of a first device and each registered device using the device database; calculate and assign a confidence score for each registered device to the first device by matching the capabilities of each registered device with the capabilities of the first device; discard at least one device from the list of registered devices that is associated with a confidence score less than a device interaction threshold; and identify at least one device from the list of registered devices that is associated with a confidence score greater than or equal to the device interaction threshold as the at least one second group of related devices for the first device.

[0037] The controller may be configured to: identify types of data collected from a first device and the at least one second device, wherein the data for each device corresponds to at least one of the following: the state, content, location, and current operating context of the corresponding device; collect the identified types of data from the first device and the at least one second device in online or offline mode; determine the correlation between the first device and the at least one second device based on the data collected from the first device and the at least one second device and the capabilities of the first device and the at least one second device; and recommend at least one suggestion for controlling at least one operation of at least one of the at least one second device and the at least one first device based on the determined correlation between the first device and the at least one second device.

[0038] The controller can be configured to: create at least one device-capability pair, wherein each device-capability pair includes a capability and at least one of the at least one second device and a first device that interacts through the capability; query a device capability mapping database based on each device-capability pair, wherein the device capability mapping database includes a mapping of data for each capability and type of the capability; and receive data from the device capability mapping database for the type of capability to be collected for each capability corresponding to at least one of the at least one second device and the first device.

[0039] The controller may be configured to determine a plurality of correlation features based on the capabilities of the first device and the at least one second device and at least one of the data collected from the first device and the at least one second device, wherein the plurality of correlation features include at least one of the following: a continuity of action feature between the first device and the at least one second device, a conflict resolution feature, a fault resolution feature, and a screen intelligence feature.

[0040] The controller can be configured to: rank each of a plurality of relevance features based on a relevance index associated with the corresponding relevance feature, wherein the relevance index includes at least one of an action continuity relevance index, a conflict resolution relevance index, and a fault resolution relevance index; create at least one semantic action based on the ranking of the plurality of relevance features and at least one of data collected from a first device and the at least one second device; and recommend the at least one semantic action created as at least one suggestion for controlling at least one operation of at least one of the at least one second device and the at least one first device.

[0041] The controller can be configured to render the at least one suggestion as keyboard-covered content on the control interface of the first device on the electronic device.

[0042] The controller can be configured to: recommend a configuration for a first device based on at least one type of content rendered on the at least one second device; and recommend a configuration for the first device based on data collected from the at least one second device and the correlation between the first device and the at least one second device.

[0043] These and other aspects of the exemplary embodiments herein will be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. However, it should be understood that while the following description indicates exemplary embodiments and their many specific details, it is given by way of illustration rather than limitation. Many changes and modifications may be made within the scope of the exemplary embodiments herein without departing from the spirit of the embodiments herein, and the exemplary embodiments herein include all such modifications. Attached Figure Description

[0044] The embodiments described herein are illustrated in the accompanying drawings, throughout which the same reference numerals indicate corresponding portions in the various figures. The embodiments described herein will be better understood from the following description with reference to the accompanying drawings, wherein:

[0045] Figure 1a A system for the Internet of Things (IoT) according to embodiments disclosed herein is described;

[0046] Figure 1b A system for the Internet of Things (IoT) according to embodiments disclosed herein is described;

[0047] Figure 2 It is a block diagram depicting various components of an electronic device for controlling the operation of multiple devices according to embodiments disclosed herein;

[0048] Figure 3 A device controller executable in an electronic device for controlling the operation of a plurality of devices, according to embodiments disclosed herein, is described.

[0049] Figure 4 An example concept diagram depicts an IoT system according to embodiments disclosed herein, wherein each device is controlled based on the current operating context of the device based on its capabilities and dependencies;

[0050] Figure 5 These are example diagrams depicting context-dependent means for determining a targeted device according to embodiments disclosed herein;

[0051] Figure 6 This is an example diagram depicting the identification of the type of data collected from the pointed device and the context-related device according to the embodiments disclosed herein;

[0052] Figure 7 These are example diagrams depicting the collection of data from a pointed device and a context-dependent device in online and offline modes according to embodiments disclosed herein;

[0053] Figure 8a This is an example diagram depicting the training of an action continuity module according to embodiments disclosed herein and using the trained action continuity module to determine action continuity features between each context-dependent device and the directed device.

[0054] Figure 8b This is an example diagram depicting the training of an action continuity module according to embodiments disclosed herein and using the trained action continuity module to determine action continuity features between each context-dependent device and the directed device.

[0055] Figure 9aThis is an example diagram depicting the training of a first reinforcement module according to an embodiment disclosed herein and using the trained first reinforcement module to determine conflict resolution features between each context-dependent device and the pointed device.

[0056] Figure 9b This is an example diagram depicting the training of a first reinforcement module according to an embodiment disclosed herein and the use of the trained first reinforcement module to determine conflict resolution features between each context-dependent device and the pointed device.

[0057] Figure 10a This is an example diagram depicting the training of a second reinforcement module according to embodiments disclosed herein and using the trained second reinforcement module to determine fault resolution features between each context-dependent device and the pointed device;

[0058] Figure 10b This is an example diagram depicting the training of a second reinforcement module according to embodiments disclosed herein and using the trained second reinforcement module to determine fault resolution features between each context-dependent device and the pointed device;

[0059] Figure 11 These are recommended example diagrams depicting one or more semantic actions of a means for controlling a pointing device or an associated context-dependent means according to embodiments disclosed herein;

[0060] Figure 12 It is a flowchart depicting the operation of multiple devices in an IoT environment according to embodiments disclosed herein;

[0061] Figure 13a This document describes use case scenarios for controlling the operation of one of a plurality of devices in an IoT environment according to embodiments disclosed herein.

[0062] Figure 13b This document describes use case scenarios for controlling the operation of one of a plurality of devices in an IoT environment according to embodiments disclosed herein.

[0063] Figure 14a This document describes use case scenarios for controlling the operation of one of a plurality of devices in an IoT environment according to embodiments disclosed herein.

[0064] Figure 14b This document describes use case scenarios for controlling the operation of one of a plurality of devices in an IoT environment according to embodiments disclosed herein.

[0065] Figure 15a This document describes use case scenarios for controlling the operation of one of a plurality of devices in an IoT environment according to embodiments disclosed herein.

[0066] Figure 15bThis document describes use case scenarios for controlling the operation of one of a plurality of devices in an IoT environment according to embodiments disclosed herein.

[0067] Figure 16a This document describes use case scenarios for controlling the operation of one of a plurality of devices in an IoT environment according to embodiments disclosed herein.

[0068] Figure 16b This document describes a use case scenario for controlling the operation of one of a plurality of devices in an IoT environment according to embodiments disclosed herein. Detailed Implementation

[0069] The exemplary embodiments herein, along with their various features and advantageous details, are described more fully with reference to the non-limiting embodiments illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted to avoid unnecessarily obscuring the embodiments herein. The description herein is intended only to facilitate understanding of how the exemplary embodiments herein may be practiced, and further to enable those skilled in the art to practice the exemplary embodiments herein. Therefore, this disclosure should not be construed as limiting the scope of the exemplary embodiments herein.

[0070] Embodiments herein disclose methods and systems for controlling the operation of devices in an Internet of Things (IoT) environment based on the correlation between devices.

[0071] Now refer to the accompanying drawings, and more specifically to... Figure 1a Figure 16, in which similar reference characters consistently represent corresponding features throughout all figures, illustrates an example embodiment.

[0072] The embodiments herein may be used interchangeably with terms such as “first device,” “pointed device,” and “user-selected device” to refer to a user-pointable device for control in an Internet of Things (IoT) environment.

[0073] The embodiments herein may use interchangeably terms such as “second device,” “context-dependent / related device,” “associated device,” etc., to refer to a device that depends on the pointed device and affects the operation of the pointed device in the IoT environment.

[0074] The embodiments herein may be used interchangeably with terms such as “data,” “metadata,” “context data,” etc., to refer to at least one of the following: the state, location, device information, manufacturer-provided information, current operating context, etc., of each of the multiple devices present in an IoT environment.

[0075] The embodiments described herein may be used interchangeably throughout the document with terms such as “operation,” “action,” “task,” etc.

[0076] The embodiments described herein may be used interchangeably in the document with terms such as “semantic action,” “recommendation,” “suggestion,” “notification,” and “alert.”

[0077] Figure 1a and Figure 1b An Internet of Things (IoT) system 100 according to embodiments disclosed herein is described. The IoT system 100 includes an IoT cloud server 102, multiple devices 104a-104n, and an electronic device 106.

[0078] The IoT cloud server 102, multiple devices 104a-104n, and electronic device 106 can be connected to each other. In one example, the IoT cloud server 102, multiple devices 104a-104n, and electronic device 106 can be connected to each other using a communication network 108. The communication network 108 can include, but is not limited to, at least one of the following: a wired network, a value-added network, a wireless network, a satellite network, or a combination thereof. Examples of wired networks can be, but are not limited to, a local area network (LAN), a wide area network (WAN), Ethernet, etc. Examples of wireless networks can be, but are not limited to, a cellular network, Wi-Fi, Bluetooth, Bluetooth Low Energy, Zigbee, Wi-Fi Direct (WFD), Ultra Wideband (UWB), Infrared Data Association (IrDA), Near Field Communication (NFC), etc. In another example, electronic device 106 and multiple devices 104a-104n can be directly connected to each other (e.g., via direct communication, via an access point, etc.). In another example, electronic device 106 and multiple devices 104a-104n can be connected to IoT cloud server 102, and electronic device 106 can be connected to multiple devices 104a-104n via IoT cloud server 102. In another example, electronic device 106 can be connected to multiple devices 104a-104n using communication network 108, and electronic device 106 and multiple devices 104a-104n can be connected to IoT cloud server 102 using communication network 108. In another example, electronic device 106, multiple devices 104a-104n, and IoT cloud server 102 can be connected to each other via repeaters, hubs, and gateways. It should be understood that electronic device 106, multiple devices 104a-104n, and IoT cloud server 102 can be connected to each other in any of a variety of ways (including the methods described above), and can be connected to each other simultaneously in two or more of the various ways (including the methods described above).

[0079] The IoT cloud server 102 described herein can be a server that acquires, stores, and manages device information, capability mappings, manufacturer-provided information, and location information for each of the multiple devices 104a-104n existing in an IoT environment. Examples of IoT environments include, but are not limited to, smart home environments, smart office environments, and smart hospital environments. Device information may include, but is not limited to, identification values ​​(e.g., device ID information) and device types for each of the multiple devices 104a-104n. In the examples described herein, identification value / device ID information may include, but is not limited to, information such as, but not limited to, Media Access Control (MAC) identifiers (MAC IDs), serial numbers, and unique device IDs. Capability mappings include information about one or more capabilities of each of the multiple devices 104a-104n. Examples of capabilities of devices 104a-104n may include, but are not limited to, audio, video, display, energy limitations, and data sensing capabilities. Location information includes information about the location of each of the multiple devices 104a-104n. The location of devices (104a-104n) can indicate the area or room in the IoT environment where devices (104a-104n) are present (e.g., living room, kitchen, bedroom, study, children's room, hospital ward, conference room, factory unit, etc.). Information provided by the manufacturer of devices 104a-104n may include, but is not limited to, at least one of the following: the model number of device 104a-104n, the year of manufacture of device 104a-104n, the lifespan of device 104a-104n, etc. The IoT cloud server 102 can update the device information, capability mapping, and location information of device 104a-104n when any of the multiple devices 104a-104n are added or removed in the IoT environment.

[0080] The IoT cloud server 102 can also acquire and store information such as, but not limited to, power-on / off status and operations performed by each of the multiple devices 104a-104n. Examples of operations may include, but are not limited to, audio playback, video playback, light control, energy management, air purification, and sensing environmental factors (such as temperature, humidity, smoke, etc.). The IoT cloud server 102 can be configured as a hardware device independent of the electronic device 106, but is not limited thereto. The IoT cloud server 102 can be a component of the electronic device 106, or it can be a server designed to be classified as software.

[0081] The multiple devices 104a-104n can be IoT devices capable of exchanging information with each other and IoT devices capable of exchanging information with other devices (such as IoT cloud server 102, electronic device 106, etc.). Examples of the multiple devices 104a-104n can be, but are not limited to, smartphones, mobile phones, video phones, computers, tablet PCs, netbooks, laptops, wearable devices, vehicle infotainment systems, workstations, servers, personal digital assistants (PDAs), smart plugs, portable multimedia players (PMPs), MP3 players, mobile medical devices, lights, voice-assisted devices, cameras, home appliances, one or more sensors, etc. Examples of home appliances can be, but are not limited to, televisions (TVs), digital video disc (DVD) players, audio devices, refrigerators, air conditioners (ACs), air purifiers, chimneys, stoves, vacuum cleaners, ovens, microwave ovens, washing machines, dryers, set-top boxes, home automation control panels, security control panels, game consoles, electronic keys, portable cameras, electronic photo frames, coffee makers, toasters, rice cookers, pressure cookers, etc. Examples of sensors may include, but are not limited to, temperature sensors, humidity sensors, infrared sensors, gyroscope sensors, atmospheric sensors, proximity sensors, RGB sensors (brightness sensors), photoelectric sensors, thermostats, ultraviolet (UV) light sensors, dust sensors, fire detection sensors, carbon dioxide (CO2) sensors, smoke sensors, window contact sensors, water sensors, or any other equivalent sensors. The function of each sensor can be intuitively inferred by one of ordinary skill in the art based on its name, therefore its detailed description is omitted.

[0082] Multiple devices 104a-104n can be deployed in various locations, areas, or rooms within an IoT environment, allowing users to interact with and control the operation of each device 104a-104n.

[0083] Multiple devices 104a-104n can perform one or more operations / actions based on their capabilities. Examples of operations may include, but are not limited to, playing media (audio, video, etc.), capturing media, purifying air, performing cooling or heating of a defined area, controlling lights, sensing various environmental factors (e.g., temperature, smoke, humidity, etc.). Multiple devices 104a-104n can simultaneously perform one or more corresponding actions.

[0084] Once deployed in an IoT environment, multiple devices 104a-104n can register with the IoT cloud server 102 by transmitting device information, capabilities, manufacturer-provided information, and location information to the IoT cloud server 102 or the electronic device 106. Users can use the electronic device 106 to register multiple devices 104a-104n with the IoT cloud server 102.

[0085] Multiple devices 104a-104n can also transmit data / metadata to the IoT cloud server 102 or electronic device 106 via communication network 108. The data for each device (104a-104n) corresponds to, but is not limited to, at least one of the following: the state of the device (104a-104n) (i.e., power on / power off of device 104a-104n), the content being played on the device (104a-104n), the location of the device (104a-104n), device information of the device (104a-104n), information provided by the manufacturer of the device (104a-104n), the current operating context of the device (104a-104n), etc. In the examples herein, the content may include at least one of the following: media (such as audio, video, images, Graphics Interchange Format (GIF), etc.), text, web pages, etc. In the examples in this paper, the current operating context of the device (104a-104n) describes the operations performed by the device (104a-104n) at the current moment based on its capabilities.

[0086] The electronic device 106 referred to herein can be a device for controlling the operation of multiple devices 104a-104n. The electronic device 106 can be a user device used by a user to connect to and / or interact with and / or control the operation of multiple devices 104a-104n. Examples of the electronic device 106 can be, but are not limited to, smartphones, mobile phones, video phones, computers, tablet PCs, netbooks, laptops, wearable devices, personal digital assistants (PDAs), workstations, servers, IoT devices, or any other device that can be used to connect to and / or interact with and / or control the operation of multiple devices 104a-104n.

[0087] Electronic device 106 obtains, stores, and maintains device information, capability mappings, manufacturer-provided information, location information, etc., for each device (104a-104n) existing in the IoT environment by directly communicating with each device (104a-104n) via communication network 108. Optionally, electronic device 106 can obtain device information, capability mappings, manufacturer-provided information, location information, etc., for each device 104a-104n from IoT cloud server 102 via communication network 108. Furthermore, electronic device 106 maintains a separate database for storing device information, capability mappings, location information, etc., for each device (104a-104n) existing in the IoT environment. In the example herein, electronic device 106 maintains device database 212, device capability mapping database 214, etc. (e.g., ...). Figure 2(As depicted). The device database includes a mapping of a series / multiple devices 104a-104n with associated capabilities (registered with electronic device 106 or IoT cloud server 102 when each device is deployed in an IoT environment). The device capability mapping database includes a mapping of each capability in the capabilities to data / type data. The type data can be metadata associated with each device (104a-104n).

[0088] The electronic device 106 also obtains, determines, or generates control commands for controlling each of the multiple devices 104a-104n by utilizing device information, capability mapping, location information, etc. of each device (104a-104n). The electronic device 106 can send control commands to any device among the devices 104a-104n to perform operations based on the stored capabilities of the respective device 104a-104n. The electronic device 106 can receive the results of the operations performed according to the control commands from the device (104a-104n) performing the operations.

[0089] In an embodiment, electronic device 106 may be configured to control the operation of each of the plurality of devices 104a-104n based on the correlation between the plurality of devices 104a-104n.

[0090] To control the operation of each device (104a-104n), electronic device 106 identifies a first device (e.g., device 104a) among the plurality of devices 104a-104n that is pointed to for interaction and control. In one embodiment, an application deployed on electronic device 106 may point to the first device 104a for interaction and control. Examples of applications may be, but are not limited to, video streaming applications, audio applications, device-based control applications, etc. In another embodiment, a user may use an interactive application deployed on electronic device 106 to point to the first device 104a for interaction and control. The interactive application displays a user interface for one or more devices 104a-104n present in the IoT environment and enables the user to select / point to any device among devices 104a-104n. In another embodiment, a user may use ultra-wideband (UWB) communication supported by electronic device 106 to point to the first device 104a for interaction and control. In yet another embodiment, a user may use short-range communication supported by electronic device 106 to point to the first device 104a for interaction and control of its operation. Examples of short-range communication may include, but are not limited to, Bluetooth Low Energy (BLE), Wi-Fi, infrared wireless communication, Near Field Communication (NFC), Zigbee, Global Microwave Access Interoperability (WiMAX), or any other IEEE 802.15.4 compliant protocol. In another example, a user may use any other communication method supported by communication network 108 to point to the first device 104a.

[0091] Upon identifying the first device 104a, the electronic device 106 checks whether any other devices are present in the IoT environment. If no devices are present, the electronic device 106 provides normal control to control the operation of the first device 104a. Examples of normal control may include, but are not limited to, on / off commands, settings for the first device 104a, etc.

[0092] If multiple devices 104a-104n exist in an IoT environment, electronic device 106 identifies one or more second devices (e.g., one or more devices 104b-104n) for the first device 104a. The one or more second devices 104b-104n can be context-dependent devices or devices context-dependent with the first device 104a. In the examples herein, a context-dependent device may refer to a device having similar capabilities to the first device 104a. To identify one or more second devices 104b-104n, electronic device 106 identifies a first set of related devices and a second set of related devices for the first device 104a. The first set of related devices and the second set of related devices can be devices among the multiple devices 104a-104n that have capabilities matching those of the first device 104a.

[0093] To identify the first group of relevant devices, electronic device 106 retrieves a list of multiple devices / devices / registered device lists 104a-104n for the first device 104a from device database 212. Electronic device 106 uses the device database to identify the capabilities of the first device 104a and each device present in the device lists 104a-104n. Electronic device 106 uses a capability-based device clustering module / model 302a (e.g., ...). Figure 3 (As shown) a device cluster is formed for the first device 104a based on the capabilities of the identified first device 104a and each device present in the device lists 104a-104n. The formed device cluster may be a first group of relevant devices from the lists 104a-104n of devices for the first device 104a, which have capabilities similar to the first device 104a. Combined with... Figure 3 Detailed description of capability-based device clustering module / model 302a.

[0094] To identify the second group of relevant devices, electronic device 106 retrieves a list 104a-104n of devices for first device 104a from device database 212. Electronic device 106 uses device database 212 to identify the capabilities of first device 104a and each device present in the list 104a-104n. Electronic device 106 uses device interaction prediction module / model 302b (e.g., ...) Figure 3(As shown) For each device in the lists 104a-104n, a confidence score is calculated and assigned regarding the first device 104. The electronic device 106 can use the device interaction prediction module 302b to calculate the confidence score for each device by matching the capabilities of the corresponding device with the capabilities of the first device 104a. The confidence score for each device depicts the confidence level of a possible interaction between the corresponding device and the first device 104a. When assigning a confidence score for each device, the electronic device 106 discards one or more devices from the acquired lists 104a-104n that are already associated with a confidence score below a device interaction threshold. The electronic device 106 identifies a group of devices from the lists 104a-104n, where each device in this group is already associated with a confidence score greater than or equal to the device interaction threshold. The electronic device 106 determines the group of devices associated with confidence scores greater than or equal to the device interaction threshold as a second group of relevant devices for the first device 104a. The device interaction threshold can be statistically set based on the model confidence score. In the example, the device interaction threshold can be set to 90%. In such a scenario, a device with a confidence score greater than or equal to 90% is considered capable of interacting with the first device 104a. Combined with... Figure 3 Detailed description of the device interaction prediction module / model 302b.

[0095] When determining the first group of related devices and the second group of related devices, electronic device 106 creates a union of the first group of related devices and the second group of related devices to identify one or more second devices 104b-104n related to the context of the first device 104a. The created union is a set that includes all devices present in either the first group of related devices or the second group of related devices. Electronic device 106 selects a device present in either the first group of related devices or the second group of related devices as one or more second devices 104b-104n for the first device 104a.

[0096] When identifying one or more second devices 104b-104n for the first device 104a, the electronic device 106 identifies the type of data to be collected from the first device 104a and one or more second devices 104b-104n. In the example, the type of data includes, but is not limited to, at least one of the following: device status, content being played on the device, device information, manufacturer-provided information, device location, current operating context of the device, etc. To identify the type of data to be collected from the first device 104a and one or more second devices 104b-104n, the electronic device 106 creates one or more device-capability pairs. Each device-capability pair includes a capability and at least one of the second devices 104b-104n and the first device 104a that interact through the corresponding capability. The electronic device 106 accesses the mapping of each capability to type of data in the capabilities by querying a device capability mapping database. The electronic device 106 uses the mapping of each capability to type of data in the capabilities to determine the type of data for each capability present in the created one or more device-capability pairs. The electronic device 106 associates data of a type determined for each capability with at least one of the corresponding second devices 104b-104n and the first device 104a.

[0097] Electronic device 106 collects data of the type of identification from first device 104a and one or more second devices 104b-104n.

[0098] In this embodiment, electronic device 106 collects identified type data from a first device 104a and one or more second devices 104b-104n in online mode. In online mode, electronic device 106 establishes a session with IoT cloud server 102 via communication network 108. When establishing a session with IoT cloud server 102, electronic device 106 sends a data request to IoT cloud server 102 indicating the type of data to be collected from the first device 104a and one or more second devices 104b-104n. IoT cloud server 102 collects the indicated type of data from the first device 104a and one or more second devices 104b-104n via communication network 108. IoT cloud server 102 forwards the collected type of data from the first device 104a and the second devices 104b-104n to electronic device 106 via communication network 108.

[0099] In another embodiment, electronic device 106 collects data of identified types from first device 104a and one or more second devices 104b-104n in offline mode. In offline mode, electronic device 106 establishes a session with first device 104a and one or more second devices 104b-104n using a communication network 108 supported by electronic device 106, first device 104a and one or more second devices 104b-104n. Electronic device 106 sends a data request indicating the type of data to first device 104a and one or more second devices 104b-104n. Electronic device 106 receives the indicated type of data from first device 104a and one or more second devices 104b-104n through the session established on communication network 108.

[0100] When collecting data from a first device 104a and one or more second devices 104b-104n, the electronic device 106 determines the correlation between the first device 104a and each of the one or more second devices 104b-104n based on the data collected from the first device 104a and one or more second devices 104b-104n and / or the capabilities of the first device 104a and one or more second devices 104b-104n. In an embodiment, the determined correlation between the first device 104a and each of the one or more second devices 104b-104n may correspond to multiple correlation features having multiple correlation indices. Examples of correlation features having correlation indices may be, but are not limited to, an action continuity feature with an action continuity correlation index, a conflict resolution feature with a conflict resolution correlation index, a fault resolution feature with a fault resolution correlation index, a screen intelligence feature with a screen intelligence correlation index, a setting suggestion feature with a setting suggestion correlation index, etc. For ease of description and better understanding of the embodiments, action continuity features, conflict resolution features, fault resolution features, screen intelligence features, and setting suggestion features may be considered as examples of relevant features, but it will be apparent to those skilled in the art that any other relevant features may be considered.

[0101] The action continuity characteristic between the first device 104a and each of the second devices (104b-104n) can describe the correlation between the first device 104a and each of the second devices (104b-104n) when one or more operations / actions of each of the second devices (104b-104n) continue on the first device 104a, or vice versa. The action continuity correlation index of each of the second devices (104b-104n) with respect to the first device 104a describes the probability of one or more operations of each of the second devices (104b-104n) continuing on the first device 104a, or vice versa. The conflict resolution characteristic between the first device 104a and each of the second devices (104b-104n) can describe the correlation between the operations of the first device 104a and each of the second devices (104b-104n) that may conflict with each other. The conflict resolution correlation index of each second device (104b-104n) with respect to the first device 104a describes the probability of a conflict between the operation of the first device 104a and each second device (104b-104n). The fault resolution characteristics between the first device 104a and each second device (104b-104n) describe the probability that one or more operations of the first device 104a can be expected to fail with respect to the current operating context of each second device (104b-104n). The fault resolution correlation index of each second device (104b-104n) with respect to the first device 104a describes the probability that one or more operations of the first device 104a will fail due to one or more operations of each second device (104b-104n). The setting suggestion feature between the first device 104a and each of the second devices (104b-104n) describes the correlation between the first device 104a and each of the second devices (104b-104n) when one or more settings / configurations or vice versa are suggested for each of the second devices (104b-104n) on the first device 104a. Examples of settings / configurations may include, but are not limited to, brightening / dimening settings, theme settings, display settings, audio settings, etc. The setting suggestion correlation index of each of the second devices (104b-104n) with respect to the first device 104a can describe the probability that settings or vice versa are suggested for each of the second devices (104b-104n) on the first device 104a. The screen intelligence feature between the first device 104a and each of the second devices (104b-104n) describes the correlation between the content being played on the first device 104a and each of the second devices (104b-104n) and the associated metadata.

[0102] The embodiments described herein use exemplary methods to illustrate the determination of action continuity characteristics, conflict resolution characteristics, and fault resolution characteristics; however, it will be apparent to those skilled in the art that similar methods can be used to determine other relevant characteristics.

[0103] To determine the motion continuity features between the first device 104a and each of the second devices (104b-104n), the electronic device 106 trains a motion continuity module / model 306a to determine the motion continuity features. The electronic device 106 uses softmax regression to train the motion continuity module 306a based on a motion training / labeled dataset of candidate associated devices, along with the associated device context and capabilities. The motion training dataset provides information about context-dependent devices that have been previously identified when controlling the operation of one or more of the multiple devices 104b-104n. Electronic device 106 uses a trained motion continuity module 306a to process the capabilities of the first device 104a and each of the second devices (104b-104n) and the data collected from the first device 104a and each of the second devices (104b-104n), and determines the motion continuity characteristics between the first device 104a and each of the second devices (104b-104n) by calculating the motion continuity correlation index of each of the second devices (104b-104n) with respect to the first device 104a. Combined with... Figure 3 The description details the use of motion continuity module 306a to determine motion continuity characteristics between first device 104a and one or more second devices 104b-104n.

[0104] To determine the conflict resolution characteristics between the first device 104b and each of the second devices (104b-104n), the electronic device 106 trains a first reinforcement module / model 306b based on one or more conflict strategies. One or more conflict strategies can be strategies that depict conflicts between the operation of the first device 104a and one or more of the second devices 104b-104n. Conflict strategies can be defined and dynamically changed based on conflict information that has been monitored over a period of time while controlling the operation of one or more of the multiple devices 104a-104n. The conflict information depicts one or more conflicts that have occurred in the operation of one or more context-related devices. In an example, a user sets the energy consumption (examples of devices (104a-104n)) in a smart plug to a near-limit set by the user, and the user attempts to turn on the device connected to the smart plug. This can be identified as a defined conflict rule leading to a conflict (i.e., the energy limit set in the smart plug may be exceeded when the device is turned on). Electronic device 106 uses a trained first reinforcement module 306b to process data collected from first device 104a and each of the second devices (104b-104n), and determines the conflict resolution characteristics between first device 104a and each of the second devices (104b-104n) by calculating the conflict resolution correlation index of each of the second devices 104b-104n with respect to first device 104a. Combined with... Figure 3 The first reinforcement module 306b is described in detail to determine the conflict resolution features between the first device 104a and one or more second devices 104b-104n.

[0105] To determine the fault resolution characteristics between the first device 104a and each of the second devices (104b-104n), the electronic device 106 trains a second reinforcement module / model 306c based on one or more predefined fault policies. The fault policy can be a defined and dynamically changing strategy for detecting faults in the operation of one of the context-dependent devices with respect to the current operating context of the other devices. The fault policy can be predefined based on fault information that has been monitored over a period of time while controlling the operation of one or more of the multiple devices 104a-104n. The fault information describes a fault in the operation of one of the context-dependent devices due to the operation or context of one or more of the other context-dependent devices. The electronic device 106 uses the trained second reinforcement module 306c to process data collected from the first device 104a and each of the second devices (104b-104n) and determines the fault resolution characteristics between the first device 104a and each of the second devices (104b-104n) by calculating a fault resolution correlation index for each of the second devices (104b-104n) with respect to the first device 104a. Combination Figure 3The second reinforcement module 306c is described in detail to determine the fault resolution characteristics between the first device 104a and one or more second devices 104b-104n.

[0106] When determining the correlation features between the first device 104a and each of one or more second devices 104b-104n, the electronic device 106 discards one or more correlation features between the first device 104a and each second device (104b-104n) that are already associated with a correlation index lower than a correlation threshold. The correlation threshold can be predefined as the percentage of confidence of the corresponding module (306a-306c) predicting the correlation feature. Furthermore, the user can define the correlation threshold. In the example, the correlation threshold can be set to 90%. In the examples herein, the correlation threshold can vary for action continuity correlation index, conflict resolution correlation index, fault resolution correlation index, etc. The electronic device 106 ranks each remaining correlation feature based on the correlation index associated with the corresponding correlation feature between the first device 104a and each second device (104b-104n). In the example, the electronic device 106 ranks the correlation features based on descending order of the associated correlation indices. For example, consider electronic device 106 calculating the action continuity correlation index between first device 104a and devices A and B in second devices 104b-104b as 95% and 80%, respectively, and calculating the conflict resolution correlation index between first device 104a and device B as 92%. In this scenario, electronic device 106 ranks the action continuity between first device 104a and device A as 1, and the conflict resolution feature between first device 104a and device B as 2. Electronic device 106 may discard the action continuity feature between first device 104a and device B because the associated correlation index is less than a correlation threshold (e.g., 90%).

[0107] Electronic device 106 creates one or more semantic actions based on, but not limited to, at least one of the following: a ranking of correlation characteristics between the first device 104a and each of the second devices (104b-104n), data collected from the first device 104a and each of the second devices (104b-104n), etc. Semantic actions can be recommendations, suggestions, notifications, or alarms for controlling the operation of the first device 104a and / or one or more of the second devices 104b-104n. Semantic actions may include, but are not limited to, at least one of the following: suggestions for continuing the operation of one or more of the second devices (104b-104n) on the first device 104a or vice versa, conflict notifications, fault notifications, energy consumption limit notifications, setting suggestions, etc. Conflict notifications indicate conflicts between the operation of the first device 104a and one or more of the second devices 104b-104n. Fault notifications indicate the anticipated fault or vice versa in the operation of the first device 104a due to the operation of one or more of the second devices 104b-104n. The energy consumption limit notification indicates the energy consumption of the first device 104a and / or one or more second devices 104b-104n. The setting suggestion indicates various settings / configurations of one or more second devices 104b-104n for the first device 104a or vice versa. It should be understood that any other semantic actions related to the operation of the first device 104a and / or one or more second devices 104b-104n (including those described above) may be considered.

[0108] Electronic device 106 recommends one or more semantic actions for controlling the operation of first device 104a and / or second devices 104b-104n. In an embodiment, electronic device 106 recommends one or more semantic actions by rendering them on its output 208 as keyboard-covered content on the control interface of first device 104a. In an example, the keyboard-covered content may include at least one of icons, audio, video, images, emotions, GIFs, text, symbols, etc.

[0109] When recommending one or more semantic actions to a user, electronic device 106 also receives input from the user. In the examples herein, the input may be selecting one or more semantic actions for controlling the operation of first device 104a and / or one or more second devices 104b-104n. In another example herein, based on the recommended one or more semantic actions, the input may include instructions given to electronic device 106 for controlling the operation of first device 104a and / or one or more second devices 104b-104n. Electronic device 106 controls the operation of first device 104a and / or one or more second devices 104b-104n based on the input received from the user. In the example, electronic device 106 controls the operation of first device 104a and / or one or more second devices 104b-104n by executing one or more semantic actions selected by the user. In another example, electronic device 106 controls the operation of first device 104a and / or one or more second devices 104b-104n based on instructions received from the user.

[0110] Optionally, the electronic device 106 automatically controls the operation of the first device 104a and / or one or more second devices 104b-104n by performing one or more recommended semantic actions.

[0111] The electronic device 106 can also be configured to recommend a configuration / setting for the first device 104a based on one or more contents being rendered on one or more second devices 104b-104n.

[0112] The electronic device 106 may also be configured to recommend the configuration / settings of the first device 104a for one or more second devices 104b-104n based on data collected from one or more second devices 104a-104n and the correlation between the first device 104a and one or more second devices 104b-104n.

[0113] In another embodiment, such as Figure 1b As shown, electronic device 106 can transmit information about a first device 104a that is pointed to, interacted with, and controlled for its operation to an external server 110 or an IoT cloud server 102. In one example, external server 110 may be a server deployed in an IoT environment. In another example, external server 110 may be located remotely from electronic device 106 or IoT cloud server 102. External server 110 may be configured as a hardware device independent of IoT cloud server 102, but is not limited thereto. External server 110 may be a standalone server or may be a server designed to be classified as software.

[0114] External server 110 or IoT cloud server 102 can be configured to recommend one or more semantic actions as suggestions for controlling the operation of first device 104a and / or one or more second devices 104b-104n by performing the steps (as described above) executed by electronic device 106; therefore, for the sake of brevity, repeated descriptions are omitted.

[0115] Figure 1a and Figure 1b Exemplary blocks of IoT system 100 are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, IoT system 100 may include fewer or more blocks. Furthermore, the labels or names of blocks are for illustrative purposes only and do not limit the scope of the embodiments herein. One or more blocks may be combined together to perform the same or substantially similar functions in IoT system 100.

[0116] Figure 2 This is a block diagram depicting various components of an electronic device 106 for controlling the operation of multiple devices 104a-104n according to embodiments disclosed herein.

[0117] Electronic device 106 includes memory 202, communication interface 204, input device 206, output device 208, and controller / processor 210. Electronic device 106 may also be coupled to device database 212, device capability mapping database 214, etc. Device database 212 includes mappings of a series / multiple devices 104a-104n with associated capabilities. Device capability mapping database 214 includes a mapping of each capability to data / type data. The type data may be metadata associated with each device (104a-104n).

[0118] Memory 202 may store, but is not limited to, at least one of the following: one or more applications; an interactive application configured to point to / select one of a plurality of devices 104a-104b for control; device information, capabilities, manufacturer-provided information, and location information of the plurality of devices 104a-104n; correlation characteristics determined among devices 104a-104n; semantic actions recommended for controlling the operation of any of the devices 104a-104n; input received from the user in response to the recommended semantic actions; etc. Memory 202 may also store device controller 300, which may be executed by controller 210 to control the operation of the plurality of devices 104a-104n. Memory 202 as described herein may include at least one type of storage medium: flash memory, hard disk storage, multimedia card microstorage, card-type memory (e.g., SD or XD memory), random access memory (RAM), static RAM (SRAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), programmable ROM (PROM), magnetic storage, magnetic disk, or optical disk.

[0119] Communication interface 204 may include one or more components that enable electronic device 106 to communicate with another device (e.g., another electronic device, IoT cloud server 102, multiple devices 104a-104n, external server 110, etc.) using communication methods already supported by communication network 108. Communication interface 204 may include components such as wired communicators, short-range communicators, mobile / wireless communication devices, and broadcast receivers.

[0120] A wired communicator enables electronic device 106 to communicate with other devices using communication methods (such as, but not limited to, wired LAN, Ethernet, etc.). A short-range communicator enables electronic device 106 to communicate with other devices using communication methods (such as, but not limited to, Bluetooth Low Energy (BLE), Near Field Communication (NFC), WLAN (or Wi-Fi), Zigbee, Infrared Data Association (IrDA), Wi-Fi Direct (WFD), UWB communication, Ant+ (Interoperable Wireless Transmission Capability) communication, Shared Wireless Access Protocol (SWAP), Wireless Broadband Internet (Wibro), Wireless Gigabit Alliance (WiGiG), etc.). In an embodiment, electronic device 106 can obtain its location information via a short-range wireless communicator. In one example, electronic device 106 can determine its location via an NFC tag. In another example, electronic device 106 can determine its location via a Wi-Fi identifier. In another example, electronic device 106 can determine its location by confirming that it has been connected to the Service Set Identifier (SSID) of the Wi-Fi network. The mobile communicator can transmit and receive wireless signals with at least one of a base station, external terminal, or server on a mobile communication network / cellular network. In this example, depending on the transmission and reception of text / multimedia messages, the wireless signals may include voice call signals, video call signals, or various types of data. A broadcast receiver can receive broadcast signals and / or broadcast-related information from an external source via a broadcast channel. The broadcast channel may include satellite channels and ground wave channels. In this embodiment, electronic device 106 may or may not include a broadcast receiver.

[0121] Input device 206 can be configured to receive interactive input from a user. The interactive input can be used to identify any of the first devices / devices 104a-104n selected by the user for control. The interactive input may include selecting / pointing to any of the plurality of devices 104a-104n for control using UWB or any other communication method supported by electronic device 106. Input device 206 can also be configured to receive input from the user for controlling the operation of context-sensitive devices 104a-104n. The input may include at least one of the following: selecting one or more semantic actions, providing instructions to control the operation of context-sensitive devices 104a-104n based on semantic actions, etc.

[0122] Output device 208 can be configured to provide semantic actions as recommendations for the operation of context-dependent devices 104a-104n. Output device 208 may include, for example, but not limited to, at least one of the following: a voice output device / voice assistant module, a display, a vibration motor, a user interface (UI) module, a light-emitting device, etc., to provide recommendations to the user. The UI module may provide a dedicated UI or graphical user interface (GUI) synchronized with electronic device 106, depending on the application.

[0123] Output 208 receives one or more semantic actions as recommendations from controller 210 and provides these recommendations to the user to control the operation of context-sensitive devices 104a-104n. Output 208 may provide one or more semantic actions as keyboard-covered content on electronic device 106. Upon receiving a semantic action, output 208 generates keyboard-covered content for each semantic action using an appropriate method and provides the generated keyboard-covered content on electronic device 106. The keyboard-covered content may be generated in the form of audio, video, emotion, images, buttons, graphics, text, icons, symbols, vibration signals, etc.

[0124] The controller 210 may include one or more processors. The one or more processors may be general-purpose processors (such as central processing unit (CPU), application processor (AP), etc.), graphics-only units (such as graphics processing unit (GPU)), vision processing unit (VPU) and / or artificial intelligence (AI) dedicated processors (such as neural processing unit (NPU)).

[0125] Controller 210 can be configured to control the operation of multiple devices 104a-104n deployed in an IoT environment. To control the operation of devices 104a-104n, controller 210 identifies a first device 104a to be controlled upon receiving user interaction input from input device 206. Controller 210 determines one or more second devices 104a-104n associated with the context of the first device 104a. The one or more second devices 104a-104n may have similar capabilities to the first device 104a. Controller 210 recommends one or more semantic actions as suggestions for controlling the operation of the first device 104a and / or one or more second devices 104b-104n.

[0126] The controller 210 and the processing device controller 300 control the operation of the first device 104a and / or one or more second devices 104a-104n.

[0127] like Figure 3As shown, the device controller 300 includes a device identifier 302, a data collector 304, a correlation identifier 306, and a recommendation manager 308. The device identifier 302 includes a capability-based device clustering module 302a and a device interaction prediction module 302b. The correlation identifier 306 includes an action continuity module 306a, a first reinforcement module 306b, and a second reinforcement module 306c.

[0128] In this embodiment, the device interaction prediction module 302b, the action continuity module 306a, the first reinforcement module 306b, and the second reinforcement module 306c may be neural networks comprising multiple layers. Each layer has multiple weight values, and layer operations are performed through calculations of the previous layer and operations of multiple weights / coefficients. Examples of neural networks include, but are not limited to, at least one of the following: convolutional neural networks (CNN), deep neural networks (DNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), generative adversarial networks (GAN), regression-based neural networks, deep reinforcement models (with ReLU activation), deep Q-networks, etc. The device interaction prediction module 302b, the action continuity module 306a, the first reinforcement module 306b, and the second reinforcement module 306c may include multiple nodes that can be arranged in layers. Examples of layers may include, but are not limited to, convolutional layers, activation layers, average pooling layers, max pooling layers, cascaded layers, discard layers, fully connected layers, softmax layers, etc. The topology of the layers in the device interaction prediction module 302b, action continuity module 306a, first reinforcement module 306b, and second reinforcement module 306c can vary based on their types. In this example, the device interaction prediction module 302b, action continuity module 306a, first reinforcement module 306b, and second reinforcement module 306c may include an input layer, an output layer, and a hidden layer. The input layer receives layer input (which depends on the type of modules 302b, 306a, 306b, and 306c) and forwards the received layer input to the hidden layer. The hidden layer transforms the layer input received from the input layer into a representation that can be used to generate the output in the output layer. The hidden layer extracts useful / low-level features from the input, introduces non-linearity into the network, and reduces the feature dimension so that the features are equally variable with scaling and translation. Nodes in a layer can be fully connected to nodes in adjacent layers via edges. The input received at the nodes in the input layer can be propagated to the nodes in the output layer via an activation function that computes the state of each node in the network based on the coefficients / weights associated with each edge in the connecting layer.

[0129] The device interaction prediction module 302b, action continuity module 306a, first reinforcement module 306b, and second reinforcement module 306c can be trained using at least one learning method to determine context-dependent devices, action continuity features between context-dependent devices, conflict resolution features between context-dependent devices, and fault resolution between context-dependent devices, respectively. Examples of learning methods include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, regression-based learning, etc. Functions associated with the learning methods can be executed via non-volatile memory, volatile memory, and controller 210. Controller 210 may include one or more processors. Here, one or more processors may be general-purpose processors (such as central processing unit (CPU), application processor (AP), etc.), graphics processing units only (such as graphics processing unit (GPU)), vision processing unit (VPU), and / or artificial intelligence (AI) dedicated processors (such as neural processing unit (NPU)).

[0130] One or more processors determine context-dependent devices, action continuity characteristics between context-dependent devices, conflict resolution characteristics between context-dependent devices, and fault resolution between context-dependent devices based on predefined operating rules stored in non-volatile memory and volatile memory, respectively. These rules are derived from device interaction prediction module 302b, action continuity module 306a, first reinforcement module 306b, and second reinforcement module 306c. The predefined operating rules of device interaction prediction module 302b, action continuity module 306a, first reinforcement module 306b, and second reinforcement module 306c are provided using a learning method to train modules (302b, 306a, 306b, and 306c).

[0131] Here, "provided through learning" means that by applying learning methods to multiple learning datasets (e.g., interactive training datasets, action training datasets, conflict training datasets, fault training datasets, etc.), predefined operating rules or AI models with desired characteristics are formulated. The determination of context-dependent devices, the characteristics of action continuity between context-dependent devices, the characteristics of conflict resolution between context-dependent devices, and the characteristics of fault resolution between context-dependent devices can be performed within the electronic device 106 itself, which performs learning according to the embodiment, and / or can be implemented via a separate server / system.

[0132] Device identifier 302 can be configured to identify a first device 104a that is being directed to for control. Device identifier 302 identifies the first device 104a when it receives interactive input from the user from the input device 206. Device identifier 302 can also be configured to determine one or more second devices 104b-104n that are associated with the context of the first device 104a.

[0133] To identify one or more second devices 104b-104n, device identifier 302 uses capability-based device clustering module 302a to identify a first group of related devices. To identify the first group of related devices, capability-based device clustering module 302a retrieves from device database 212 a list of devices 104a-104n registered with IoT cloud server 102 or electronic device 106 and their associated capabilities. Device clustering module 302a determines the capabilities of the first device 104a and each device present in the list of devices 104a-104n by accessing device database 212. Capability-based device clustering module 302a forms a device cluster by including a group of devices from the list of devices 104a-104n that have similar capabilities to the first device 104a. The group of devices included in the formed device cluster corresponds to the first group of related devices for the first device 104a.

[0134] Device identifier 302 uses device interaction prediction module 302b to identify a second set of related devices for the first device 104a. Device interaction prediction module 302b can be a trained model that can be used to identify the second set of related devices for the first device 104a. Device identifier 302 can train device interaction prediction module 302b based on an interaction training dataset that includes mappings between multiple devices 104a-104n and capabilities. In one example, device identifier 302 can obtain the interaction training dataset stored in memory 202. In another example, device identifier 302 can obtain the interaction training dataset from IoT cloud server 102 or an external server.

[0135] To identify the second group of relevant devices, the device interaction prediction module 302b retrieves a list of devices 104a-104n registered with the IoT cloud server 102 or electronic device 106 and their associated capabilities from the device database 212. The device interaction prediction module 302b determines the capabilities of the first device 104a and each device present in the list of devices 104a-104n by accessing the device database 212. The device interaction prediction module 302b assigns a confidence score to each device in the list of devices based on their capabilities for the first device 104a. The device interaction prediction module 302b compares the confidence score of each device with respect to the first device 104a with an interaction threshold. The device interaction prediction module 302b discards a group of devices associated with confidence scores below the interaction threshold. The device interaction prediction module 302b identifies a group of devices associated with confidence scores equal to or greater than the interaction threshold as the second group of relevant devices for the first device 104a.

[0136] Once the first group of related devices and the second group of related devices have been identified for the first device 104a, the device identifier 302 performs a merging of the first group of related devices and the second group of related devices. The device identifier 302 identifies one or more devices found in the first group of related devices or the second group of related devices as one or more second devices 104b-104n for the first device 104a. The device identifier 302 provides information about the first device 104a and one or more second devices 104b-104n to the data collector 304. The device identifier 302 also provides the capabilities of the first device 104a and one or more second devices 104b-104n to the relevance identifier 306.

[0137] Data collector 304 can be configured to identify the type of data to be collected from the first device 104a and one or more second devices 104b-104n, and collect the identified type of data from the first device 104a and one or more second devices 104b-104n. To identify the type of data to be collected from the first device 104a and one or more second devices 104b-104n, data collector 304 creates one or more device-capability pairs. Data collector 304 queries device capability mapping database 214 based on each device-capability pair. Data collector 304 receives the type of data to be collected for each capability from device capability mapping database 214. Data collector 304 associates the identified type of data for each capability with the corresponding first device 104a and / or one or more second devices 104b-104n. Data collector 304 collects the identified type of data from the first device 104a and one or more second devices 104b-104n. In one example, data collector 304 collects identified type data from first device 104a and one or more second devices 104b-104n in online mode. In another example, data collector 304 collects identified type data from first device 104a and one or more second devices 104b-104n in offline mode. Data collector 304 provides the type data collected from first device 104a and one or more second devices 104b-104n to correlation identifier 306.

[0138] The correlation identifier 306 can be configured to determine, based on capability and type data collected from the first device 104a and one or more second devices 104b-104n, a correlation feature having a correlation index between the first device 104a and each of the one or more second devices 104b-104n. The correlation feature includes, but is not limited to, at least one of the following: action continuity feature, conflict resolution feature, and fault resolution feature. The correlation index includes, but is not limited to, at least one of the following: an action continuity correlation index associated with the action continuity feature, a conflict resolution correlation index associated with the conflict resolution feature, a fault resolution correlation index associated with the fault resolution feature, etc.

[0139] The correlation identifier 306 trains the action continuity module 306a and uses the trained action continuity module 306a to determine action continuity features between the first device 104a and each of one or more second devices 104b-104n. The correlation identifier 306 uses soft maximum regression to train the action continuity module 306a by leveraging the labeled dataset of candidate devices and their correlation capabilities. The action continuity module 306a can be a trained neural network in which the number of layers, the sequence of layers used for processing, and the parameters associated with each layer can be known and fixed to determine the action continuity features between the first device 104a and one or more second devices 104b-104n. Examples of parameters associated with each neural network layer can be, but are not limited to, activation functions, biases, input weights, output weights, etc., associated with the layers of the neural network.

[0140] The correlation identifier 306 can update the trained action continuity module 306a each time it detects an action continuity feature between the first device 104a and one or more second devices 104b-104n. The correlation identifier 306 can update the action continuity module 306a, and can also update the trained action continuity module 306a by updating the weights / coefficients of the neural network layers of the action continuity module 306a.

[0141] To determine the motion continuity characteristics between the first device 104a and one or more second devices 104b-104n, the correlation identifier 306 feeds the capabilities of the first device 104a and one or more second devices 104b-104n, as well as the types of data collected from the first device 104a and one or more second devices 104b-104n, as motion inputs to the trained motion continuity module 306a. The trained motion continuity module 306a processes the motion inputs received from the correlation identifier 306 and determines the motion continuity characteristics between the first device 104a and one or more second devices 104b-104n. Determining the motion continuity characteristics between the first device 104a and each of the one or more second devices 104b-104n includes calculating a motion continuity correlation index with respect to the first device 104a for each of the one or more second devices 104b-104n. The value of the action continuity correlation index calculated for each second device (104b-104n) can be in the range [0, 100], indicating the probability that one or more actions / operations of the corresponding second device (104b-104n) must be continued on the first device 104a. An action continuity correlation index of 0 indicates the lowest correlation between the associated second devices (104b-104n) with respect to the first device 104a. An action continuity correlation index of 100 indicates the highest correlation between the associated second devices (104b-104n) with respect to the first device 104a.

[0142] The correlation identifier 306 trains the first reinforcement module 306b and uses the trained first reinforcement module 306b to determine the conflict resolution characteristics between the first device 104a and each of one or more second devices 104b-104n.

[0143] To train the first reinforcement module 306b, the correlation identifier 306 collects a conflict training dataset and one or more predefined conflict strategies. The conflict training dataset may be information that has been monitored while controlling the operation of one or more of the multiple devices 104a-104n over a period of time. The conflict training dataset includes information about the types of operations used to control these context-dependent devices. The one or more predefined conflict strategies may be strategies that depict conflicts between the operations of the context-dependent devices. Conflict strategies may be predefined based on conflict information that has been monitored while controlling the operation of one or more of the multiple devices 104a-104n over a period of time. Conflict strategies may also be manually defined by the user. In one example, the correlation identifier 306 collects the conflict training dataset and one or more predefined conflict strategies from memory 202. In another example, the correlation identifier 306 collects the conflict training dataset and one or more predefined conflict strategies from an IoT cloud server 102 or an external server, etc. The correlation identifier 306 feeds the collected conflict training dataset and one or more predefined conflict strategies to the first reinforcement module 306b. The first reinforcement module 306b processes the received training dataset and generates conflict actions. Conflict actions depict the probability of conflict (e.g., low or high) between one or more operations of a context-dependent device. If the generated conflict action is correct, the correlation identifier 306 assigns a positive reward to the first reinforcement module 306b, and if the generated conflict action is incorrect, it assigns a negative reward. The correlation identifier 306 also dynamically updates one or more conflict policies and the first reinforcement module 306a. The correlation identifier 306 updates one or more conflict policies based on the conflict actions generated by the first reinforcement module 306b with respect to the received conflict training dataset. The correlation identifier 306 updates the first reinforcement module 306b by reinforcing conflict observations into it to better generate conflict actions. Conflict observations may be based on a reduction in the efficiency of the first context-dependent device's operation compared to the operation of a second context-dependent device. For example, the first reinforcement module 306b predicts the cooling time of a room based on previous cooling data of an AC. However, the model prediction of cooling time may fail if the window is open. In this case, the first reinforcement module 306b analyzes the state of the context-sensitive device to identify whether an open window (i.e., the multi-purpose sensor attached to the window is open) affects the cooling of the AC and updates its strategy to generate conflict information / recommendations.

[0144] To determine the conflict resolution characteristics between the first device 104a and each of one or more second devices 104b-104n, the correlation identifier 306 feeds data of a type collected from the first device 104a and one or more second devices 104b-104n as conflict input to a trained first reinforcement module 306b. The trained first reinforcement module 306b processes the conflict input received from the correlation identifier 306 and determines the conflict resolution characteristics between the first device 104a and one or more second devices 104b-104n. Determining the conflict resolution characteristics between the first device 104a and each of one or more second devices 104b-104n includes calculating a conflict resolution correlation index for the first device 104a for each of the one or more second devices 104b-104n. The conflict resolution relevance index may include values ​​from 0 to 100, where values ​​0 and 100 indicate the lowest and highest probability of conflict in the operation of the first device 104a and the corresponding second devices (104b-104n), respectively.

[0145] The correlation identifier 306 trains the second reinforcement module 306c and uses the trained second reinforcement module 306c to determine the fault resolution characteristics between the first device 104a and each of one or more second devices 104b-104n.

[0146] To train the second reinforcement module 306c, the correlation identifier 306 collects a fault training dataset and one or more predefined fault policies. Fault training may include information that has been monitored while controlling the operation of one or more of the multiple devices 104b-104n over a period of time. The fault training dataset includes information about the types of operations used to control these context-dependent devices. Fault policies may be defined strategies for detecting faults in the operation of one of the context-dependent devices in relation to the current operating context of other devices. Fault policies may be predefined based on fault information that has been monitored while controlling the operation of one or more of the multiple devices 104a-104n over a period of time. Fault policies may also be manually defined by the user. The fault information describes a fault in the operation of one of the context-dependent devices due to the operation or context of one or more other context-dependent devices. In one example, the correlation identifier 306 collects the fault training dataset and one or more predefined fault policies from memory 202. In another example, the correlation identifier 306 collects the fault training dataset and one or more predefined fault policies from an IoT cloud server 102 or an external server, etc. The correlation identifier 306 feeds a collected fault training dataset and one or more predefined fault policies to the second reinforcement module 306c. The second reinforcement module 306c processes the received fault training dataset and generates fault actions. A fault action describes the probability (e.g., low or high) of a fault expected in one or more operations of one of the context-related devices due to the operation of other context-related devices. If the generated fault action is correct, the correlation identifier 306 assigns a positive reward to the second reinforcement module 306c; if the generated fault action is incorrect, the correlation identifier 306 assigns a negative reward to the second reinforcement module 306c. The correlation identifier 306 also dynamically updates one or more fault policies and the second reinforcement module 306c. The correlation identifier 306 updates one or more fault policies based on the fault actions generated by the second reinforcement module 306c with respect to the received fault training dataset. The correlation identifier 306 updates the second reinforcement module 306c by reinforcing fault observations into it to better generate fault actions. The second reinforcement module 306c can learn fault observations based on feedback from context-related devices. For example, the second reinforcement module 306c predicts successful washing machine operation. However, if a washing machine malfunctions, the second reinforcement module 306c analyzes the state of the context-related devices to identify whether the state of the corresponding devices differs from that when the washing machine operation is successful, in order to infer a new strategy. If the washing machine operation is successful, the overhead tank sensor value will never be low.However, in the event of a washing machine malfunction due to a lack of water in the top tank, the water sensor value may be low water level. This change in the water sensor state (i.e., low water level) that causes the washing machine malfunction is deduced by the second reinforcement module 306c by analyzing the state of the context-dependent device.

[0147] To determine the fault resolution characteristics between the first device 104a and each of one or more second devices 104b-104n, the correlation identifier 306 feeds data of a type collected from the first device 104a and one or more second devices 104b-104n as fault input to a trained second reinforcement module 306c. The trained second reinforcement module 306c processes the fault input received from the correlation identifier 306 and determines the fault resolution characteristics between the first device 104a and each of the one or more second devices 104b-104n. Determining the fault resolution characteristics between the first device 104a and each of the one or more second devices 104b-104n includes calculating a fault resolution correlation index for the first device 104a for each of the one or more second devices 104b-104n. The fault resolution relevance index can include values ​​from 0 to 100, where values ​​0 and 100 indicate the lowest and highest probabilities, respectively, of an expected fault in the operation of the first device 104a within the current operating context of the corresponding second device (104b-104n). The relevance identifyr 306 provides the identified relevance features, along with the associated relevance index, to the recommendation manager 308.

[0148] Recommendation manager 308 can be configured to create one or more semantic actions and provide the created semantic actions as recommendations for controlling the operation of first device 104a and / or one or more second devices 104b-104n. To create one or more semantic actions, recommendation manager 308 ranks each relevance feature based on an associated relevance index. Recommendation manager 308 creates one or more semantic actions based on the ranking of each relevance feature and data collected from first device 104a and one or more second devices 104b-104n.

[0149] The recommendation manager 308 provides one or more semantic actions to the output 208 to provide the user with one or more semantic actions as recommendations for controlling the operation of the first device 104a and / or one or more second devices 104b-104n.

[0150] The recommendation manager 308 can also be configured to control the operation of the first device 104a and / or one or more second devices 104b-104n after receiving user input from the input device 206, following the provision of recommendations to the user. Optionally, the recommendation manager 308 can also be configured to control the operation of the first device 104a and / or one or more second devices 104b-104n based on one or more semantic actions created.

[0151] The recommendation manager 308 can also be configured to recommend configurations for the first device 104a based on one or more pieces of content being rendered on one or more second devices 104b-104n. In the example, recommendations may be made for a light bulb (example first device 104a) based on the type of content played on the TV (example second devices (104b-104n) determined for the first device 104a). Configurations may include at least one of turning the bulb on, turning it off, brightness settings, dimming settings, etc.

[0152] Recommendation manager 308 can also be configured to recommend the configuration of first device 104a for one or more second devices 104b-104n based on data collected from one or more second devices 104a-104n and the correlation between first device 104a and one or more second devices 104b-104n. In the example, recommendation manager 308 may recommend a dimming setting (example configuration) of the first bulb (example first device 104a) to second bulbs present in the same location (example second devices (104b-104n) determined for first device 104a).

[0153] Figure 2 and Figure 3 Exemplary blocks of electronic device 106 are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, electronic device 106 may include fewer or more blocks. Furthermore, the labels or names of the blocks are for illustrative purposes only and do not limit the scope of the embodiments herein. One or more blocks may be combined together to perform the same or substantially similar functions in electronic device 106.

[0154] Figure 4 An example concept diagram depicts an IoT system 100 according to embodiments disclosed herein, wherein each device 104a is controlled based on the current operating context of devices 104b-104n with capabilities and dependencies.

[0155] The IoT system 100 includes an IoT cloud server 102, multiple devices 104a-104n, and an electronic device 106. The electronic device 106 can be configured to control the operation of the multiple devices 104a-104n based on their correlation with each other.

[0156] Consider an example scenario where a user uses UWB supported on electronic device 106 to point at a TV deployed in an IoT environment (example of first device 104a). In such a scenario, electronic device 106 recognizes the TV as the device being pointed at for control. Figure 5 The text describes the context-sensitive devices 104b-104n for determining TV.

[0157] like Figure 5 As shown, electronic device 106 obtains a list of devices registered with IoT cloud server 102 or electronic device 106 by accessing device database 212. The list of devices may include at least one of speakers, cameras, ovens, and AC. Electronic device 106 uses device database 212 to determine the capabilities of each device present in the obtained list of devices and the capabilities of the TV. In the example herein, electronic device 106 determines the capabilities of the TV as audio and display / video source. Similarly, electronic device 106 determines the capabilities of speakers, cameras, ovens, and AC as audio, video source, video source, and temperature, respectively. Electronic device 106 forms a device cluster for each of the determined capabilities (audio and display / video source) of the TV. The formed device cluster for audio includes the TV and speakers. The formed device cluster for display / video source includes the TV, camera, and oven. Electronic device 106 identifies speakers, cameras, and ovens as a first group of related devices for the TV.

[0158] In parallel with determining the first set of relevant devices for the TV, electronic device 106 determines a second set of relevant devices for the TV. To determine the second set of relevant devices, electronic device 106 obtains a list of devices registered with IoT cloud server 102 or electronic device 106 by accessing device database 212. The list of devices may include at least one of speakers, cameras, smart plugs, ovens with built-in cameras, and AC. Electronic device 106 determines the capabilities of the TV as audio and display / video sources. Similarly, electronic device 106 determines the capabilities of speakers, cameras, smart plugs, ovens, and AC as audio, video sources, energy, video sources, and temperature, respectively. Electronic device 106 determines the likelihood of interaction between the device and the TV based on the capabilities of the device and the TV, assigning confidence scores of speakers, cameras, smart plugs, ovens, and AC regarding the TV to 95%, 90%, 90%, 20%, and 20%, respectively. Electronic device 106 discards ovens and AC because their confidence scores are less than an interaction threshold (e.g., 50%). Electronic device 106 identifies the speaker, camera, and smart plug as the second set of related devices for the TV.

[0159] Electronic device 106 performs a combination of a first set of related devices and a second set of related devices to form a context-sensitive device for the TV. In the example, electronic device 106 identifies a speaker, camera, oven, and smart plug as context-sensitive devices for the TV.

[0160] When determining a context-sensitive device for a TV, electronic device 106 identifies the types of data that will be collected from the TV and the associated context-sensitive device. Figure 6 The text describes the types of data that will be collected from TVs and associated context-dependent devices.

[0161] like Figure 6 As shown, electronic device 106 creates device-capability pairs including audio pairs, display pairs, and energy pairs. Audio pairs include a TV and a speaker. Display pairs include a TV, a camera, and an oven. Energy pairs include a TV and a smart plug. Electronic device 106 queries device capability mapping database 214 and receives information from device capability mapping database 214 regarding the type of data to be collected from the device associated with each capability. In this example, the type of data to be collected from the device includes at least one of the following: device status, device current operating context, device location, etc.

[0162] Electronic device 106 collects data of the identified type from the TV and associated context-sensitive devices in online or offline mode. Figure 7 The document describes the collection of data from the device in both online and offline modes. In online mode, electronic device 106 establishes a session with IoT cloud server 102 via a communication method supported by communication network 108. Electronic device 106 sends a request to IoT cloud server 102 indicating the type of data / metadata to be collected from the TV and associated context-sensitive devices. IoT cloud server 102 collects the indicated type of data / metadata from the TV and associated context-sensitive devices and forwards the collected data to electronic device 106.

[0163] In offline mode, electronic device 106 establishes a session with the TV and associated context-sensitive devices via a communication method (e.g., UWB herein) supported by communication network 108. Electronic device 106 sends a request indicating the type of data required from the TV and associated context-sensitive devices. Electronic device 106 receives the determined type of data from the TV and associated context-sensitive devices. Figure 7 The text describes the types of data collected from TV and associated context-dependent devices.

[0164] When collecting data from the TV and associated context-dependent devices, electronic device 106 determines operational continuity characteristics, conflict resolution characteristics, and fault resolution characteristics between the TV and each of the associated context-dependent devices. Embodiments herein further illustrate the determination of operational continuity characteristics, conflict resolution characteristics, and fault resolution characteristics between the TV and each of the associated context-dependent devices; however, it will be apparent to those skilled in the art that any other relevant characteristics can be determined in a similar manner.

[0165] exist Figure 8a and Figure 8b The text describes the characteristics that determine the continuity of actions. For example... Figure 8a As shown, electronic device 106 uses soft maximum regression to train action continuity module 306a using a labeled dataset of candidate devices and associated capabilities. The trained action continuity module 306a can be a trained neural network in which the number of layers, the sequence of layers used for processing, and the parameters associated with each layer can be known and fixed to determine action continuity features between the first device 104a and one or more second devices 104b-104n. Examples of parameters associated with each neural network layer can be, but are not limited to, activation functions, biases, input weights, output weights, etc., associated with the layers of the neural network.

[0166] like Figure 8b As shown, electronic device 106 uses a trained action continuity module 306a to determine the action continuity characteristics between the TV and each of the associated context-related devices. Electronic device 106 feeds capabilities and data collected from the TV and the associated context-related devices to the trained action continuity module 306a. The action continuity module 306a processes the data and capabilities of the TV and the associated context-related devices to determine the action continuity characteristics between the TV and each of the associated context-related devices by calculating an action continuity correlation index for the TV for each context-related device. The action continuity correlation index indicates the probability of continuing one or more actions / operations of one or more context-related devices on the TV. In the examples presented herein, the action continuity correlation indices calculated for the speaker, camera, oven, and smart plug could be 80%, 15%, 5%, and 0, respectively. Compared to the camera, oven, and smart plug, the speaker may have the highest action continuity correlation index because the states of the camera, oven, and smart plug do not have any content / actions to continue on the TV. Therefore, because the speaker is strongly correlated with the TV, one or more actions / content / operations of the speaker can be continued on the TV.

[0167] exist Figure 9a and Figure 9bThe text describes the conflict resolution features for determining the relationship between a TV and its associated context-dependent devices. For example... Figure 9a As shown, electronic device 106 trains a first reinforcement module 306b to determine conflict resolution characteristics between a TV and associated context-sensitive devices. Electronic device 106 feeds a collected training dataset and one or more predefined conflict policies to the first reinforcement module 306b. The first reinforcement module 306b processes the received training dataset and generates conflict actions. Conflict actions describe how one or more operations of a context-sensitive device may conflict with one or more operations of other context-sensitive devices. If the generated conflict action is correct, electronic device 106 assigns a positive reward to the first reinforcement module 306b; if the generated conflict action is incorrect, electronic device 106 assigns a negative reward to the first reinforcement module 306b. Electronic device 106 also dynamically updates one or more conflict policies and the first reinforcement module 306b. Electronic device 106 updates one or more conflicts based on actions predicted by the first reinforcement module 306b with respect to the received conflict training dataset. Electronic device 106 updates the first reinforcement module 306b by reinforcing conflict observations into it to better generate conflict actions.

[0168] like Figure 9b As shown, to determine conflict resolution between the TV and associated context-sensitive devices, electronic device 106 feeds data collected from the TV and associated context-sensitive devices to a first reinforcement module 306b. The first reinforcement module 306b processes the data from the TV and associated context-sensitive devices and determines the conflict resolution between the TV and each of the context-sensitive devices by calculating a conflict resolution correlation index for the TV for each context-sensitive device. The conflict resolution correlation index depicts the probability of a conflict between the operation of the TV and the associated context-sensitive devices. In the examples herein, the conflict resolution correlation indices calculated for the speaker, camera, oven, and smart plug may be 10%, 8%, 7%, and 75%, respectively. Compared to other devices, the smart plug may have the highest correlation with the TV, which depicts the possibility that continuous power consumption may conflict with the operation of the TV.

[0169] exist Figure 10a and Figure 10b The text describes the fault-solving characteristics for determining the relationship between the TV and associated context-dependent devices. For example... Figure 10aAs shown, electronic device 106 trains a second reinforcement module 306c to determine fault resolution characteristics between a TV and associated context-dependent devices. Electronic device 106 feeds a fault training dataset and one or more predefined fault policies to the second reinforcement module 306c. The second reinforcement module 306c processes the received fault training dataset and generates fault actions. A fault action describes a possible fault in one or more operations of the TV due to one or more operations of one or more context-dependent devices. If the generated fault action is correct, electronic device 106 assigns a positive reward to the second reinforcement module 306c; if the generated fault action is incorrect, electronic device 106 assigns a negative reward to the second reinforcement module 306c. Electronic device 106 also dynamically updates one or more fault policies and the second reinforcement module 306c. Electronic device 106 updates one or more fault policies based on the fault actions generated by the second reinforcement module 306c with respect to the received fault training dataset. Electronic device 106 updates the second reinforcement module 306c by reinforcing fault observations into the second reinforcement module 306c to better generate fault actions.

[0170] like Figure 10b As shown, to determine fault resolution between the TV and associated context-dependent devices, electronic device 106 feeds data collected from the TV and associated context-dependent devices to a trained second reinforcement module 306c. The second reinforcement module 306c processes the data from the TV and associated context-dependent devices and determines fault resolution between the TV and each context-dependent device by calculating a fault resolution correlation index for the TV for each context-dependent device. The fault resolution correlation index depicts the probability of a fault in the operation of the TV due to the operation of the associated context-dependent device. In the example presented herein, the fault resolution correlation indices calculated for the speaker, camera, oven, and smart plug may be 10%, 10%, 10%, and 70%, respectively. Compared to other devices, the smart plug may have the highest correlation with the TV, which depicts the possibility that continuous power consumption may cause operational failure of the TV.

[0171] Based on the determined correlation characteristics between the TV and associated context-dependent devices, electronic device 106 provides recommendations for controlling the operation of the TV or associated context-dependent devices. Figure 11 The document describes recommendations for providing operation for controlling a TV or associated context-sensitive devices.

[0172] like Figure 11As described herein, electronic device 106 ranks the motion continuity characteristics, conflict resolution characteristics, and fault resolution characteristics between the TV and each of the associated context-associated devices based on capabilities and a relevance index associated with each context-associated device. The relevance index includes at least one of a motion continuity relevance index, a conflict resolution relevance index, and a fault resolution relevance index. In the example herein, electronic device 106 ranks the motion continuity characteristic between the TV and the speaker as 1, the conflict resolution characteristic between the TV and the smart plug as 2, and the fault resolution characteristic between the TV and the smart plug as 3.

[0173] Electronic device 106 creates one or more semantic actions based on a ranking of relevance characteristics between the TV and associated related devices. In the examples herein, semantic actions include suggestions to continue playing content on the TV via speakers, and alarms indicating that continued use of the TV may damage it.

[0174] Electronic device 106 provides information about context-dependent devices for the TV and their current operating context, as well as recommended semantic actions for the user to control the TV. Electronic device 106 renders the semantic actions as content overlaid by the keyboard.

[0175] Electronic device 106 controls the operation of the TV when the user selects any of the semantic actions. In one example, if the user selects to continue the suggested content playing on the speaker on the TV, electronic device 106 enables the TV to play the content / audio playing on the speaker. In another example, if the user selects to trigger an alarm indicating that continued use of the TV may damage it, electronic device 106 turns off the TV.

[0176] Figure 12 This is a flowchart depicting the operation of multiple devices 104a-104n in an IoT environment according to embodiments disclosed herein.

[0177] In step 1202, electronic device 106 identifies the first device 104a to be controlled. In the example, the user (in step 1201) uses at least one of the communication methods supported by communication network 108 to point at the first device 104a for control. In step 1203, electronic device 106 checks whether any other devices exist in the IoT environment. If no other devices exist in the IoT environment, then in step 1204, electronic device 106 provides the user with normal commands for controlling the first device 104a.

[0178] If other devices are present in the IoT environment, then in step 1205, electronic device 106 determines one or more second devices 104b-104n that are context-related devices to the first device 104a. Electronic device 106 determines one or more second devices 104b-104n to the first device 104a by forming a cluster of related devices to the first device 104a based on capabilities and interactions.

[0179] In step 1206, electronic device 106 identifies the type of data to be collected from first device 104a and one or more second devices 104b-104n. Electronic device 106 creates device-capability pairs, where each pair includes a capability and a first device 104a and / or one or more second devices 104b-104n interacting through the corresponding capability. Electronic device 106 queries device capability mapping database 214 and receives information about the type of data to be collected from the first device 104a and / or the second devices 104b-104n associated with each capability. In step 1207, electronic device 106 collects the identified type of data from the first device 104a and / or one or more second devices 104b-104n. In this example, electronic device 106 collects the identified type of data from the first device 104a and / or one or more second devices 104b-104n in online mode. In another example, electronic device 106 collects data of the identified type from first device 104a and / or one or more second devices 104b-104n in offline mode.

[0180] In step 1208, electronic device 106 determines, for each of one or more second devices 104b-104n, a correlation feature with a correlation index regarding the first device 104a. Examples of correlation features may include, but are not limited to, at least one of the following: operation continuity feature, conflict resolution feature, fault resolution feature, screen intelligence feature, setting suggestion feature, etc. Examples of correlation indices include, but are not limited to, at least one of the following: operation continuity correlation index associated with operation continuity feature, conflict resolution correlation index associated with conflict resolution feature, fault resolution correlation index associated with fault resolution feature, screen intelligence correlation index associated with screen intelligence feature, setting suggestion correlation index associated with setting suggestion feature, etc. Electronic device 106 determines, for each of one or more second devices 104b-104n, a correlation feature with a correlation index regarding the first device 104a based on the capabilities of the first device 104a and one or more second devices 104b-104n and / or based on data collected from the first device 104a based on the capabilities of the first device 104a and one or more second devices 104b-104n.

[0181] In step 1209, electronic device 106 ranks each correlation feature between each of the one or more second devices 104b-104n and the first device 104a based on the associated correlation index and data collected from the first device 104a and the corresponding second devices (104b-104n).

[0182] In step 1210, electronic device 106 creates one or more semantic actions based on the ranking of each relevance feature between each of the one or more second devices 104b-104n and the first device 104. Examples of one or more semantic actions may be, but are not limited to, suggestions for continuing an operation / action of one of the second devices 104b-104n on the first device 104a, conflict notifications, fault notifications, contextual action suggestions, settings / configurations, etc.

[0183] In step 1211, electronic device 106 generates keyboard overlay content for each of the one or more semantic actions created. In step 1212, electronic device 106 renders the keyboard overlay content corresponding to one or more semantic actions as a recommendation for the user to control the first device 104a and / or one or more second devices 104b-104n.

[0184] Figure 13a and Figure 13b An example use case scenario is described, which describes the operation of one of a plurality of devices 104a-104n in an IoT environment according to embodiments disclosed herein.

[0185] Consider an example scenario where a user points to a washing machine / washer using UWB supported by electronic device 106 (example of first device 104a). In this case, electronic device 106 identifies the washing machine as the device pointed to by the user for operation. Electronic device 106 acquires a list of devices including a water sensor, thermostat, oven, AC, light bulb, and smart plug, and identifies the water sensor, thermostat, and smart plug from the list of devices as context-sensitive devices for the washing machine (second devices 104b-104n). Electronic device 106 identifies the type of data that will be collected from the washing machine, water sensor, thermostat, and smart plug. In the example, the type of data that will be collected from the devices includes at least one of the following: the status of the device, the location of the device, and the current operating context of the device (e.g., water level from the water sensor, weather data from the thermostat, and energy limit from the smart plug). Electronic device 106 collects the identified type of data from the washing machine, water sensor, thermostat, and smart plug in either online or offline mode.

[0186] When collecting data, electronic device 106 determines, for the water sensor, thermostat, and smart plug, motion continuity characteristics with an motion continuity correlation index, conflict resolution characteristics with a conflict resolution correlation index, and fault resolution characteristics with a fault resolution correlation index for the washing machine. Electronic device 106 ranks the fault resolution characteristics between the water sensor and the washing machine as 1 (e.g., because the water level is low), and ranks the motion continuity characteristics between the thermostat and the washing machine as 2 (e.g., because the weather data depicts a rain forecast for the next 2 hours).

[0187] Electronic device 106 generates one or more semantic actions based on a ranking of relevance features and data associated with the washing machine, water sensor, thermostat, and smart plug. In an example, a semantic action may include at least one of the following: a suggestion to delay turning on the washing machine based on weather data, and a suggestion to start the water pump based on a sensed low water level. Electronic device 106 recommends one or more of the created semantic actions to the user for operating the washing machine. Electronic device 106 may recommend one or more of the created semantic actions by rendering them as keyboard overlay content on the display / output device 208 of electronic device 106.

[0188] Figure 14a and Figure 14b Another example use case scenario is described, which describes the operation of one of a plurality of devices 104a-104n in an IoT environment according to embodiments disclosed herein.

[0189] Consider an example scenario where a user points to an AC (example of first device 104a) using UWB supported by electronic device 106. In such a scenario, electronic device 106 identifies the AC as a device pointed to by the user for operation. Electronic device 106 acquires a list of devices including window contact sensors, thermostats, ovens, light bulbs, and smart plugs, and identifies the window contact sensors, thermostats, and smart plugs from the list of devices as context-sensitive devices for the AC (second devices 104b-104n). Electronic device 106 identifies the type of data that will be collected from the AC, window contact sensors, thermostats, and smart plugs. In the example, the type of data that will be collected from the devices includes at least one of the following: the status of the device, the location of the device, and the current operating context of the device (e.g., contact sensor status, temperature, and energy limits). Electronic device 106 collects the identified type of data from the AC, window contact sensors, thermostats, and smart plugs in online or offline mode.

[0190] When collecting data, electronic device 106 determines, for the window contact sensor, thermostat, and smart plug, motion continuity characteristics with an motion continuity correlation index, conflict resolution characteristics with a conflict resolution correlation index, and fault resolution characteristics with a fault resolution correlation index for the AC. Electronic device 106 ranks the conflict resolution characteristics between the window contact sensor and the AC as 1 (e.g., when the window is open) and the conflict resolution characteristics between the smart plug and the AC as 2 (e.g., when the energy limit is reached).

[0191] Electronic device 106 generates one or more semantic actions based on a ranking of relevance features and data associated with the AC, window contact sensor, thermostat, and smart plug. In the example, a semantic action may include at least one of the following: a notification to close the window for better AC operating efficiency, and an energy consumption limit notification. Electronic device 106 recommends one or more semantic actions created for operating the AC to the user. Electronic device 106 may recommend one or more semantic actions created by rendering them as keyboard overlay content on the display / output device 208 of electronic device 106.

[0192] Figure 15a and Figure 15b Another example use case scenario is described, which describes the operation of one of a plurality of devices 104a-104n in an IoT environment according to embodiments disclosed herein.

[0193] Consider an example scenario where a user points to a TV using UWB supported by electronic device 106 (example of first device 104a). In such a scenario, electronic device 106 identifies the TV as the device being pointed to by the user for operation. Electronic device 106 acquires a list of devices including speakers, cameras, ovens with built-in cameras, AC, water sensors, smart plugs, and light bulbs, and identifies the speakers, cameras, ovens, and smart plugs from the list of devices as context-sensitive devices for the TV (second devices 104b-104n). Electronic device 106 identifies the type of data that will be collected from the TV, speakers, cameras, ovens, and smart plugs. In the example, the type of data that will be collected from the devices includes at least one of the following: the state of the device, the location of the device, the current operating context of the device (e.g., content being played on the speakers, motion and video sources from the cameras, camera sources from the ovens, and energy limits from the smart plugs). Electronic device 106 collects the identified type of data from the TV, speakers, cameras, ovens, and smart plugs in online or offline mode.

[0194] When collecting data, electronic device 106 determines motion continuity characteristics with a motion continuity correlation index, conflict resolution characteristics with a conflict resolution correlation index, and fault resolution characteristics with a fault resolution correlation index for the speaker, camera, oven, and smart plug related to the TV. Electronic device 106 ranks the motion continuity characteristics between the speaker and the TV as 1, the conflict resolution characteristics between the smart plug and the TV as 2, the motion continuity characteristics between the camera and the TV as 3, and the motion continuity characteristics between the oven and the TV as 4.

[0195] Electronic device 106 generates one or more semantic actions based on a ranking of relevance features and data associated with a TV, speaker, camera, oven, and smart plug. In the example, a semantic action may include at least one of the following: a suggestion to continue playing a song from the speaker on the TV, an energy consumption limit notification, a suggestion to continue displaying a motion source recently captured by the camera on the TV, and a suggestion to continue displaying a camera source captured by the oven's camera on the TV. Electronic device 106 recommends one or more of the created semantic actions to the user for operating the TV. Electronic device 106 may also recommend one or more of the created semantic actions by rendering them as keyboard overlay content on the display / output device 208 of electronic device 106.

[0196] Figure 15a and Figure 15b Another example use case scenario is described, which describes the operation of one of a plurality of devices 104a-104n in an IoT environment according to embodiments disclosed herein.

[0197] Consider an example scenario where a user points to a TV using UWB supported by electronic device 106 (example of first device 104a). In such a scenario, electronic device 106 identifies the TV as the device being pointed to by the user for operation. Electronic device 106 acquires a list of devices including speakers, cameras, ovens with built-in cameras, AC, water sensors, smart plugs, and light bulbs, and identifies the speakers, cameras, ovens, and smart plugs from the list of devices as context-sensitive devices for the TV (second devices 104b-104n). Electronic device 106 identifies the type of data that will be collected from the TV, speakers, cameras, ovens, and smart plugs. In the example, the type of data that will be collected from the devices includes at least one of the following: the state of the device, the location of the device, the current operating context of the device (e.g., content being played on the speakers, motion and video sources from the cameras, camera sources from the ovens, and energy limits from the smart plugs). Electronic device 106 collects the identified type of data from the TV, speakers, cameras, ovens, and smart plugs in online or offline mode.

[0198] When collecting data, electronic device 106 determines motion continuity characteristics with a motion continuity correlation index, conflict resolution characteristics with a conflict resolution correlation index, and fault resolution characteristics with a fault resolution correlation index for the speaker, camera, oven, and smart plug related to the TV. Electronic device 106 ranks the motion continuity characteristics between the speaker and the TV as 1, the conflict resolution characteristics between the smart plug and the TV as 2, the motion continuity characteristics between the camera and the TV as 3, and the motion continuity characteristics between the oven and the TV as 4.

[0199] Electronic device 106 generates one or more semantic actions based on a ranking of relevance features and data associated with a TV, speaker, camera, oven, and smart plug. In the example, a semantic action may include at least one of the following: a suggestion to continue playing a song from the speaker on the TV, an energy consumption limit notification, a suggestion to continue displaying a motion source recently captured by the camera on the TV, and a suggestion to continue displaying a camera source captured by the oven's camera on the TV. Electronic device 106 recommends one or more of the created semantic actions to the user for operating the TV. Electronic device 106 may also recommend one or more of the created semantic actions by rendering them as keyboard overlay content on the display / output device 208 of electronic device 106.

[0200] Figure 16a and Figure 16b Another example use case scenario is described, which describes the operation of one of a plurality of devices 104a-104n in an IoT environment according to embodiments disclosed herein.

[0201] Consider an example scenario where a user points at a first light bulb (example of first device 104a) using UWB supported by electronic device 106. In this case, electronic device 106 identifies the light bulb as a device pointed at by the user for operation. Electronic device 106 acquires a list of devices including a TV, speaker, camera, oven with a built-in camera, AC, water sensor, smart plug, and a second light bulb located in the same position as the first light bulb, and identifies the TV, second light bulb, and smart plug from the list of devices as context-sensitive devices for the light bulb (second devices 104b-104n). Electronic device 106 identifies the type of data that will be collected from the first light bulb, TV, second light bulb, and smart plug. In the example, the type of data that will be collected from the devices includes at least one of the following: the state of the device, the location of the device, the current operating context of the device (e.g., content being played on the TV, the state and settings from the second light bulb, and the energy limit from the smart plug). Electronic device 106 collects the identified type of data from the first light bulb, TV, second light bulb, and smart plug in either online or offline mode.

[0202] When collecting data, electronic device 106 determines setting recommendation features with a setting relevance index for the first bulb, considering the TV, the second bulb, and the smart plug. Electronic device 106 ranks the setting recommendation feature between the TV and the bulb as 1, and the setting recommendation feature between the second bulb and the first bulb as 2.

[0203] Electronic device 106 generates one or more semantic actions based on a ranking of relevance features and data associated with a first light bulb, TV, second light bulb, and smart plug. In the example, a semantic action may include at least one of the following: a topic suggestion for the first light bulb based on content played on the TV, and a setting for the second light bulb (e.g., a brightening or dimming setting). Electronic device 106 recommends one or more semantic actions created for operating the first light bulb to the user. Electronic device 106 may also recommend one or more semantic actions created by rendering them as keyboard overlay content on the display / output device 208 of electronic device 106. Therefore, controlling devices 104a-104n in an IoT environment based on the relevance between devices 104a-104n can enhance the user experience while controlling the devices.

[0204] The embodiments disclosed herein can be implemented by at least one software program running on at least one hardware device and performing network management functions to control elements. Figure 1. Figure 2 and Figure 3 The element shown can be at least one of a hardware device or a combination of a hardware device and a software module.

[0205] The embodiments disclosed herein describe methods and systems for controlling the operation of devices in an IoT environment. Therefore, it should be understood that the scope of protection extends to programs, and in addition to computer-readable means containing messages therein, such computer-readable storage means containing program code means for implementing one or more steps of the method when the program is run on a server, mobile device, or any suitable programmable device. In preferred embodiments, the method is implemented by or in conjunction with a software program written in, for example, Very High Speed ​​Integrated Circuit Hardware Description Language (VHDL) or another programming language, or by one or more VHDL or several software modules executed on at least one hardware device. The hardware device can be any kind of portable device that can be programmable. The device may also include devices that can be, for example, hardware devices (e.g., ASICs), or a combination of hardware and software devices (e.g., ASICs and FPGAs), or at least one microprocessor and at least one memory with software modules located therein. The method embodiments described herein can be implemented partly in hardware and partly in software. Optionally, the invention can be implemented on different hardware devices, for example, using multiple CPUs.

[0206] The foregoing description of the specific embodiments so fully reveals the general nature of the embodiments herein that others, by applying present knowledge, can readily modify and / or adapt these specific embodiments for various applications without departing from the general conception, and therefore, such adaptations and modifications should and are intended to be understood within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology used herein is for descriptive purposes and not for limitation. Therefore, although embodiments herein have been described by way of example, those skilled in the art will recognize that the embodiments herein can be practiced with modifications within the spirit and scope of the embodiments described herein.

Claims

1. A method for controlling the operation of a device in an Internet of Things (IoT) environment, the method comprising: In response to a user pointing an electronic device at a first device, the electronic device identifies the first device; The electronic device determines at least one second device that is related to the context of the first device; A capability-based device clustering model is used to identify at least one first group of related devices for the first device; Retrieve a list of registered devices from the device database; By matching the capabilities of each registered device with the capabilities of the first device, a confidence score about the first device is calculated and assigned for each registered device; as well as Based on the correlation between the capabilities of the first device and the capabilities of the at least one second device, and the operations performed by the at least one second device at the current moment, the electronic device recommends at least one suggestion for controlling at least one operation of at least one of the at least one second device and at least one of the first device. The confidence score is based on the device interaction between the registered devices in the list of registered devices and the first device.

2. The method according to claim 1, wherein, The user points to the first device using Ultra Wideband (UWB) supported by the electronic device.

3. The method according to claim 1, wherein, Determining the at least one second device by the electronic device includes: Use a device interaction prediction model to identify at least one second group of related devices for the first device; Create a union of the at least one first group of related devices and the at least one second group of related devices, wherein the created union is a set including at least one device present in either the at least one first group of related devices or the at least one second group of related devices; and The at least one device present in the at least one first group of related devices or the at least one second group of related devices is identified as the at least one second device in relation to the context of the first device.

4. The method according to claim 3, wherein, Identifying the at least one first group of related devices for the first device includes: Retrieve a list of registered devices in the IoT environment from the device database, which includes a list of devices and associated capabilities registered by the user when deploying each device in the IoT environment; The ability to identify the first device and each registered device using a device database; and A cluster of devices for the first device is formed based on the capabilities of the identified first device and each registered device, wherein the formed cluster of devices includes at least one first group of related devices from the list of registered devices that have capabilities matching the capabilities of the first device.

5. The method according to claim 3, wherein, Identifying the at least one second group of related devices for the first device includes: Discard at least one device from the list of registered devices that is associated with a confidence score below the device interaction threshold; and From the list of registered devices, at least one group of devices associated with a confidence score greater than or equal to the device interaction threshold are identified as the at least one second group of related devices for the first device.

6. The method according to claim 1, wherein, The at least one recommendation made by the electronic device for controlling at least one operation of at least one of the at least one second device and at least one first device includes: Identify the type of data collected from the first device and the at least one second device, wherein the data for each device corresponds to at least one of the following: the state, content, location, and operation performed by the corresponding device at the current moment based on the capabilities of the corresponding device; Data of the identified type is collected from the first device and the at least one second device in either online or offline mode; The correlation between the first device and the at least one second device is determined based on data collected from the first device and the at least one second device, as well as the capabilities of the first device and the at least one second device; and Based on the determined correlation between the first device and the at least one second device, at least one recommendation is made for controlling the at least one operation of the at least one second device and at least one of the first devices.

7. The method according to claim 6, wherein, Identifying the type of data collected from the first device and the at least one second device includes: Create at least one device-capability pair, wherein each device-capability pair includes a capability and at least one of the at least one second device and a first device that interacts through the capability; The device capability mapping database is queried based on each device-capability pair, wherein the device capability mapping database includes a mapping of data between each capability and its type; and Receive data from the device capability mapping database for the type of capability to be collected for each capability corresponding to at least one of the at least one second device and at least one first device.

8. The method according to claim 6, wherein, In online mode, data on the identified type is collected from the first device and the at least one second device, including: Establish a session with an IoT cloud server, wherein the IoT cloud server collects and stores data from the first device and the at least one second device; Sending a data request indicating the type of data required by the first device and the at least one second device to the IoT cloud server; and Receive data of the type indicated by the first device and the at least one second device from the IoT cloud server.

9. The method according to claim 6, wherein, In offline mode, data on the identified type is collected from the first device and the at least one second device, including: A session with the first device and the at least one second device is established using a communication network supported by the electronic device, the first device, and the at least one second device; Sending a data request of the indicated type to the first device and the at least one second device; and In response to a data transmission request, data of an indicated type is received from the first device and the at least one second device.

10. The method according to claim 6, wherein, Determining the correlation between the first device and the at least one second device includes: Multiple correlation features are determined based on the capabilities of the first device and the at least one second device, and at least one of the data collected from the first device and the at least one second device, wherein the multiple correlation features include at least one of the following: action continuity features between the first device and the at least one second device, conflict resolution features, fault resolution features, and screen intelligence features.

11. The method according to claim 10, wherein, Determining the continuity of action between the first device and the at least one second device includes: The action continuity model is trained based on a labeled dataset of candidate associated devices, along with the associated device context and capabilities; and The trained motion continuity model is used to process the capabilities of the first device and the at least one second device and the collected data to determine the motion continuity characteristics between the first device and each of the at least one second device.

12. The method of claim 11, further comprising: By using the capabilities of the trained action continuity model to process the first device and each of the second devices and the collected data, an action continuity correlation index with respect to the first device is determined for each of the at least one second device, wherein the action continuity correlation index of each second device indicates the probability of continuing at least one action of each of the at least one second device on the first device.

13. The method according to claim 10, wherein, Determining the conflict resolution features between the first device and the at least one second device includes: A first reinforcement model is trained based on at least one predefined conflict strategy to detect at least one conflict between the operation of the first device and the at least one second device; and The trained first reinforcement model is used to process data collected from each of the first device and the at least one second device to determine conflict resolution features between the first device and each of the at least one second device, wherein the conflict resolution features depict the at least one conflict between the operation of the first device and each of the at least one second device.

14. The method of claim 13, further comprising: By using a trained first reinforcement model to process data collected from the first device and each of the second devices, a conflict resolution relevance index is determined for each of the second devices regarding the first device, wherein the conflict resolution relevance index determined for each of the second devices depicts the probability of conflict between the operation of the first device and each of the second devices.

15. An electronic device comprising: Memory; as well as The controller, coupled to the memory, is configured to perform the following operations: In response to the user pointing the electronic device at the first device, the first device is identified; Determine at least one second device that is related to the context of the first device; as well as A capability-based device clustering model is used to identify at least one first group of related devices for the first device; Retrieve a list of registered devices from the device database; By matching the capabilities of each registered device with the capabilities of the first device, a confidence score about the first device is calculated and assigned for each registered device; as well as Based on the correlation between the capabilities of the first device and the capabilities of the at least one second device, and the operations performed by the at least one second device at the current moment, at least one recommendation is made for controlling at least one operation of at least one of the at least one second device and at least one of the first device. The confidence score is based on the device interaction between the registered devices in the list of registered devices and the first device.

Images