Method and system for enabling seamless indirect interaction

Abstract

Methods, systems, and apparatus are provided for enabling indirect interaction between users in an Internet of Things (IoT) environment. A method includes receiving, by a first device, an utterance from a first user, where the utterance is related to at least one task to be performed by the first user; identifying, by the first device, one or more second users related to the at least one task based on receiving the utterance; providing, by the first device, an interactive interface to one or more second devices, the one or more second devices being closer to the one or more second users than the first device; receiving, by the first device, one or more inputs corresponding to the at least one task from one or more second users through the interactive interface; and appending, by the first device, the received one or more inputs to the at least one task.

Description

Technical Field

[0001] The present disclosure relates to human-computer interaction, and more particularly to enabling seamless indirect interaction between one or more members in a particular environment. Background Art

[0002] Smart devices in a specific environment may be able to interact with the outside world and with each other. However, specific methods and devices are needed to enable indirect interactions between devices within the environment.

[0003] As an example, consider an environment such as a family where the mother is going to go shopping. In order to prepare to do so, she may interact directly with each family member (e.g., by calling, chatting, face-to-face, etc.) to obtain their specific requirements / needs.

[0004] As another example scenario, consider an environment such as a garage where a person may be trapped in a cramped space and may need a new tool box, but no one else is within earshot. The trapped person can either leave the cramped space and retrieve the tool box by himself, or can contact another person to retrieve the tool box.

[0005] As yet another example, consider an environment such as a home where a child may be studying and may be disturbed by the loud volume of a television. The child may get up and ask that the television be turned off, or turned off.

[0006] These examples show that smart homes may not be smart enough to enable seamless indirect interactions, resulting in inconvenient, repetitive actions, and annoying processes that require more effort and time. Even if smart homes provide some opportunities for indirect interactions, users may still choose less efficient direct interactions. Summary of the invention

[0007] Solution to the problem

[0008] Methods, systems, and apparatus for enabling seamless indirect interactions in an environment are provided, wherein the system enables seamless indirect interactions between devices and users present in an Internet of Things (IoT) environment.

[0009] Methods, systems, and apparatus for predicting at least one context based on an utterance received from a user are also provided.

[0010] Methods, systems, and apparatus for identifying one or more second users associated with the predicted context are also provided.

[0011] Methods, systems, and apparatus for predicting the current location of one or more second users based on IoT data are also provided.

[0012] Also provided are methods, systems, and apparatus for associating obtained user context data and environmental context data from a database with at least one context from an utterance.

[0013] Also provided are methods, systems, and apparatus for providing at least one of interaction and suggestion to one or more second users through an interactable interface using a deep learning method.

[0014] Also provided are methods, systems, and apparatus for appending one or more inputs from a second user related to a task to be performed by a first user.

[0015] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0016] According to one aspect of the present disclosure, a method for enabling indirect interaction between users in an Internet of Things (IoT) environment includes: receiving, by a first device, an utterance from a first user, wherein the utterance is related to at least one task to be performed by the first user; based on receiving the utterance, identifying, by the first device, one or more second users related to the at least one task; providing, by the first device, an interactive interface to one or more second devices, wherein the one or more second devices are closer to the one or more second users than the first device; receiving, by the first device, one or more inputs corresponding to at least one task from the one or more second users through the interactive interface; and attaching, by the first device, the received one or more inputs to at least one task.

[0017] Identifying one or more second users related to the task may include: predicting, by the first device, at least one context based on an utterance received from the first user; and identifying, by the first device, one or more second users related to the predicted at least one context.

[0018] Predicting the at least one context and identifying the one or more second users may be performed using a trained learning method.

[0019] The method may also include: obtaining IoT data by the first device; determining, by the first device, a location history of at least one device last accessed by one or more second users based on the IoT data; and determining, by the first device, a current location of the one or more second users based on the location history.

[0020] The one or more second devices may be selected by the first device based on availability of the one or more second devices and capabilities of the one or more second devices to perform at least one of delivering a message and receiving a message.

[0021] Providing an interactive interface to one or more second devices may include generating at least one of interactions and suggestions to provide to one or more second users, and generating at least one of the interactions and suggestions may include: obtaining user context data and environmental context data by the first device; and associating the user context data and environmental context data with at least one predicted context by the first device.

[0022] The method may also include: generating, by the first device, at least one suggestion based on a match between at least a portion of the user context data and the environmental context data and at least one predicted context, wherein the at least one suggestion is provided based on data stored in one or more second devices or is provided as a recommendation related to at least one predicted context; and generating, by the first device, at least one interaction for directly conveying a message based on the utterance based on a mismatch between at least a portion of the user context data and the environmental context data and at least one predicted context.

[0023] The one or more inputs may include at least one of a requirement corresponding to the at least one task and action content to be performed according to the at least one task.

[0024] According to one aspect of the present disclosure, a device for enabling indirect interaction between users in an Internet of Things (IoT) environment includes: at least one processor, which is configured to: receive an utterance from a first user, wherein the utterance is related to at least one task to be performed by the first user; based on receiving the utterance, identify one or more second users related to the at least one task; provide an interactive interface to one or more target devices, the one or more target devices being closer to one or more second users than the device; receive one or more inputs corresponding to the at least one task from the one or more second users through the interactive interface; and attach the received one or more inputs to at least one task.

[0025] The at least one processor may also be configured to: predict at least one context based on the utterance received from the first user; and identify one or more second users associated with the predicted at least one context.

[0026] The at least one processor may also be configured to perform the prediction of the at least one context and the identification of the one or more second users using a trained learning method.

[0027] The at least one processor may be further configured to: obtain IoT data; determine a location history of at least one device last accessed by one or more second users based on the IoT data; and determine a current location of the one or more second users based on the location history.

[0028] The at least one processor may be further configured to select the one or more target devices based on availability of the one or more target devices and capabilities of the one or more target devices to perform at least one of delivering the message and receiving the message.

[0029] At least one processor may also be configured to provide an interactive interface to one or more target devices by generating at least one of interactions and suggestions to be provided to one or more second users, and in order to generate at least one of the interactions and suggestions, at least one processor may also be configured to: obtain user context data and environmental context data; and associate the user context data and environmental context data with at least one predicted context.

[0030] At least one processor may also be configured to: generate at least one suggestion based on a match between at least a portion of the user context data and the environmental context data and at least one predicted context, wherein the at least one suggestion is provided based on data stored in one or more target devices or provided as a recommendation related to at least one predicted context; and generate at least one interaction for a message based on direct communication of the speech based on a mismatch between at least a portion of the user context data and the environmental context data and at least one predicted context.

[0031] According to one aspect of the present disclosure, a system for enabling indirect interaction between users in an Internet of Things (IoT) environment includes: one or more target devices; and a source device, which includes at least one processor, and the at least one processor is configured to: receive an utterance from a first user, wherein the utterance is related to at least one task to be performed by the first user; based on receiving the utterance, identify one or more second users related to the at least one task; provide an interactive interface to the one or more target devices, wherein the one or more target devices are closer to the one or more second users than the source device; receive one or more inputs corresponding to the at least one task from the one or more second users through the interactive interface; and attach the received one or more inputs to the at least one task. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0033] Figure 1 is an example block diagram depicting components of a system for providing seamless indirect interaction between devices and users present in an Internet of Things (IoT) environment according to one or more embodiments;

[0034] Figure 2 An architecture of a system for providing seamless indirect interaction according to one or more embodiments is shown;

[0035] Figure 3 is a flow chart of a method for providing seamless indirect interaction between devices and users present in an IoT environment according to one or more embodiments;

[0036] Figure 4 is a block diagram of a system integrated with an intelligent application module according to one or more embodiments;

[0037] Figure 5 is a use case diagram for shopping according to one or more embodiments;

[0038] Figure 6 illustrates the architecture of an intelligent generator module according to one or more embodiments;

[0039] Figure 7 is a use case in which a child's study is interrupted by a loud television according to one or more embodiments;

[0040] Figure 8 is a use case diagram of a person being trapped in a cramped place and needing a new toolbox according to one or more embodiments;

[0041] Fig. 9 is a use case diagram for ordering dinner online according to one or more embodiments;

[0042] Fig.10 is a use case diagram for food preparation according to one or more embodiments; and

[0043] Fig.11 is a use case diagram for rent payment according to one or more embodiments. DETAILED DESCRIPTION

[0044] The present disclosure and its various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the drawings and described in detail in the following description. The description of known components and processing techniques may be omitted so as not to unnecessarily obscure the embodiments of this article. The examples used herein are only intended to facilitate understanding of the manner in which the embodiments of this article may be practiced, and further enable those skilled in the art to practice the embodiments of this article. Therefore, the examples should not be interpreted as limiting the scope of the embodiments of this article.

[0045] Embodiments herein may relate to methods, apparatuses, and systems for seamless indirect interaction of devices in an Internet of Things (IoT) environment, wherein an engine enables seamless indirect interaction between devices and users present in the IoT environment. Referring now to the drawings, and more particularly to Figures 1 to 11 , wherein like reference numerals denote corresponding features throughout the drawings, exemplary embodiments are described below.

[0046] Figure 1A system 100 for enabling seamless indirect interaction in an IoT environment according to one or more embodiments is depicted. The system 100 may include a source device 104 corresponding to a first user 102, and one or more target devices 106 corresponding to one or more second users 108. In an embodiment, the source device 104 may be referred to as a first device, and the one or more target devices 106 may be referred to as one or more second devices. In an embodiment, the first user 102 may be referred to as a source user or a source entity, and the one or more second users 108 may be referred to as one or more target users or one or more target entities. The target device 106 may also be a real-world device present in the real-world environment of the second user 108. Examples of the target device 106 may include a desktop computer, a laptop computer, a mobile device such as a smart phone, a personal digital assistant, a wearable device, a kitchen appliance, and a smart appliance, but the embodiment is not limited thereto. The target device 106 may be one or more other devices present in a location closer to the one or more second users than the first device 104. The second user 108 may be, for example, a target user.

[0047] The source device 104 may include a processor 110 and a communication module 112. The source device 104 may be a real-world device operating in the real-world environment of the first user 102. Examples of the source device 104 may include a desktop computer, a laptop computer, a mobile device such as a smart phone, a personal digital assistant, a wearable device, a kitchen appliance, and a smart appliance, but the embodiment is not limited thereto.

[0048] In an embodiment, the processor 110 may be configured to enable a device such as the source device 104 to collect information or requirements from one or more target users based on one or more tasks. A task may be a task to be performed by and / or on behalf of the first user 102, and it may depend on the input of one or more target users. The processor 110 may determine the location of one or more target devices 106 and entities related to the task, and may provide suggestions to one or more target users so that the target users can be used by at least one of the source device 104 and the first user 102 to make an informed decision. A plurality of modules may be used to interface a device to one or more target devices.

[0049] In an embodiment, the processor 110 may include one or more of a microprocessor, a circuit, and other hardware configured for processing. The processor 110 may be configured to execute instructions stored in a database.

[0050] The processor 110 may be at least one of a single processor, multiple processors, multiple homogeneous or heterogeneous cores, multiple central processing units (CPUs) of different kinds, a microcontroller, special media, and other accelerators. The processor 110 may be an application processor (AP), a graphics processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an artificial intelligence (AI) dedicated processor such as a neural processing unit (NPU).

[0051] In an embodiment, the communication module 112 may be configured to enable communication between the energy device 104 and one or more target devices 106. The server may be configured or programmed to execute instructions of the first device 104. The communication module 112 through which the source device 104 and the server communicate may be in the form of a wired network, a wireless network, or a combination thereof. Example wired and wireless communication networks may include a global positioning system (GPS), a global system for mobile communications (GSM), a local area network (LAN), wireless fidelity (Wi-Fi) compatibility, and near field communication (NFC), but the embodiment is not limited thereto. Depending on the use environment, examples of wireless communication may also include one or more of Bluetooth, ZigBee, short-range wireless communication such as ultra-wideband (UWB), medium-range wireless communication such as Wi-Fi, and long-range wireless communication such as 3G / 4G / 5G or Worldwide Interoperability for Microwave Access (WiMAX), but the embodiment is not limited thereto.

[0052] although Figure 1 Various hardware components of system 100 are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, system 100 may include fewer or more components. In addition, the markings or names of the components are only for illustrative purposes and do not limit the scope of the present disclosure. One or more components can be combined together to perform the same or substantially similar functions in system 100.

[0053] Figure 2The architecture of a system 100 for providing seamless indirect interaction between devices and users present in an IoT environment according to one or more embodiments is shown. In an embodiment, the environment may refer to a home, an office, etc. The processor 110 may include a context predictor module 202, an entity recognizer module 204, a location detector module 206, a target device selector module 208, and an intelligence generator module 210. The processor 110 may detect input from a user (e.g., a first user 102 or one or more second users 108). In an embodiment, the input may include at least one of speech, text, etc., but the embodiment is not limited thereto. Based on the input, the context predictor module 202 may determine one or more contexts. For example, the context predictor module 202 may determine the context of a speech received from the first user 102. In an embodiment, the context predictor module 202 may determine the context of the speech in real time. The context predictor module 202 may predict the context using a trained learning method, wherein the learning method may be trained using data maintained in a database. The entity recognizer module 204 may determine entities (or users) that may be involved in the interaction. The entity recognizer module 204 can extract entities (or users) based on the received utterances and classify the entities (or users) into source entities and target entities. The entity recognizer module 204 can use a trained learning method to identify one or more second users 108, wherein the learning method can be trained using data maintained in a database. The location detector module 206 can determine the location of the first user 102 and one or more second users 108. The location detector module 206 can obtain the location of the data based on camera data, user profile data, and historical data from the database. In addition, the location detector module 206 can determine the location history of one or more devices last accessed by one or more second users 108 based on IoT data, and can predict the current location of one or more second users 108 based on the determined location history. The target device selector module 208 can determine one or more target devices 106 corresponding to the entity associated with the interaction. The target device selector module 208 can use a learning method, based on the availability of one or more target devices 106 and the ability to perform at least one of delivering and receiving messages to select one or more target devices 106. In an embodiment, the same device can be selected for both categories. In an embodiment, a different device may be selected for each category. The system 100 may provide at least one interaction / suggestion to the one or more second users 108. The system 100 may also provide feedback to the one or more second users 108 in the form of an action or an utterance.

[0054] The database may include one or more volatile and non-volatile memory components capable of storing data and instructions to be executed. Examples of memory modules may include NAND, embedded multimedia cards (eMMC), secure digital (SD) cards, universal serial bus (USB), serial advanced technology attachment (SATA), and solid-state drives (SSD), but embodiments are not limited thereto. The memory module may also include one or more computer-readable storage media. Examples of non-volatile storage elements may include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable (EEPROM) memory. In addition, in some examples, the memory module may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier or propagation signal. However, the term "non-transitory" should not be interpreted to mean that the memory module is non-removable. In some instances, non-transitory storage media may store data that may change over time (e.g., in a random access memory (RAM) or cache memory).

[0055] The intelligent generator module 210 can use a deep learning method to generate at least one interaction and suggestion to provide to one or more second users through an interactive interface. The intelligent generator module 210 can obtain user context data and environmental context data from a database. In addition, the intelligent generator module 210 can associate the obtained user context data and environmental context data with at least one context from the discourse. In addition, if at least a portion of the obtained user context data and environmental context data matches at least one context from the discourse, the intelligent generator module 210 can generate at least one suggestion. The intelligent generator module 210 can provide suggestions based on data stored in one or more target devices 106 based on information related to past searches or previous history or as a recommendation related to at least one predicted context. If at least a portion of the obtained user context data and environmental context data does not match at least one context from the discourse, the intelligent generator module 210 can generate at least one interaction to directly convey the message from the discourse.

[0056] The system 100 may enable the first user 102 to collect information or requirements based on one or more tasks from one or more second users 108. In addition, the system 100 may attach one or more inputs from the one or more second users 108 related to the tasks to be performed by the first user 102, wherein the one or more inputs may include at least one of the requirements and action content to be performed.

[0057] Figure 3is a flow chart illustrating a method 300 for enabling seamless indirect interaction in an environment according to one or more embodiments. Operations 302 to 318 described below may be processed by the processor 110. At operation 302, the method includes receiving, by the source device 104, an utterance from a first user 102, wherein the utterance relates to one or more tasks to be performed by the first user 102 and / or performed on behalf of the first user 102 and / or that the first user 102 wants to perform.

[0058] At operation 304 , the method includes predicting, by the source device 104 , at least one context based on the utterance received from the first user 102 .

[0059] At operation 306, the method includes identifying, by the source device 104, one or more second users 108 associated with the predicted at least one context, wherein the method of predicting at least one context and identifying the one or more second users 108 is performed using a trained learning method, and the learning method is trained using data maintained in a database.

[0060] At operation 308, the method includes obtaining IoT data by the source device 104, wherein the IoT data may include at least one of camera data, user profile data, and historical data from a database, but the embodiment is not limited thereto. Using the IoT data, the source device 104 may determine a location history of at least one last accessed device by one or more second users 108. The source device may predict a current location of the one or more second users 108 based on the determined location history.

[0061] At operation 310 , the method includes identifying one or more target devices 108 that are closer to one or more second users 108 than the first device 104 , wherein the source device 104 selects the one or more target devices 106 based on availability and capabilities for performing at least one of delivering and receiving messages using a learning method.

[0062] At operation 312, the method includes providing, by the source device 104, an interactable interface via one or more target devices 106 present in a location closer to the one or more second users 108, wherein the source device 104 uses a deep learning method to generate at least one of an interaction and a suggestion to be provided to the one or more second users 108 through the interactable interface. The method also includes obtaining, by the source device 104, user context data and environmental context data from a database, and associating, by the source device 104, the obtained user context data and environmental context data with at least one context from the utterance.

[0063] At operation 314 , the method includes generating, by the source device 104 , at least one suggestion based on matching at least a portion of the obtained user context data and environmental context data with at least one context from the utterance.

[0064] At operation 316 , the method includes generating, by the source device 104 , at least one interaction for directly communicating the message from the utterance based on at least a portion of the obtained user context data and the environmental context data not matching the at least one context from the utterance.

[0065] At operation 318 , the method includes receiving, by the source device 104 , one or more inputs corresponding to the task from the one or more second users 108 through the interface, and appending, by the source device 104 , the received one or more inputs to the task to be performed by the first user 102 .

[0066] One or more operations of the method 300 described above may be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, the operations of the method 300 may be omitted. Figure 3 Some of the actions listed in .

[0067] The processor 110 may control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in the nonvolatile memory and the volatile memory. The predefined operating rules or artificial intelligence models are provided through training or learning.

[0068] Here, providing by learning can mean making a predefined operating rule or AI model of desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device itself that executes the AI ​​according to the embodiment, and / or can be implemented by a separate server / system.

[0069] A learning algorithm may be a method for training a predetermined device (e.g., a robot) using a plurality of learning data to enable, allow, or control the device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0070] Figure 4An example architecture of a system 100 according to one or more embodiments is shown, which can be integrated with an intelligent application module and includes AI voice and IoT connectivity. The architecture describes the interconnection of components of all systems with device features based on context and reminders, and performs multiple functions. Functions may include device control notifications, intelligent recommendations, and action scheduling, but the embodiments are not limited thereto. The system 100 can send and receive information while communicating with the device and performing its functions. For example, a user can turn on or off device notifications and control the functions of the device. The system 100 can perform actions based on intelligent recommendations provided by the device in a predetermined manner. The system 100 can communicate with other smart devices and obtain complete details about the location of users and devices in the IoT environment through a target device selection unit and a location detection unit. The location detector can be GPS and Bluetooth as well as crowdsourced Wi-Fi hotspots, cellular tower locations, etc. For example, a target device 106 can be assigned to and / or associated with a physical location. A target device 106 with the same or similar name / brand can be selected. The target device 106 can be paired with a device by default, and so on.

[0071] Figure 5 An example scenario according to one or more embodiments is shown, in which a mom 502 in a house is going out shopping. The system 100 can predict the context of a task (e.g., shopping) of a source user 102 (e.g., mom 502), and can identify target users 108 (e.g., children 508a, 508b, 508c, and 508d) that may be interested in it. The system 100 can determine the location of smart devices and entities, and determine one or more target devices 106 corresponding to the entities. The system 100 can interact with one or more target devices, wherein the children 508a-508d are asked whether they have any shopping requirements. The children 508a-508d can provide feedback on one or more items to be picked up. The system 100 can then provide feedback / additional actions / other utterances to the mother accordingly.

[0072] In the example shown herein, mom 502 provides an utterance asking her children 508a-508d about their requests (if any), such as "Hey, kids, I'm shopping." The system 100 provides suggestions or interactions to her children 508a-508d through an interactive interface as "Your mom is shopping." In bedroom 1, the system 100 provides a suggestion "Should I tell her to buy a Samsung Galaxy display that's on sale with a larger screen size and refresh rate than your current display?" The child 508a responds: "Yes." In bedroom 2, child 508b responds, "Please, Mom, pick up my laundry." In bedroom 3, children 508c and 508d respond, "We want chocolate! And if Mom asks about the chocolate in the fridge, tell her we're done." The system 100 appends the inputs received from the target device 106 (associated with children 508a-508d) to the source device 104 (associated with mom 502) through the interactive interface, which are "Add Samsung Galaxy display to shopping list," "Create reminder to bring laundry to child 508b," "Add chocolate to shopping list," and "Create action: If asked about "chocolate in the fridge," respond "children are done." Therefore, in the example shown herein, the system 100 only provides suggestions to one user (e.g., child 508a) as recommendations.

[0073] The context may be shopping, ordering a meal, preparing food, paying rent, etc. In the example shown herein, the context is identified as shopping.

[0074] In the example shown herein, after classification, mom 502 may be considered a source entity or source user 102 , and children 508 a - 508 d may be considered target entities or one or more target users 108 .

[0075] In the example shown herein, the location detector module 206 identifies that the first user 102 (eg, mom 502) is in the living room, and the second user 108 is in various bedrooms (eg, child 508a is in bedroom 1, child 508b is in bedroom 2, and children 508c and 508d are in bedroom 3).

[0076] In bedroom 1, the target device selector module 208 may select a smart display as an input and output device. In bedroom 2, the target device selector module 208 may select a smart speaker as an input and output device. In bedroom 3, the target device selector module 208 may select a smart speaker as an input device and a smart TV as an output device.

[0077] The intelligent generator module 210 may provide at least one interaction and suggestion to the target user 108 through an interactive interface. The intelligent generator module 210 may not involve unintended entities. In the example shown herein, the husband may be in the kitchen and may not be a target entity.

[0078] Figure 6 An example of the architecture of the intelligent generator module 210 according to one or more embodiments is illustrated. The intelligent generator module 210 may include a decision module. The intelligent generator module 210 may obtain user context data and environmental context data from a database. The decision module of the intelligent generator module 210 may provide interaction or suggestions to one or more second users 108 to make wise decisions. Given Y = "speech context", the intelligent generator module 210 may provide a probability function of X = "user context" plus "environmental context". Then, the decision module may provide interaction or suggestions according to the function shown in equation 1 below. The function may provide suggestions when at least some parts of X and Y are related through an interactive interface using a deep learning method, or may generate interactions with one or more second users. That is, directly convey the message.

[0079] Probability function (X: (user context, environment context) | Y: (utterance context)) = {recommendation, if X & Y are relevant} {interaction, if X & Y are not relevant}.

[0080] In the example discussed above, the context (e.g., shopping) may be identified as “kids I am shopping” based on input / utterance received from the first user 102 (e.g., mom 502). Furthermore, when one or more second users have been determined to be in different bedrooms, an environmental context (e.g., sale: (Samsung store / summer sale)) may be determined based on the identified context (e.g., shopping).

[0081] In the case of the child 502a context, the obtained user context data and the environmental context data may match one context from the utterance, and the system may generate a suggestion as a recommendation with the identified device (display: old, small screen, and low resolution), as “Your mom is shopping. Should I tell her to buy the on-sale Samsung Galaxy display with a larger screen size and refresh rate than your current display?” In the case of the user context of children 502b-502d, the obtained user context data and the environmental context data may not match one context from the utterance, so the system 100 only provides an interaction instead of a suggestion.

[0082] Figure 7An example scenario according to one or more embodiments is shown, in which the loud volume of a TV 706 disturbs a child 702 who is studying. The child 702 may provide a request to the father 708 to reduce the volume when she is studying, such as "Dad, I am studying, lower the volume". The system 100 provides a suggestion to the father 708 to lower the volume on the TV 706: "The child is disturbed. I should lower the volume".

[0083] Figure 8 An example scenario is shown in accordance with one or more embodiments, where a person 802 is trapped under his vehicle and needs a toolbox that he cannot reach. Person 802 may provide utterances requesting his wife 808 to fetch the toolbox from the bedroom, such as "Tell Honey to bring my new toolbox from the bedroom." System 100 provides a suggestion to wife 808 in front of refrigerator 806 (because wife 808 has been determined to be in the kitchen) "Bob is going to get his new toolbox from the bedroom. He's in the garage," to fetch the toolbox from the bedroom and provide the toolbox to the person in the garage.

[0084] Fig. 9 An example scenario is shown in which a person 902 provides an utterance asking his roommate 908a and roommate 908b about dinner ordering, such as "Hey, what do you guys want to order for dinner?" The context identified herein is dinner ordering. After classifying the entities, person 902 can be considered a source entity or source user 102, and roommates 908a and 908b can be considered target entities or one or more target users 108. The location detector module 206 can determine a first user 102 (e.g., person 802) as being in a living room, one second user 108 (e.g., roommate 908a) can be determined as being in bedroom 1, and another second user 108 (e.g., roommate 908b) can be determined as being in bedroom 2. In bedroom 1, the target device selector module 208 selects a smart display as an input and output device. In bedroom 2, the target device selector module 208 selects a laptop as an input and output device. In the case of roommate 908a, system 100 provides a suggestion to roommate 908a through an interactive interface based on roommate 908a's history of mainly ordering Chicken Biryani for dinner, "Last time you had Chicken Biryani for dinner. Would you like it again this time?" Roommate 908b responds "Yes". In the case of roommate 908b, system 100 provides interaction to roommate 908b, and he responds "I will have egg fried rice". System 100 appends the input received from the target devices associated with roommates 908a and 908b to the device associated with person 902 through the interactive interface, such as "Roommate 908a wants Chicken Biryani" and "Roommate 908b wants egg fried rice", respectively.

[0085] Fig.10 An example scenario is shown, in which a mother 1002 in a house provides an utterance asking her child 1008a and her child 1008b about food preparation, such as "Hi everyone, what food is prepared?" The context identified herein is food preparation. After entity classification, the mother 1002 can be considered as a source entity or source user 102, and the children 1008a and 1008b can be considered as target entities or one or more target users 108. When in the kitchen, a second user 108 (e.g., child 1008a) can be determined to be in bedroom 1, and another second user 108 (e.g., child 1008b) can be determined to be in bedroom 2. In bedroom 1, the target device selector module 208 can select a smart display as an input and output device. In bedroom 2, the target device selector module 208 can select a speaker as an input and output device. In the case of child 1008a, the system 100 can provide interaction through an interactive interface, such as "Mom is asking what food is prepared". The child 1008a can reply "Anything is OK!". In the case of child B, the system 100 may provide the interaction to child 1008b, and child 1008b may respond, "I will have a burger." The system 100 may append the input received from the target devices associated with children 1008a and 1008b to the device associated with mom 1002 through the interactable interface, such as "child 1008b wants a burger and child 1008a has no preference."

[0086] Fig.11An example scenario is shown in which a person 1102 provides an utterance asking his roommates 1108a and 1108b about rent payment, e.g., "Hey guys, we need to pay the rent for the house." The context identified herein is rent payment. After classifying the entities, the person 1102 can be considered a source entity or source user 102, and the roommates 1108a and 1108b can be considered target entities or one or more target users 108. The location detector module 206 can determine that the source user 102 (e.g., person 1102) is in the living room, one second user 108 (e.g., roommate 1108a) can be determined to be in bedroom 1, and another second user 108 (e.g., roommate 1108b) can be determined to be in bedroom 2. In bedroom 1, the target device selector module 208 selects a smart display as an input and output device. In bedroom 2, the target device selector module 208 selects a laptop as an input and output device. In the case of roommate 1108a, system 100 provides general advice to his flatmate via the interactive interface, such as "Need to pay. Should I launch your Samsung Wallet?" Roommate 1108a responds, "No, tell him to pay on my behalf and I'll pay him back tomorrow." In the case of roommate 1108b, he responds, "Yes!" System 100 appends the input received from the target devices associated with roommates 1108a and 1108b via the interactive interface to the device associated with person 1102 as "Roommate 1108a will pay his portion tomorrow" and "Amount $$$ from roommate 1108b," respectively.

[0087] Embodiments herein enable concurrent interactions, where multiple interactions simultaneously are possible in parallel. Embodiments herein enable ease of use, where context or specification of entities are not explicitly required. Embodiments herein enable reduction of human effort, where system 100 suggests and performs actions based on previous contexts that reduce human effort. Embodiments herein provide faster execution, where the time spent completing a task will be less than manually. Embodiments herein reduce cognitive load, where system 100 will provide the most relevant suggestions based on context and priority.

[0088] The various actions, actions, blocks, steps, etc. in method 300 may be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, some of the actions, actions, blocks, steps, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the present disclosure.

[0089] The embodiments disclosed herein may be implemented by at least one software program running on at least one hardware device and performing network management functions to control an element. The element may be at least one of a hardware device or a combination of a hardware device and a software module.

[0090] The foregoing description of specific embodiments is not intended to be limiting, and the above embodiments may be modified and / or adapted for various applications without departing from the general concept. Therefore, such adaptations and modifications should and are intended to be understood within the meaning, scope, and range of the disclosed embodiments. It should be understood that the wording or terminology employed herein is for descriptive and not limiting purposes. Therefore, although specific embodiments are described above according to at least one embodiment, those skilled in the art will recognize that the embodiments herein may be practiced by modification within the spirit and scope of the embodiments as described herein.

Claims

1. A method for enabling indirect interaction between users in an Internet of Things (IoT) environment, the method comprising: receiving, by a first device, an utterance from a first user, wherein the utterance is related to at least one task to be performed by the first user; Based on receiving the utterance, identifying, by the first device, one or more second users associated with the at least one task; Providing an interactive interface by the first device to one or more second devices, the one or more second devices being closer to one or more second users than the first device; receiving, by the first device through the interactive interface, one or more inputs corresponding to the at least one task from one or more second users; and The one or more inputs received are appended, by the first device, to the at least one task.

2. The method according to claim 1, wherein: Identifying one or more second users associated with the task includes: predicting, by the first device, at least one context based on the utterance received from the first user; and One or more second users associated with the predicted at least one context are identified by the first device.

3. The method according to claim 2, wherein: Predicting at least one context and identifying one or more second users are performed using a trained learning method.

4. The method according to claim 1, further comprising: Obtaining IoT data by the first device; determining, by the first device, based on the IoT data, a location history of at least one device last accessed by one or more second users based on the IoT data; as well as A current location of one or more second users is determined by the first device based on the location history.

5. The method according to claim 1, wherein: The one or more second devices are selected by the first device based on availability of the one or more second devices and capabilities of the one or more second devices to perform at least one of delivering a message and receiving a message.

6. The method according to claim 2, wherein: Providing the interactable interface to the one or more second devices includes generating at least one of an interaction and a suggestion to be provided to the one or more second users, and Wherein generating at least one of an interaction and a suggestion comprises: Obtaining, by the first device, user context data and environmental context data; and The user context data and the environmental context data are associated with the predicted at least one context by the first device.

7. The method according to claim 6, further comprising: generating, by the first device, at least one suggestion based on matching at least a portion of the user context data and the environmental context data with the predicted at least one context, wherein the at least one suggestion is provided based on data stored in one or more second devices or is provided as a recommendation related to the predicted at least one context; and Based on at least a portion of the user context data and the environmental context data not matching the predicted at least one context, generating, by the first device, at least one interaction for directly communicating a message based on the utterance.

8. The method according to claim 1, wherein: The one or more inputs include at least one of a requirement corresponding to the at least one task and action content to be performed according to the at least one task.

9. An apparatus for enabling indirect interaction between users in an Internet of Things (IoT) environment, the apparatus comprising: At least one processor configured to: receiving an utterance from a first user, wherein the utterance is related to at least one task to be performed by the first user; identifying one or more second users associated with the at least one task based on receiving the utterance; providing an interactable interface to one or more target devices, the one or more target devices being closer to one or more second users than the device; receiving, through the interactable interface, one or more inputs corresponding to the at least one task from one or more second users; and The received one or more inputs are appended to the at least one task.

10. The device according to claim 9, wherein: The at least one processor is further configured to: predicting at least one context based on the utterance received from the first user; and One or more second users associated with the predicted at least one context are identified.

11. The device according to claim 10, wherein: The at least one processor is further configured to perform prediction of at least one context and identification of one or more second users using a trained learning method.

12. The device according to claim 9, wherein: The at least one processor is further configured to: Obtain IoT data; determining a location history of at least one device last accessed by one or more second users based on the IoT data; and A current location of the one or more second users is determined based on the location history.

13. The apparatus according to claim 9, wherein: The at least one processor is further configured to select the one or more target devices based on availability of the one or more target devices and capabilities of the one or more target devices to perform at least one of delivering a message and receiving a message.

14. The apparatus according to claim 10, wherein: The at least one processor is further configured to provide an interactable interface to the one or more target devices by generating at least one of an interaction and a suggestion to be provided to the one or more second users, and Wherein, in order to generate the at least one of the interaction and the suggestion, the at least one processor is further configured to: Obtaining user context data and environmental context data; and The user context data and the environmental context data are associated with the predicted at least one context.

15. A system for enabling indirect interaction between users in an Internet of Things (IoT) environment, the system comprising: one or more target devices; as well as A source device, comprising at least one processor, wherein the at least one processor is configured to: receiving an utterance from a first user, wherein the utterance is related to at least one task to be performed by the first user; identifying one or more second users associated with the at least one task based on receiving the utterance; providing an interactable interface to one or more target devices, wherein the one or more target devices are closer to the one or more second users than the source device; receiving, through the interactable interface, one or more inputs corresponding to the at least one task from one or more second users; and The received one or more inputs are appended to the at least one task.

Images