Smart home bubbles forming

Abstract

A second action is identified by the edge device. The second action is associated with a second network-connected device among a plurality of network-connected devices in the environment. The second network-connected device is unrelated to the first network-connected device. The edge device detects a relationship between the first network-connected device and the second network-connected device. The edge device generates a network-connected device bubble. The network-connected device bubble is based on the detected relationship.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to smart homes, and more particularly to identifying relationships between actions in a smart home and creating bubbles. [Background technology]

[0002] A smart home can include common household items with network connectivity (e.g., networked devices). The networked devices can communicate with each other and with a user using the network connectivity. The networked devices can perform various actions (e.g., play music, turn on lights, etc.) based on requests from a user. Summary of the Invention

[0003] According to embodiments, a method, system, and computer program product are disclosed, in which a first action is identified by an edge device at a first time, the first action being associated with a first network-connected device of a plurality of network-connected devices in an environment.

[0004] A second action is identified by the edge device, the second action being associated with a second network-connected device of the plurality of network-connected devices in the environment that is unrelated to the first network-connected device, a relationship between the first network-connected device and the second network-connected device is detected by the edge device, and a network-connected device bubble is generated by the edge device, the network-connected device bubble being based on the detected relationship.

[0005] According to an embodiment, the network-connected device bubble may coordinate data and actions for devices with which it has a relationship.

[0006] The above summary is not intended to describe each example embodiment or every implementation of the present disclosure. [Brief explanation of the drawings]

[0007] The drawings included in this application are incorporated into and form a part of this specification. These drawings illustrate embodiments of the present disclosure and, together with the description, explain the principles of the present disclosure. The drawings are only illustrative of some embodiments and are not intended to limit the disclosure.

[0008] [Figure 1] FIG. 1 illustrates representative major components of an example computer system that can be used in accordance with some embodiments of the present disclosure. [Figure 2] FIG. 1 illustrates a model representative of one or more artificial neural networks capable of detecting relationships and generating network-connected device bubbles, according to some embodiments of the present disclosure. [Figure 3] FIG. 1 illustrates an example of implementing a bubble-based smart home (BBSH) in a real-world environment, according to some embodiments of the present disclosure. [Figure 4A] FIG. 1 illustrates an example of a user interface operated by a BBSH, according to some embodiments of the present disclosure. [Figure 4B] FIG. 10 illustrates an example of a user interface after BBSH execution, according to some embodiments of the present disclosure. [Figure 5] FIG. 1 illustrates an example of a method performed by a BBSH according to some embodiments of the present disclosure.

[0009] While the invention is susceptible to various modifications and alternative forms, specific aspects of the invention have been shown by way of example in the drawings and will be described in detail. However, it is not intended to limit the invention to the particular embodiments described. Rather, it is intended to cover all modifications, equivalents, and alternatives falling within the scope of the invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Aspects of the present disclosure relate to smart homes, and more particularly to generating action bubbles in smart homes. While the present disclosure is not necessarily limited to such applications, various aspects of the present disclosure can be understood through the discussion of various examples using this context.

[0011] A smart home can include common household items (e.g., network-connected devices) with network connectivity in an environment. The environment can be a home, an office, a library, a school, or other living space having one or more spaces or areas. The network-connected devices can communicate with each other and with a user using the network connectivity. The network-connected devices can perform various actions (e.g., play music, turn on lights, etc.) based on a request from a user. Smart homes are becoming increasingly popular. Similarly, the number of network-connected devices is also proliferating as technology advances. For example, the costs of microprocessors, network equipment, environmental sensors, etc. have decreased dramatically over time. The proliferation and proliferation of such devices can result in an abundance, over-presence, or congestion of smart devices within a particular environment.

[0012] Smart homes can have various problems associated with the proliferation of network-connected devices. One problem is that the visibility and setup of network-connected devices may be suboptimal. For example, network-connected devices may rely on different protocol setups, various network standards, and redundant application interactions. Suboptimal visibility and setup of network-connected devices can lead to user frustration. For example, a user may want to adjust the lighting for watching a program in a home study by adjusting the lighting on a smart outlet, which is a network-connected device that cooperates with a network-connected smart television device. To do this, the user may need to navigate multiple applications, use tedious menus, or perform lengthy, repetitive voice interactions, or a combination of these.

[0013] Another issue is the unintended use of network-connected devices. As a first example, a first user may attempt to send a clip of streaming content to a streaming device, but a third party, either accidentally or intentionally, sends the streaming content to an unintended device. This unintended device may be a neighbor's device in an adjacent room / floor / office or apartment. As a second example, a first user may gain access to a second user's network-connected device (such as a smart appliance or smart speaker) through the network-connected device's proximity or credential broadcast. As a third example, a network-connected device, such as a voice assistant or smart router, may unintentionally establish visibility between devices, automatically bridging two networks or network-connected devices.

[0014] A bubble-based smart home (BBSH) can overcome problems associated with network-connected devices in such environments. A BBSH may operate by identifying and detecting relationships between devices. A BBSH may operate by performing machine learning or other forms of artificial intelligence on the behavior of various smart home environments. A BBSH may operate by identifying one or more actions performed in a smart home environment (hereinafter, environment). A BBSH may identify actions that are unrelated to one another. For example, a BBSH may be configured to scan for actions performed on devices present on a first network (e.g., a first wireless network). The BBSH may further be configured to scan for actions performed on devices present on a second network. As another example, a BBSH may be configured to scan for actions performed on a first device. The BBSH may also be configured to identify unrelated second devices and actions directed to the unrelated second devices.

[0015] The BBSH may be configured to detect relationships. The BBSH may be configured to detect relationships based on actions taken by a user. This may be done, for example, by associating the order in which operations are performed by the user, by associating the user's location within the environment, or by associating a time component (such as seconds or minutes between a first action and a second action).

[0016] The BBSH may be configured to detect relationships based on one or more devices in an environment. For example, the BBSH may be configured to operate by associating a first network-connected device with a second network-connected device. For example, the BBSH may be configured to operate by associating actions of a first device with actions of a second device based on their relative distances to each other, to another device, or to a user. The BBSH may be configured to operate by associating actions directed to two separate devices based on actions directed to a third device. For example, the BBSH may operate by receiving commands via a computer interface or via voice commands from a voice assistant. By receiving a command from a third device directed to an unrelated device, the BBSH may detect that a relationship exists.

[0017] BBSH may detect relationships by modeling or interpreting the underlying interface of a network-connected device. Specifically, a network-connected device may operate in two different operating modes, including pairing mode and listening mode. In listening mode, the network-connected device may be configured to perform an action requested by a user or from another device on the same network. A network-connected device may operate in pairing mode before operating in listening mode. In other words, a prerequisite for operating in listening mode may be pairing in pairing mode. In pairing mode, the network-connected device may actively search for a network or actively attempt to join a network under the control of a user or the owner of the environment. For example, a smart speaker may operate in pairing mode by attempting to join or by being open to receiving commands from a user through the interface of a third-party device. The third-party device may be a computer graphical user interface (GUI) or other third-party device configured to receive commands.

[0018] The BBSH may automatically pair with network-connected devices that are in listening mode (e.g., the BBSH may allow a new network-connected device to request to join the BBSH's network). Specifically, the BBSH may detect that the new network-connected device has attempted to join the network one or more times, or may detect a scan or ping by the new network-connected device. The BBSH may respond and directly pair with the new network-connected device. By automatically pairing, the BBSH may prevent unauthorized third parties (e.g., neighbors) from pairing with the network-connected device. After the BBSH pairs the network-connected device to the network, the network-connected device may be prevented from receiving transmissions from devices on the network that are not the BBSH. In some embodiments, the BBSH may present this newly paired network-connected device to the user in unauthorized mode. Specifically, the BBSH may present this newly paired device but prompt the user to confirm its ownership, membership, authorization, or other relevant relationship with the network of network-connected devices (e.g., smart home). The prompt in the unauthorized mode may be for the smart home user to confirm their involvement with the smart home. For example, the BBSH may present a notification on the user's smartphone (a first existing network-connected device) that a newly paired smart speaker (a new network-connected device) is now available in the smart home. Upon receiving a request through a first interface on the smartphone, the BBSH may prompt the user to confirm permission through a second existing network-connected device (such as a desktop computer or voice assistant).In some embodiments, only after confirmation or authentication by the user, the BBSH may be configured to display, present, or otherwise allow this newly added network-connected device to the rest of the network-connected devices in the smart home.

[0019] The BBSH may operate based on edge devices within the environment. The edge devices may be computing devices that operate locally to the environment, such as desktop PCs, smartphones, or other computers with memory, input / output, and processing power. For example, one or more of the network-connected devices may have an appropriate amount of processing power, such as a computer with sufficient RAM, CPU, and input / output to perform the operations of the BBSH. In some embodiments, the BBSH may perform machine learning on data from the environment using one or more of the following example techniques: Exemplary techniques include K-nearest neighbor (KNN), learning vector quantization (LVQ), self-organizing map (SOM), logistic regression, ordinary least squares regression (OLSR), linear regression, stepwise regression, multivariate adaptive regression spline (MARS), ridge regression, lasso regression (LASSO), elastic net, least-angle regression (LARS), probabilistic classifier, naive Bayes classifier, binary classifier, linear classifier, hierarchical classifier, canonical correlation analysis (CCA), factor analysis, independent component analysis (ICA), linear discriminant analysis (LDA), multidimensional scaling (MDS), non-negative matrix factorization (NMF), and others. metric factorization), partial least squares regression (PLSR)In some embodiments, the BBSH may perform machine learning using one or more of the following example techniques: Exemplary techniques include principal component analysis (PCA), principal component regression (PCR), Sammon mapping, t-distributed stochastic neighbor embedding (t-SNE), bootstrap aggregating, harmonic mean, gradient boosted decision trees (GBRT), gradient boosting machines (GBM), inductive bias algorithms, Q-learning, state-action-reward-state-action (SARSA), temporal difference (TD) learning, apriori algorithms, equivalence class transformation (ECLAT) algorithms, Gaussian process regression, gene expression programming, group method of data handling (GMDH), inductive logic programming, example-based learning, logistic model trees, and information fuzzy networks (IFNs). networks, hidden Markov models, Gaussian naive Bayes, multinomial naive Bayes, averaged one-dependence estimators (AODE), Bayesian networks (BN), classification and regression trees (CART), chi-squared automatic interaction detection (CHAID), and region-based convolutional neural networks (RCNN).networks), expectation maximization algorithm, feedforward neural networks, logic learning machines, self-organizing maps, single linkage clustering, fuzzy clustering, hierarchical clustering, Boltzmann machines, convolutional neural networks, recurrent neural networks, hierarchical temporal memory (HTM), or other machine learning methods, or a combination thereof.

[0020] The BBSH may generate networked device bubbles through modeling. The networked device bubbles may be paired with user-facing identifying information related to multiple actions in the smart home. Specifically, the networked device bubbles may be created based on the observation of trigger actions and user behavior, and based on the context or commonality of relationships between entities or devices. For example, the BBSH may analyze user voice commands, device usage patterns, and camera footage through machine learning to generate networked device bubbles for the home.

[0021] The BBSH may continuously receive information about network-connected devices and the user's environment. For example, the BBSH may receive feeds from the user's wearable or mobile devices, etc., thus enabling recognition of the user's activities. The wearable and mobile devices may record physical activity (e.g., exercise and sleep), movement patterns (e.g., entering and exiting various rooms), and frequency of performing activities (e.g., how often the user enters the kitchen in the morning on a weekday). In some embodiments, the BBSH may model or detect the existence of relationships by correlating the user's actions with actions directed at various network-connected devices in the environment. For example, the BBSH may be configured to monitor and analyze the timing of performing actions requesting streaming media, turning lights on or off, etc. At the same time, the BBSH may be configured to monitor the user's activities based on movements from sensors in the user's portable, mobile, and wearable devices. Significant statistical correlations between activity (e.g., accelerometer and gyroscope data) and subsequent actions taken by (or commands sent to) specific network-connected devices may be identified over a period of several weeks in one environment.

[0022] The BBSH may utilize cameras placed within the environment to determine what activities are being performed by the user. For example, the BBSH may utilize an image processing device (not shown). The image processing device may be a collection of hardware and software, such as an application specific integrated circuit.

[0023] The image processing device may be configured to perform various image analysis techniques. The image analysis techniques may be machine learning-based techniques, deep learning-based techniques, or both. These techniques may include, but are not limited to, region-based convolutional neural networks (R-CNN), you only look once (YOLO), edge matching, clustering, grayscale matching, gradient matching, invariant models, geometric hashing, scale-invariant feature transform (SIFT), speeded up robust feature (SURF), histogram of oriented gradients (HOG) features, and single-shot multibox detector (SSD). In some embodiments, the image processing device may be configured to assist in face identification (e.g., by analyzing facial images using a model built based on training data).

[0024] In some embodiments, objects may be identified by an object detection algorithm (such as R-CNN, YOLO, SSD, SIFT, Hog features, or other machine learning or deep learning object detection algorithms, or both). The output of the object detection algorithm may include one or more identities of one or more respective objects with corresponding match certainties. For example, a scene in a smart home environment may be analyzed. User movements within the environment may be identified by an associated object detection algorithm.

[0025] In some embodiments, the features of an object may be determined by a supervised machine learning model constructed using training data. For example, an image may be input to the supervised machine learning model, and various classifications detected in the image may be output by the model. For example, characteristics such as the object's material (e.g., fabric, metal, plastic, etc.), shape, size, color, etc. may be output by the supervised machine learning model. Furthermore, the object's identification (e.g., plant, human face, dog, etc.) may be output as the classification determined by the supervised machine learning model. For example, if a user photographs a smart speaker, the supervised machine learning algorithm may be configured to output the identity of the object and the environment (e.g., the smart speaker in the bedroom) as well as various characteristics of the vehicle (e.g., model, color, etc.).

[0026] In some embodiments, the properties of an object may be determined using photogrammetry techniques. For example, the shape and dimensions of an object may be approximated using photogrammetry techniques. As an example, if a user provides an image of a basket, the diameter, depth, thickness, etc. of the basket may be approximated using photogrammetry techniques. In some embodiments, the properties of an object may be identified by referencing an ontology. For example, when an object is identified (e.g., using R-CNN), the identity of the object may be referenced in an ontology to determine the object's corresponding attributes. The ontology may indicate attributes such as the object's color, size, shape, use, etc.

[0027] The characteristics may include the shape of the object, the dimensions of the object (e.g., height, length, width), the number of objects, the color of the object, or other attributes of the object, or a combination thereof. In some embodiments, the output may generate a list including the object's identity and / or characteristics. In some embodiments, the output may include an indication that the object's identity or characteristics are unknown. This indication may include a request for additional input data that can be analyzed so that the object's identity and / or characteristics can be confirmed. In some embodiments, the various objects, object attributes, and relationships between objects (e.g., hierarchical and direct relationships) may be represented within a knowledge graph (KG) structure. Objects may be matched to other objects based on shared characteristics, patterns, or behaviors to enable the BBSH to identify or understand various user actions and activities. For example, by analyzing past movement patterns, hand movement patterns, and the amount of time spent in different areas, the BBSH can identify what type of activity the user is manually performing (e.g., loading clothes in the washing machine, cooking, changing thermostat settings, etc.).

[0028] A network-connected device bubble can be thought of as a group of automated scenarios, pairings, or other actions performed by heterogeneous network-connected devices. A network-connected device bubble can be formed from information obtained by a BBSH. The BBSH can leverage the network-connected device bubble to perform or initiate further actions, such as performing actions on one or more devices in the network-connected device bubble. Performing an action may complete an operation the user may have intended to perform, thereby reducing confusion or overhead when operating or interacting with components of the smart home, such as a graphical user interface or a voice-based assistant interface.

[0029] FIG. 1 illustrates representative major components of an example computer system 100 (or computer) usable in accordance with some embodiments of the present disclosure. It is understood that the individual components may vary in complexity, number, type, or configuration, or combinations thereof. The specific example in this disclosure is merely illustrative and is not necessarily limited to such variations. Computer system 100 may include a processor 110, memory 120, an input / output interface (hereinafter, I / O or I / O interface) 130, and a main bus 140. Main bus 140 may provide a communication path for other components of computer system 100. In some embodiments, main bus 140 may be connected to other components, such as a dedicated digital signal processor (not shown).

[0030] The processor 110 of the computer system 100 may be configured with one or more cores 112A, 112B, 112C, and 112D (collectively referred to as cores 112). The processor 110 may further include one or more memory buffers or caches (not shown) that temporarily store instructions and data for the cores 112. The cores 112 may execute instructions on input provided from the cache or memory 120 and output results to the cache or memory. The cores 112 may be configured with one or more circuits configured to perform one or more methods according to embodiments of the present disclosure. In some embodiments, the computer system 100 may include multiple processors 110. In some embodiments, the computer system 100 may be a single processor 110 with a single core 112.

[0031] The memory 120 of the computer system 100 may include a memory controller 122. In some embodiments, the memory 120 may include a random-access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) for storing data and programs. In some embodiments, the memory may be in the form of a module (e.g., a dual in-line memory module). The memory controller 122 may communicate with the processor 110 to facilitate the storage and retrieval of information to and from the memory 120. The memory controller 122 may communicate with the I / O interface 130 to facilitate the storage and retrieval of input or output to and from the memory 120.

[0032] I / O interface 130 may include I / O bus 150, terminal interface 152, storage interface 154, I / O device interface 156, and network interface 158. I / O interface 130 may connect main bus 140 to I / O bus 150. I / O interface 130 may route instructions and data from processor 110 and memory 120 to various interfaces of I / O bus 150. I / O interface 130 may also route instructions and data from various interfaces of I / O bus 150 to processor 110 and memory 120. The various interfaces may include terminal interface 152, storage interface 154, I / O device interface 156, and network interface 158. In some embodiments, the various interfaces may include a subset of the interfaces described above (e.g., an embedded computer system for industrial applications may not include terminal interface 152 and storage interface 154).

[0033] Logical modules throughout computer system 100 (including, but not limited to, memory 120, processor 110, and I / O interface 130) may communicate faults and changes to one or more components to a hypervisor or operating system (not shown). The hypervisor or operating system may allocate the various resources available to computer system 100 and track the location of data in memory 120 and the location of processes assigned to the various cores 112. In embodiments that combine or rearrange elements, aspects and capabilities of the logical modules may be combined or rearranged. These variations will be apparent to those skilled in the art.

[0034] FIG. 2 illustrates a model 200 representative of one or more artificial neural networks capable of detecting relationships and generating network-connected device bubbles in accordance with an embodiment of the present disclosure. The model neural network (hereafter referred to as neural network) 200 is comprised of multiple layers. The neural network 200 includes an input layer 210, a hidden section 220, and an output layer 250. Note that while the model 200 illustrates a feedforward neural network, other neural network layouts, such as a recurrent neural network configuration (not shown), are also contemplated. In some embodiments, the neural network 200 may be a design-and-run neural network, where the layout exhibited by the model can be created by a computer programmer. In some embodiments, the neural network 200 may be a design-by-run neural network, where the layout exhibited by the model can be generated by a process of inputting data and analyzing the data according to one or more defined heuristics. The neural network 200 may operate in a forward propagation fashion by receiving input and outputting the results of that input. Neural network 200 can adjust the values of various components of the neural network through backpropagation.

[0035] The input layer 210 includes a set of input neurons 212-1, 212-2, ... 212-n (collectively referred to as input neurons 212) and a set of input connections 214-1, 214-2, 214-3, 214-4, etc. (collectively referred to as input connections 214). The input layer 210 represents input from data that the neural network is to analyze (e.g., a set of characteristics for a particular network-connected device in an environment, one or more heuristics or preferences of a user). Each input neuron 212 can represent a subset of the input data. For example, the neural network 200 is provided with a device identifier as input, where the device identifier is represented by a set of values. In this example, input neuron 212-1 can be a first identifier for a first network-connected device in an environment, input neuron 212-2 can be a relative latency value for the first network-connected device, etc. The number of input neurons 212 can correspond to the size of the input. For example, if the neural network is designed to analyze an image that is 256 pixels by 256 pixels, the layout of the neural network 200 may include a set of 65,536 input neurons. The number of input neurons 212 may correspond to the type of input. For example, if the input is a 256 pixel by 256 pixel color image, the layout of the neural network 200 may include a set of 196,608 input neurons (65,536 input neurons for the red value of each pixel, 65,536 input neurons for the green value of each pixel, and 65,536 input neurons for the blue value of each pixel). The type of input neurons 212 may correspond to the type of input. As a first example, the neural network 200 may be designed to analyze a black-and-white image, and each input neuron may be a decimal value between 0.00001 and 1 that represents the grayscale shade of the pixel (where 0.00001 represents a pixel that is completely white and 1 represents a pixel that is completely black).As a second example, neural network 200 may be designed to analyze a color image, and each input neuron may be a three-dimensional vector representing the color value of a given pixel in the input image (where the first component of the vector is an integer value of red between 0 and 255, the second component of the vector is an integer value of green between 0 and 255, and the third component of the vector is an integer value of red between 0 and 255).

[0036] Input connections 214 represent the output of input neurons 212 to hidden section 220. Each input connection 214 varies depending on the value of each input neuron 212 and is based on multiple weights (not shown). For example, first input connection 214-1 has a value provided to hidden section 220 based on input neuron 212-1 and the first weight. Continuing with the example, second input connection 214-2 has a value provided to hidden section 220 based on input neuron 212-1 and the second weight. Continuing with the example, third input connection 214-3 is based on input neuron 212-2 and the third weight. In other words, input connections 214-1 and 214-2 may share the same output component of input neuron 212-1, input connections 214-3 and 214-4 may share the same output component of input neuron 212-2, and all four input connections 214-1, 214-2, 214-3, 214-4 may have output components with four different weights. Neural network 200 may have different weights for each connection 214, although in some embodiments, similar weights are also possible. In some embodiments, the values of each of the input neurons 212 and connections 214 may necessarily be stored in memory.

[0037] Hidden section 220 includes one or more layers that receive inputs and generate outputs. Hidden section 220 includes a first hidden layer of computational neurons 222-1, 222-2, 222-3, 222-4, ... 222-n (collectively referred to as computational neurons 222), a second hidden layer of computational neurons 226-1, 226-2, 226-3, 226-4, 226-5, ... 226-n (collectively referred to as computational neurons 226), and a set of hidden connections 224 connecting the first and second hidden layers. Note that model 200 represents only one of many possible neural networks that can model the environment of a network-connected device as part of a BBSH according to some embodiments of the present disclosure. Thus, hidden section 220 may be configured with more or fewer hidden layers (e.g., one hidden layer, seven hidden layers, twelve hidden layers, etc.), with two hidden layers shown as an example.

[0038] The first hidden layer 222 includes computational neurons 222-1, 222-2, 222-3, 222-4, ... 222-n. Each computational neuron in the first hidden layer 222 can receive one or more of the connections 214 as inputs. For example, computational neuron 222-1 receives input connection 214-1 and input connection 214-2. Each computational neuron in the first hidden layer 222 also provides an output, represented by the dotted hidden connections 224 leaving the first hidden layer 222. Each computational neuron 222 performs an activation function during forward propagation. In some embodiments, the activation function may be a process (e.g., a perceptron) that receives multiple binary inputs and computes a single binary output. In some embodiments, the activation function may be a process that receives multiple non-binary inputs (e.g., a number between 0 and 1, 0.671, etc.) and computes a single non-binary output (e.g., a number between 0 and 1, a number between -0.5 and 0.5, etc.). Various functions may be implemented to calculate activation functions (e.g., sigmoid neurons or other logistic functions, hyperbolic tangent neurons, softplus functions, softmax functions, normalized linear units, etc.). In some embodiments, each computational neuron 222 also includes a bias (not shown). The bias may be used to determine the likelihood or evaluation of a given activation function. In some embodiments, each value of the bias for each computational neuron necessarily needs to be stored in memory.

[0039] An example of the model 200 may include using a sigmoid neuron for the activation function of the computational neuron 222-1. A mathematical formula (Equation 1 below) may represent the activation function of the computational neuron 222-1 as f(neuron). The logic of the computational neuron 222-1 may be the sum of each input connection (i.e., input connection 214-1 and input connection 214-3) provided to the computational neuron 222-1, represented as j in Equation 1. For each j, a weight w is multiplied by the value x of the given connected input neuron 212. The bias of the computational neuron 222-1 is represented as b. When each input connection j is summed, the bias b is subtracted. The operation of this example is completed as follows: The more positive the sum and bias in the activation f(neuron), the closer the output of the computational neuron 222-1 approaches 1. The more negative the sum and bias in the activation f(neuron), the closer the output of the computational neuron 222-1 approaches 0. And given the sum and bias in the activation f(neuron) between more positive and more negative numbers, the output will change slightly as the weights and biases change slightly. TIFF0007725158000001.tif13165

[0040] The second hidden layer 226 includes computational neurons 226-1, 226-2, 226-3, 226-4, 226-5, ... 226-n. In some embodiments, the computational neurons of the second hidden layer 226 can operate similarly to the computational neurons of the first hidden layer 222. For example, the computational neurons 226-1 through 226-n can each operate with a similar activation function as the computational neurons 222-1 through 222-n. In some embodiments, the computational neurons of the second hidden layer 226 can operate differently from the computational neurons of the first hidden layer 222. For example, the computational neurons 226-1 through 226-n can have a first activation function, and the computational neurons 222-1 through 222-n can have a second activation function.

[0041] Similarly, the connections to, from, and between various layers of the hidden section 220 may vary. For example, the input connections 214 may be fully connected to the first hidden layer 222, and the hidden connections 224 may be fully connected from the first hidden layer to the second hidden layer 226. In an embodiment, fully connected may mean that each neuron in a given layer may be connected to every neuron in the previous layer. In an embodiment, fully connected may mean that each neuron in a given layer may function entirely independently and not share any connections. In a second example, the input connections 214 may not be fully connected to the first hidden layer 222, and the hidden connections 224 may not be fully connected from the first hidden layer to the second hidden layer 226.

[0042] Similarly, parameters to, from, and between various layers of hidden section 220 may also vary. In some embodiments, the parameters may include weights and biases. In some embodiments, there may be more or fewer parameters than weights and biases. For example, model 200 may be in the form of a convolutional network. A convolutional neural network may include a series of heterogeneous layers (e.g., input layer 210, convolutional layer 222, pooling layer 226, and output layer 250). In such a network, the input layer may hold raw pixel data of an image in a three-dimensional volume of width, height, and color. The convolutional layers of such a network output connections local only to the input layer and may identify features in a small portion of an image (e.g., the eyebrows on the face of a first subject in a photo of four subjects, the front fender of a vehicle in a photo of a truck, etc.). In this example, the convolutional layers may include weights and biases as well as additional parameters (e.g., depth, stride, and padding). The pooling layer of such a network may receive the output of the convolutional layer as input, but may perform a fixed functional operation (e.g., an operation that does not consider any weights and biases). Also, in this example, the pooling layer may not include any convolutional parameters, and may not include any weights and biases (e.g., perform a downsampling operation).

[0043] Output layer 250 includes a set of output neurons 250-1, 250-2, 250-3, ... 250-n (collectively referred to as output neurons 450). Output layer 250 holds the results of neural network 200's analysis. In some embodiments, output layer 250 may be a classification layer used to identify features of inputs to neural network 200. For example, neural network 200 may be a classification network trained to identify Arabic numerals. In such an example, neural network 200 may include 10 output neurons 250 corresponding to the Arabic numerals identified by the network (e.g., output neuron 250-2 having a higher activation value than output neuron 250 may indicate that the neural network has determined that the image contains the numeral "1"). In some embodiments, output layer 250 may be a real-valued target (e.g., attempting to predict an outcome when the input is a set of previous outcomes), and there may be only a single output neuron (not shown). The output layer 250 is fed by output connections 252, which provide activations from the hidden section 220. In some embodiments, the output connections 252 may include weights and the output neurons 250 may include biases.

[0044] Training a neural network represented by model 200 may include performing backpropagation. Backpropagation is distinct from forward propagation. Forward propagation may include providing data to input neuron 212, performing calculations for connections 214, 224, 252, and performing calculations for computational neurons 222 and 226. Forward propagation may also be dependent on the layout of a given neural network (e.g., regression, number of layers, number of neurons in one or more layers, whether a layer is fully connected to other layers, etc.). Backpropagation may determine errors in parameters (e.g., weights and biases) within neural network 200 by starting from output neuron 250 and propagating the error backward through each of the various connections 252, 224, 214 and layers 226, 222.

[0045] Backpropagation involves running one or more algorithms based on one or more training data to reduce the difference between what a given neural network decides given an input and what the given neural network should decide given that input. The difference between a network decision and the correct decision is sometimes called the objective function (or cost function). When a given neural network is first created, fed data, and computed by forward propagation, the result or decision may be an incorrect decision.

[0046] For example, neural network 200 may be a classification network that is provided with a 128-pixel by 250-pixel image input containing captured image data from a camera or other visual sensor and may determine that a first network-connected device is most likely to be a relevant network-connected device that should be added to an existing network bubble, next most likely to be an unrelated network-connected device that should not be added, and next most likely to be a user moving through the environment. Continuing with the example, backpropagation may be performed to change the values of the weights on connections 214, 224, and 252, and to change the values of the biases on the first layer of computational neurons 222, the second layer of computational neurons 226, and output neuron 250. Continuing with the example, performing backpropagation may subsequently output a more accurate classification of the same 128-pixel by 250-pixel image input containing a depiction of the environment (e.g., more granularly ranking unrelated network-connected devices, relevant network-connected devices, and users from most likely to least likely, etc.).

[0047] Equation 2 provides an example of an objective function in the form of a quadratic cost function (e.g., mean squared error). Other functions may be chosen, and mean squared error is chosen as an example. In Equation 2, all weights w and biases b in the example network. The example network is provided with a given number of training inputs n in a subset (or all) of the training data having input values x. The example network may produce an output a from x, but it must also produce a desired output y(x) from x. Backpropagating or training the example network requires reducing or minimizing the objective function "O(w, b)" through changes to the set of weights and biases. Successful training of the example network should reduce the difference between the example network's answer a and the correct answer y(x), not only for input values x, but also for given new input values (e.g., from additional training or validation data). TIFF0007725158000002.tif11165

[0048] Many options are available for the backpropagation algorithm, both in terms of the objective function (e.g., mean squared error, cross-entropy cost function, etc.) and the reduction of the objective function (e.g., gradient descent, batch stochastic gradient descent, Hessian optimization, momentum-based gradient descent, etc.). Backpropagation may involve using a gradient descent algorithm (e.g., computing partial derivatives of the objective function with respect to the weights and biases of all training data). Backpropagation may also involve determining a stochastic gradient descent algorithm (e.g., computing partial derivatives of the training inputs on a subset or batch of training data). Additional parameters may be involved in various backpropagation algorithms (e.g., the learning rate for gradient descent). Large changes to the weights and biases through backpropagation can lead to erroneous training (e.g., overfitting to the training data, reducing to a local minimum, or excessive reduction beyond the global minimum). Therefore, modifying the objective function with more parameters can prevent erroneous training (e.g., utilizing an objective function that incorporates regularization to prevent overfitting). This also allows for fewer changes to neural network 200 at any given iteration, and as a result of the need for fewer iterations at any given time, the backpropagation algorithm may need to repeat many times to achieve accurate learning.

[0049] For example, neural network 200 may have untrained weights and biases, and backpropagation may involve stochastic gradient descent to train the network on a subset of training inputs (e.g., a batch of 10 training inputs from the entire set of training inputs). Continuing the example, neural network 200 may continue training on a second subset of training inputs (e.g., a second batch of 10 training inputs from the entire set excluding the first batch), which may be repeated until all training inputs have been used in the gradient descent calculation (e.g., one epoch of training data). In other words, if there are 10,000 total training images and a batch size of 100 training inputs is used in one training iteration, 1,000 iterations will be required to complete one epoch of training data. Many epochs may be performed to continue training the neural network. There may be many factors that determine the selection of additional parameters (e.g., large batch sizes may result in inadequate training, small batch sizes may result in too many training iterations, large batch sizes may not fit into memory, small batch sizes may not efficiently utilize discrete GPU hardware, too few training epochs may not produce a complete trained network, too many training epochs may overfit the trained network, etc.). In another example, long short-term memory (LSTM) machine learning techniques / networks may be utilized in smart homes as part of BBSH, applying historical data of users, devices, device classifications, locations, times, and other variable inputs to the trained / trained application / implementation to suggest the creation of new network-connected device bubbles and / or actions as new data is fed into the model. The training sequence may utilize optimization algorithms such as gradient descent, combined with time-consuming backpropagation to calculate the gradients required for the optimization process and / or any other neural networks, and training completed through production tactics.

[0050] 3 is a diagram illustrating an example of implementing BBSH in a real-world environment 300, according to some embodiments of the present disclosure. The environment 300 may be a smart home, such as a residential living space. The residential living space is shown by way of example. In some embodiments, the environment 300 may be an office environment or other area.

[0051] Environment 300 may include one or more rooms, including first room 310, second room 320, third room 330, and fourth room 340. Environment 300 may also be associated with a user 350 who moves and operates within the environment. Outside environment 300, a neighboring space 360 may exist. Neighboring space 360 may be unrelated to environment 300 or may not otherwise form part of environment 300. For example, environment 300 may be user 350's apartment, and neighboring space 360 may be a space in an adjacent apartment or common area. Neighboring space 360 may include unrelated network-connected devices 362. BBSH may be configured to remove, conceal, or otherwise prevent user 350 from accessing unrelated network-connected devices 362. For example, BBSH may remove unrelated network-connected devices 362 from the user interfaces of any devices within environment 300 so that unrelated network-connected devices 362 are unreachable.

[0052] The first room 310 may be an office, living room, study, or other related space for work, leisure, or both. The first room 310 may include multiple network-connected devices, including a smart TV 312, a laptop computer 314, and an environmental sensor 316. The environmental sensor 316 may be configured to receive signals from the environment 300. For example, the environmental sensor 316 may be a camera or microphone configured to receive and detect a user within the environment based on a video or audio signal, respectively. The second room 320 may be a bedroom or other space accessible to a user 350. The second room 320 may include an access point 322. The access point 322 may be configured to receive network communications from other network-connected devices within the environment 300.

[0053] The third room 330 may be a kitchen, dining room, front room, or other relevant space for the user 350. The third room 330 may include multiple network-connected devices, including a voice-based assistant (hereinafter, "assistant") 332 and a smart light 334. The fourth room 340 may be a bathroom, utility room, or other relevant room. The fourth room 340 may include multiple network-connected devices, including a smart speaker 342 and a smart washing machine 344. As shown in FIG. 3 , the user 350 may be standing in the third room 330 and have access to the second room 320 and the fourth room 340. The user 350 may also be visible to the sensor 316 from the first room 310.

[0054] The BBSH may be configured to operate within the environment 300 via one or more edge devices capable of executing processes to identify actions, detect relationships between network-connected devices, and generate network-connected device bubbles. For example, the smart television 312, the laptop computer 314, the access point 322, or the assistant 332, or a combination thereof, may be edge devices capable of executing within the BBSH. Each of the edge devices may operate as a computer, such as the computer 100 of FIG. 1. Each of the edge devices may operate to execute a neural network capable of modeling actions and other network-connected devices, such as the neural network 200 configured to detect relationships.

[0055] Each edge device may perform BBSH by monitoring environment 300 and detecting relationships. Edge devices may detect relationships by modeling usage patterns and recording a history of actions and interactions with users (such as user 350). For example, the first network-connected device 332 (e.g., a voice-based assistant) may be configured to identify various actions related to other network-connected devices in environment 300. Assistant 332 may identify user 350 requesting music to play, and in response, the assistant may begin playing music. Assistant 332 may identify and record an interaction with user 350 to play music. At a later time, Assistant 332 may receive a request from user 350 to dim light 334. Assistant 332 may respond by instructing light 334 to be dimmed or reduced in brightness. Assistant 332 may further identify and record an interaction with user 350 to dim light 334.

[0056] Assistant 332 may record the user's actions as part of BBSH execution. Assistant 332 may identify patterns, behaviors, or relationships based on the recorded and identified interactions in environment 300. BBSH operation may include Assistant 332 performing modeling (such as through neural network 200) to detect relationships between playing music and changing lights in third room 330. Based on BBSH operation, Assistant 332 may determine a high probability (e.g., a probability exceeding a predetermined bubble threshold) that two actions can combine to form a bubble. For example, Assistant 332 may model user 350's behavior over time and determine that when user 350 requests Assistant 332 to play music, there is a 78% probability that user 350 will also request that lights 334 be dimmed. The predetermined bubble threshold may be a probability greater than 75%.

[0057] As a result, the assistant 332 may determine that a first network-connected device bubble 370 should be generated. The first network-connected device bubble 370 may include a model of the particular action, the network-connected device (e.g., the assistant 332 and the light 334), and other relevant contextual information. The relevant contextual information may be based on the user's 350 patterns or usage. For example, the relevant contextual information for the first network-connected device bubble 370 may indicate that an unrelated action should be associated only during a particular time of day (e.g., between 10:00 AM and 11:00 AM). In another example, the relevant contextual information may be that the first network-connected device bubble 370 may only be present when the user 350 is in a particular portion, room, section, or area within the environment 300 (e.g., the third room 330). Based on the first network-connected device bubble 370, the BBSH may initiate one or more of the actions that comprise the first network-connected device bubble 370. For example, the user 350 may request that the light 334 be dimmed. In response to receiving the request, the assistant 332 may identify that this action is part of the first network-connected device bubble 370 and the BBSH may initiate an action to play music through the assistant 332.

[0058] The network-connected device bubbles may operate in multiple rooms, sections, or areas of the environment 300. For example, a user may request, through the assistant 332, that a music video be played on the smart television 312 and the laptop computer 314 and that the lights 334 be strobed. Based on modeling these actions of unrelated devices, the BBSH executing on the assistant 332 may detect a relationship between the network-connected devices in the first room 310 and the third room 330. The BBSH executing on the assistant 332 may generate a second network-connected device bubble 380. The BBSH may delegate or reassign the performance of various operations to one or more other devices. For example, the performance of one or more operations of the BBSH may first be performed to identify, detect, and generate the second network-connected device bubble 380. The assistant 332 may determine that one or more operations should be reassigned to another network-connected device in the environment 300. This determination may include monitoring or communicating with other network-connected devices in the environment. For example, the assistant 332 may communicate with the laptop computer 314. The assistant 332 may request or monitor the computing resources of the laptop computer 314. The assistant 332 may determine that the overall memory utilization of the laptop computer 314 is below a predetermined operational threshold (e.g., less than 20% memory utilization). The assistant 332 may then assign a BBSH action to the laptop computer 314 if it determines that the current operation of the laptop computer 314 is below the predetermined operational threshold. In response, the laptop computer 314 may monitor and attempt to identify possible actions associated with the second network-connected device bubble 380.

[0059] The BBSH may detect relationships based on one or more usage patterns of a user. For example, relationship detection may be based on modeling the frequency or timing at which certain actions are requested from one or more network-connected devices in the environment 300. The BBSH may detect relationships between devices based on location or proximity. For example, the BBSH may receive information or signals from a portable device of the user 350, including the user's smartphone 352 or wearable computing device 354, or both. The BBSH may be configured to determine that the user is in proximity to the smart speaker 342 based on connection confirmation, echolocation, GPS, accelerometer data, time-of-flight, or other related location detection or movement algorithms of one or more portable devices of the user 350. The BBSH may generate a third network-connected device bubble 390 based on the user's location.

[0060] The BBSH may automatically add other devices or actions to the third network-connected device bubble 390. For example, the BBSH may determine the relative proximity of the smart washing machine 344 to the smart speaker 342. The BBSH may determine that there is a relationship between the washing machine 344 and the smart speaker 332 without receiving any indication of the relationship from the user, such as based solely on the locations of the washing machine 344 and the smart speaker 342. Based on this relationship determination, the BBSH may generate the third network-connected device bubble 390. As part of generating the third network-connected device bubble 390, the BBSH may be configured to automatically associate a type of operation of the smart speaker 342 (e.g., playing audio for the user) with an attribute of the washing machine 344 (e.g., a device tagged as making undesirable noise). The BBSH may define as part of the third network-connected device bubble 390 that when it receives a request from the user 350 to play a podcast on the smart speaker 342, it should automatically (e.g., without user input) initiate an action to pause the operation of the washing machine 344.

[0061] The BBSH may be configured to receive feedback from the user. For example, based on execution by the assistant 332, the BBSH may determine that the user has requested the smart speaker 342 to play music. The BBSH may identify the request and determine that the music played by the smart speaker 342 in response to the user's request is associated with the third network-connected device bubble 390. Upon determining an action associated with the third network-connected device bubble 390, the BBSH may instruct the smart washing machine 344 to stop operating. The user 350 may instruct the BBSH (e.g., by speaking a voice command into the smartphone 352) to resume operation of the smart washing machine 344. In response, the BBSH may adjust, update, or otherwise modify the third network-connected device bubble 390. The update may be to not stop the washing machine 344 at the time the user instructed the washing machine to resume operation. The update may be to not stop the washing machine 344 when music is played on the smart speaker 342, but to keep the washing machine 344 stopped when a podcast is played on the smart speaker 342.

[0062] FIG. 4A illustrates an example user interface 400-1 operated by a BBSH according to some embodiments of the present disclosure. User interface 400-1 may be a graphical user interface provided to a user at a first time. User interface 400-1 may include a desktop 410-1, a navigation bar 420-1, and a pop-up menu 430-1. Pop-up menu 430-1 may be configured to display multiple network-connected devices with which the user can interact. Specifically, pop-up menu 430-1 may include at least a first smart speaker 440-1, a second smart speaker 450-1, a third smart speaker 460-1, and a fourth smart speaker 470-1. Pop-up menu 430-1 may be provided by a computer system, such as computer 100, for user interaction in a smart home environment. Pop-up menu 430-1 may receive user actions, such as the user interacting with a first smart speaker 440-1 and then repeating the interaction with a second smart speaker 450-1 shortly thereafter (e.g., within 5 minutes or 45 seconds).

[0063] The BBSH may execute on a computer system providing user interface 400-1 or an associated computer system. The BBSH may identify interactions with pop-up menu 430-1, detect relationships, and invoke actions across multiple network-connected devices. Over time (e.g., hours, days, weeks, months), the BBSH may detect relationships and generate one or more network-connected device bubbles based on the relationships. As a first example, the BBSH may determine that a user consistently plays music on both first smart speaker 440-1 and second smart speaker 450-1, but only between the hours of 12:15 PM and 3:35 PM. As a second example, the BBSH may determine that a user accesses fourth smart speaker 470-1 infrequently or not at all.

[0064] 4B illustrates an example user interface 400-2 after execution of BBSH, according to some embodiments of the present disclosure. User interface 400-2 may be a graphical user interface provided to a user at a second time (e.g., a time after the first time shown in FIG. 4A). User interface 400-2 may include a desktop 410-2, a navigation bar 420-2, and a pop-up menu 430-2. Pop-up menu 430-2 may be configured to display multiple network-connected devices after being modified by BBSH. Specifically, pop-up menu 430-2 may include at least a first network-connected device bubble 480-2 and a third smart speaker 460-1.

[0065] The BBSH may render the pop-up menu 430-2 by including a first network-connected device bubble 480-2 based on the relationships determined at a first time (prior to generating the user interface 400-2). The first network-connected device bubble 480-2 may be labeled based on one or more attributes of the associated network-connected devices. For example, the words "play" and "speaker" may be common to both the first smart speaker 440-1 and the second smart speaker 450-1. The first network-connected device bubble 480-2 may be labeled based on one or more attributes of an action or pattern recognized by the BBSH. For example, the word "afternoon" may be used in the first network-connected device bubble 480-2 of the pop-up menu 430-2 to take advantage of the time period between 12:00 PM and 3:35 PM that was common to the relationships detected by the BBSH.

[0066] The BBSH may determine that there is a negative relationship between one or more devices in the environment. For example, the BBSH executing on the computer providing the user interface 400-2 may generate a second network-connected device bubble that associates the fourth smart speaker 470-1 with the computer. The negative relationship in the second network-connected device bubble may indicate that the fourth smart speaker 470-1 will not be used with the computer. In response to the second network-connected device bubble, the BBSH may block, remove, or otherwise prevent the display of the fourth smart speaker 470-1 in the user interface 400-2 at a second time.

[0067] The changes and updates reflected in the pop-up menu 430-2 may provide one or more advantages to a user when performing operations. For example, a user may experience less confusion when interacting with the user interface 400-2. In another example, the usability and speed of operations when interacting with the user interface 400-2 may be improved. The changes and updates reflected in the pop-up menu 430-2 may achieve other practical benefits. For example, the first smart speaker 440-1, the second smart speaker 450-1, and the fourth smart speaker 470-1 may be removed from the memory or rendering pipeline of a computer that renders the user interface 400-2. As a result, the user interface 400-2 may be able to render more quickly or with reduced memory or processing performance compared to the user interface 400-1.

[0068] 5 illustrates an example method 500 performed by a smart home server (BBSH) according to some embodiments of the present disclosure. Method 500 may be performed on a computer system such as computer 100. Method 500 may be performed by an edge device such as voice-based assistant 232. Method 500 may be performed by a central server or a device delegated to perform one or more operations of a smart home, such as laptop computer 314.

[0069] After start 505, a first action may be identified at step 510. The first action may be associated with a user. The first action may be associated with a first network-connected device of the plurality of network-connected devices. For example, a user may wake up, walk across a room, and adjust stereo settings. In another example, a user may send a command to a smart speaker via a voice-based assistant or via a smartphone. The first action may be performed by the first network-connected device (e.g., play music). At step 520, a second action may be identified. The second action may be associated with the same user as the first action. The second action may be associated with a different user. For example, a second user may start a slideshow on a smart TV. The second action may be directed to a second network-connected device of the plurality of network-connected devices. The second network-connected device may be unrelated to the first network-connected device. For example, a first network-connected device may not share a name, task, group, defined function, category, manufacturer, or other relationship with a second network-connected device. Also, the first network-connected device and the second network-connected device may not have knowledge of each other. As a first example, the first network-connected device may be a smart speaker connected to a voice-based assistant via a first personal area network, and the second device may be a laptop computer connected to the voice-based assistant via a separate second wireless network.

[0070] At step 530, a relationship may be detected between a first action of a first network-connected device and a second action of a second network-connected device. The relationship may be detected by processing historical data, such as identifying a pattern between the first action and the second action. The relationship may be detected by performing one or more machine learning or artificial intelligence operations on the data. For example, a neural network 200 may be implemented to detect the relationship at step 530. If a relationship is detected (Y at step 540), a network-connected device bubble may be generated at step 550. The network-connected device bubble may be an entry, model, relationship, or other structure configured to clarify and explain the first action, the second action, the first device, the second device, and other devices or actions. After the network device bubble is generated at step 550 or after no relationship is detected (N at step 540), method 500 ends at step 595.

[0071] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0072] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, or the like, and suitable combinations thereof. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0073] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computer / processing device. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, or a wireless network, or a combination thereof). The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computer / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computer / processing device for storage.

[0074] The computer-readable program instructions for carrying out the operations of the present invention can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk and C++, and procedural programming languages such as the "C" programming language and similar programming languages. The computer-readable program instructions can execute entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuitry for carrying out aspects of the present invention.

[0075] Aspects of the present invention are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. Each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer-readable program instructions.

[0076] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of such computer or other programmable data processing apparatus, create means for performing the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner. The computer-readable storage medium having instructions stored thereon thereby constitutes an article of manufacture including instructions for performing aspects of the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams.

[0077] Computer-readable program instructions may also be loaded into a computer, other programmable device, or other device and a series of operational steps executed on the computer, other programmable device, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable device, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0078] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing specific logical functions. In some implementations, the functions shown in the blocks may be performed in an order different from that shown in the figures. For example, depending on the functionality involved, two blocks shown in succession may actually be accomplished as a single step, may be executed simultaneously or substantially simultaneously, may be executed in a partially or fully overlapping manner, or the blocks may even be executed in reverse order. Note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs specific functions or operations or executes a combination of dedicated hardware and computer instructions.

[0079] While various embodiments of the present disclosure have been described by way of example, they are not intended to be exhaustive or limited to these embodiments. As will be apparent to those skilled in the art, many modifications and variations are possible without departing from the scope of the described embodiments. The terms used herein are selected to better explain the principles, practical applications, or technical improvements to commercially recognized technologies of the embodiments, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0080] While various embodiments of the present disclosure have been described by way of example, they are not intended to be exhaustive or limited to these embodiments. As will be apparent to those skilled in the art, many modifications and variations are possible without departing from the scope of the described embodiments. The terms used herein are selected to better explain the principles, practical applications, or technical improvements to commercially recognized technologies of the embodiments, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. Pairing a plurality of network-connected devices in an environment by an edge device; generating, by the edge device, a user interface displaying the plurality of network-connected devices; training, by the edge device, a neural network responsive to the environment based on training data, the training data including user actions on the plurality of network-connected devices and user activity recorded by sensors in the environment; identifying, by the edge device, a first action associated with a first network-connected device of the plurality of network-connected devices at a first time; identifying, by the edge device, a second action associated with a second network-connected device of the plurality of network-connected devices; detecting, by the edge device, a relationship between the first network-connected device and the second network-connected device based on the first action and the second action using the neural network; generating, by the edge device, a network-connected device bubble associating the first and second network-connected devices based on the relationship; In response to generating the network-connected device bubble, updating the user interface to display only the first and second network-connected devices; detecting, by the edge device, an additional network-connected device from the plurality of network-connected devices; determining, by the edge device, a negative relationship between the additional network-connected device and the first network-connected device using the neural network; generating a second network-connected device bubble associating the first network-connected device with the additional network-connected device based on the negative relationship; In response to generating a second network-connected device based on the negative relationship, removing display information on the user interface to prevent the additional network-connected device from being provided to the user; A method comprising:

2. identifying, by the edge device, the first action at a second time after the first time; determining, by the edge device, that the first action at the second time is associated with the network-connected device bubble; initiating, by the edge device, the second action on the second network-connected device contemporaneously with the second time; The method of claim 1 further comprising:

3. determining, by the edge device, the processing capabilities of a network-connected device of the plurality of network-connected devices in the environment; assigning, by the edge device, the network-connected device bubble to the one network-connected device based on the processing performance; identifying, by the one network-connected device, the first action at a second time after the first time; determining, by the one network-connected device, that the first action at the second time is associated with the network-connected device bubble; initiating, by the first network-connected device, the second action on the second network-connected device contemporaneously with the second time; The method of claim 1 further comprising: intercepting the user interface including data from the plurality of network-connected devices; hiding data from the additional network-connected device based on the second network-connected device bubble; and The method of claim 1 further comprising:

5. creating a new network-connected device entry corresponding to the network-connected device bubble; providing an entry for the new network-connected device in the user interface; and The method of claim 4 further comprising:

6. the edge device is a voice-based assistant, and identifying the first action includes identifying a voice command of the user. The method of claim 1.

7. presenting, by the edge device, a third action to the user relating to the plurality of network-connected devices in the environment. The method of claim 1.

8. The detection of the relationship is also based on a portable device of the user that initiated the first action. The method of claim 1.

9. Identifying, by the edge device, a first activity of the user in the environment based on data from the sensor; and detecting, by the edge device, a relationship between the second network-connected device and the additional network-connected device based on identifying the first activity using the neural network; generating, by the edge device, a third network-connected device bubble associating the second network-connected device and the additional network-connected device based on the relationship between the second network-connected device and the additional network-connected device; The method of claim 1 further comprising: updating the user interface to display the second network-connected device and the additional network-connected device at a second time; and 10. The method of claim 9, further comprising: Detecting at least one device in a second environment. The method of claim 1 further comprising:

12. In response to said pairing, preventing said plurality of network-connected devices from receiving transmissions from said at least one device. The method of claim 11 further comprising:

13. a memory containing one or more instructions; a processor communicatively coupled to the memory, wherein the processor, in response to reading the one or more instructions, The edge device pairs multiple network-connected devices in the environment; and generating, by the edge device, a user interface displaying the plurality of network-connected devices; training, by the edge device, a neural network responsive to the environment based on training data, the training data including user actions on the plurality of network-connected devices and user activity recorded by sensors in the environment; identifying, by the edge device, a first action associated with a first network-connected device of the plurality of network-connected devices at a first time; identifying, by the edge device, a second action associated with a second network-connected device of the plurality of network-connected devices; detecting, by the edge device, a relationship between the first network-connected device and the second network-connected device based on the first action and the second action using the neural network; generating, by the edge device, a network-connected device bubble associating the first and second network-connected devices at the first time based on the relationship; In response to generating the network-connected device bubble, updating the user interface to display only the first and second network-connected devices at the first time; detecting, by the edge device, an additional network-connected device from the plurality of network-connected devices; determining, by the edge device, a negative relationship between the additional network-connected device and the first network-connected device using the neural network; generating a second network-connected device bubble associating the first network-connected device with the additional network-connected device based on the negative relationship; In response to generating a second network-connected device based on the negative relationship, removing display information on the user interface to prevent the additional network-connected device from being provided to the user; The system is configured to run

14. The processor: identifying, by the edge device, the first action at a second time after the first time; determining, by the edge device, that the first action at the second time is associated with the network-connected device bubble; initiating, by the edge device, the second action on the second network-connected device contemporaneously with the second time; The system of claim 13 further configured to:

15. The processor: determining, by the edge device, the processing capabilities of a network-connected device of the plurality of network-connected devices in the environment; assigning, by the edge device, the network-connected device bubble to the one network-connected device based on the processing performance; identifying, by the one network-connected device, the first action at a second time after the first time; determining, by the one network-connected device, that the first action at the second time is associated with the network-connected device bubble; initiating, by the first network-connected device, the second action on the second network-connected device contemporaneously with the second time; The system of claim 13 further configured to:

16. A computer program comprising program instructions, the program instructions comprising: The edge device pairs multiple network-connected devices in the environment; and generating, by the edge device, a user interface displaying the plurality of network-connected devices; training, by the edge device, a neural network responsive to the environment based on training data, the training data including user actions on the plurality of network-connected devices and user activity recorded by sensors in the environment; identifying, by the edge device, a first action associated with a first network-connected device of the plurality of network-connected devices at a first time; identifying, by the edge device, a second action associated with a second network-connected device of the plurality of network-connected devices; detecting, by the edge device, a relationship between the first network-connected device and the second network-connected device based on the first action and the second action using the neural network; generating, by the edge device, a network-connected device bubble associating the first and second network-connected devices at the first time based on the relationship; In response to generating the network-connected device bubble, updating the user interface to display only the first and second network-connected devices at the first time; detecting, by the edge device, an additional network-connected device from the plurality of network-connected devices; determining, by the edge device, a negative relationship between the additional network-connected device and the first network-connected device using the neural network; generating a second network-connected device bubble associating the first network-connected device with the additional network-connected device based on the negative relationship; In response to generating a second network-connected device based on the negative relationship, removing display information on the user interface to prevent the additional network-connected device from being provided to the user; A computer program that causes a computer to execute the following.

17. The program instructions include: Intercepting the user interface including the plurality of network-connected devices; hiding data from the additional network-connected device based on the second network-connected device bubble; and 17. The computer program of claim 16, further configured to:

18. The program instructions include: creating a new network-connected device entry corresponding to the network-connected device bubble; providing an entry for the new network-connected device in the user interface; and 20. The computer program of claim 17, further configured to:

19. the edge device is a voice-based assistant, and identifying the first action includes identifying a voice command of the user.

17. A computer program according to claim 16.

20. The program instructions include: and further configured to present, by the edge device, a third action to the user related to the plurality of network-connected devices in the environment.

17. A computer program according to claim 16.

Images