A system and method for surfacing an application, including application identification and action notifications.

Abstract

Providing a system and method for displaying control elements for a device based on user input and / or past user interactions. [Solution] This can be accomplished by the system detecting a first input (e.g., a voice query) and then calculating the user's field of view when the first input is detected. The system may use both a first attribute associated with the first input and the user's field of view to identify a first device among multiple devices. The system may then identify an action corresponding to the first device and the first attribute. Display elements associated with the action are read and then displayed for the user. The system may then cause the first device to perform an action in accordance with the user's selection of the display elements.

Description

Technical Field

[0001] The present disclosure relates to techniques for controlling a smart device based on one or more user inputs.

Background Art

[0002] With the rapid spread of smart devices, consumers now have access to a wide range of interconnected devices that can improve daily life by automating tasks, presenting remote control options, and / or providing valuable insights through data. These devices (e.g., thermostats, lights, speakers, TVs, smartphones, laptops, and / or similar such devices) often feature advanced capabilities and functionality that are expanding their adoption. As these devices have evolved to include more complex and comprehensive features, they have also introduced new inefficiencies related to user interaction.

[0003] The user interfaces of many of these devices are complex and may require the user to navigate through various screens, menus, and / or settings to perform desired actions. The user interface may also require the user to perform multiple steps or actions to complete a single task. For example, the steps to adjust a smart thermostat may require the user to launch an application, select the correct device, and configure the temperature. In another example, the steps to activate a light in a smart home may involve launching an application, locating the correct room, selecting a configuration, and confirming the selection. Both examples can frustrate the user because the multiple actions take more time than desired. Furthermore, the problem is exacerbated if the user desires to take multiple actions (e.g., adjusting a thermostat, activating a light, and starting music).

[0004] When interacting with smart devices, users may spend a considerable amount of time learning device functionality, completing repetitive tasks, and / or navigating lengthy sequences of actions. As the number of devices in a typical household increases, the cumulative time spent managing these devices becomes more pronounced, impacting the overall user experience and diminishing the intended benefits of automation and ease of use. [Overview of the Initiative] [Means for solving the problem]

[0005] Therefore, techniques for displaying control elements for a device based on a user's gaze and / or past behavior are disclosed herein. For example, the system may detect a first input. The first input may be a voice command, a gesture, a gaze, and / or similar inputs. The system may use a first device comprising one or more sensors for detecting the first input. The first device may also detect the user's orientation when the first input is detected. For example, the user may be wearing an Extended Reality (XR) device (e.g., Apple Vision Pro (AVP), Meta Quest virtual reality headset, etc.), and the XR device detects the user's head orientation when the first input is detected. In another embodiment, the first device may use one or more image detection models to process images of the user captured by one or more cameras and determine the user's head orientation. The first device may then calculate the user's field of view based on the user's orientation.

[0006] The system may also identify multiple devices connected to a network. In some embodiments, the network is the user's home network, and the multiple devices are smart devices connected to the home network. The system may select a set of devices from the multiple devices connected to the network based at least in part on each device in a set of devices associated with a first input. For example, the first input may be a voice command such as "increase brightness," and the set of devices may include a television, a smartphone, and a light, since a television, a smartphone, and a light all have adjustable brightness settings. The system may then select a subset of the set of devices based at least in part on each device in a subset of devices located within a first field of view. For example, the user's first field of view may be a first room. The system may select a television and a light because they are in the first room, but may not select a smartphone because it is in a second room outside the first field of view.

[0007] The system may then identify and / or generate display elements associated with one or more devices of a subset of devices. The display elements may be associated with a first input. For example, if the first input is the voice command "increase brightness," the display element may be a brightness setting control for a television. The display elements may also include status information. For example, if the first input is the voice command "increase brightness," the display element may be a brightness setting control for a television indicating the current brightness setting. The system may display a display element for the user. For example, the system may display a brightness setting control for a television on an XR device worn by the user. In some embodiments, the system may display more than one display element. For example, the system may display a first display element (e.g., a brightness setting control for a television) and a second display element (e.g., a brightness setting control for a light) on an XR device worn by the user. Therefore, in response to user input, the system identifies the device based on one or more attributes of the user input and one or more attributes of the device.

[0008] The system may also group devices according to past user interactions to facilitate the prediction of user intent. In some embodiments, the system may identify a first device based on a first input. For example, the first device may be a smartphone, and the system identifies the first device when the user selects a fitness application on the smartphone. The system may also identify a first set of attributes associated with the smartphone and / or the fitness application. For example, the first set of attributes may include brightness, health, notifications, media sharing, and / or similar attributes. The system may also identify a second device based on a second input. For example, the second device may be a thermostat, and the system identifies the second device when the user gazes at the thermostat. The system may also identify a second set of attributes associated with the thermostat. For example, the second set of attributes may include comfort, health, energy, and / or similar attributes.

[0009] If one or more attributes of a second set of attributes match those of a first set of attributes, the system may identify one or more actions. For example, the system may determine that both the second set of attributes and the first set of attributes contain "health." Because both the second set of attributes and the first set of attributes contain "health," the system may determine one or more actions (e.g., turning on the air conditioning or adjusting the temperature) related to a second device (e.g., a thermostat). The system may then cause the second device (e.g., a thermostat) to perform one or more actions (e.g., turning on the air conditioning). The system may automatically cause the second device to perform one or more actions based at least partially on the second input and past user interactions. For example, the system may determine that the user consistently uses the thermostat to turn on the air conditioning while using a fitness application on their smartphone. Therefore, when the user selects a fitness application on their smartphone and looks at the thermostat, the system may automatically turn on the air conditioning based on past user interactions.

[0010] In some embodiments, the system also displays one or more display elements related to one or more actions. For example, an XR device worn by a user may display a temperature control display in response to the user selecting a fitness application on a smartphone and looking at a thermostat. In some embodiments, none of the previous user interactions associate the first and second devices. For example, it may be the first time that a user uses a fitness application on a smartphone and looks at a thermostat. In such embodiments, the system may display one or more display elements (e.g., a temperature control display) based on one or more attributes (e.g., health) of the second device (e.g., a thermostat) that match one or more attributes (e.g., health) of the first device (e.g., a smartphone).

[0011] In some embodiments, the system also surfaces applications based on one or more user inputs. For example, the system may receive a voice query (e.g., "open fitness application"). The system may identify a first application (e.g., a fitness application) in response to the voice query. The system may then identify a plurality of applications that share one or more attributes with the first application. For example, the first application (e.g., a fitness application) may be associated with a first attribute (e.g., health). The system may also identify a second application (e.g., a music playlist application) and a third application (e.g., a thermostat application) because they are also associated with the first attribute (e.g., health). The system may then issue a first prompt to the second and third applications. The first prompt may indicate that input related to the first attribute (e.g., health) has been detected. The second application may then issue a second prompt indicating the first action (for example, playing a training playlist). The system may then display a notification corresponding to the first action. For example, the system may display a first notification (for example, "Do you want me to start playing your workout playlist?"). In response to the user selecting the first notification, the system may cause the second application (for example, a music playlist application) to perform the first action (for example, playing a training playlist). The present invention provides, for example, the following items: (Item A1) It is a method, The first device detects a first input, A step of determining that the first input corresponds to the first attribute, A first device detects the user's orientation when a first input is detected, The process involves, at least partially, calculating a first field of view based on the user's orientation, A step of identifying multiple devices connected to a network, wherein each of the multiple devices is associated with at least one attribute, and The steps include, at least partially, selecting a set of devices from a set of devices based on each device in the set of devices associated with a first attribute, A step of selecting a subset of devices from a set of devices based on each device in a subset of devices located at least partially within a first field of view, The steps include identifying a first display element associated with a second device of a subset of devices, The steps include displaying the first display element and Methods that include... (Item A2) The first device is a wearable extended reality (XR) device, and the first device displays a first display element, as described in the above item. (Item A3) The steps include receiving a second input associated with a first display element, The steps include causing a second device to execute a command, at least partially, based on the step of receiving a second input, and The method described in any one of the above items, further including: (Item A4) The steps include identifying a second display element associated with a third device of a subset of devices, The first device displays the second display element, and The method described in any one of the above items, further including: (Item A5) The first device displays a first display element and a second display element at the same time, as described in any one of the above items. (Item A6) The steps include receiving a second input associated with a first display element, The steps include causing a second device to execute a command, at least partially, based on the step of receiving a second input, and The method described in any one of the above items, further including: (Item A7) A step of processing a first input using a natural language processing (NLP) model to determine at least one characteristic associated with the first input, wherein the first input is a speech query containing at least one word, and A step of accessing a database containing multiple entries, wherein each entry of the multiple entries is associated with an attribute and one or more characteristics, A step of identifying a first entry of multiple entries, wherein the first entry associates at least one characteristic with a first attribute. The steps include, at least in part, determining that the first input corresponds to a first attribute based on the step of identifying a first entry, and The method described in any one of the above items, further including: (Item A8) Steps to register multiple devices, A step of identifying at least one attribute for each device among multiple devices, The data structure includes the step of storing at least one attribute for each device. The method described in any one of the above items, further including: (Item A9) The steps include receiving a second input associated with a first display element, A step of transmitting a request to a second device, wherein the request indicates a command associated with a second input, and The method according to any one of the above items, further comprising (Item A10) The method according to any one of the above items, wherein the first display element corresponds to an action associated with the first attribute. (Item A11) An apparatus, a control circuitry, at least one memory including computer program code for one or more programs, wherein the at least one memory and the computer program code are configured, using the control circuitry, to cause the apparatus to perform at least the following, namely, detecting a first input; determining that the first input corresponds to a first attribute; detecting a user orientation when the first input is detected; calculating a first viewing field based at least in part on the user orientation; identifying a plurality of devices connected to a network, each device of the plurality of devices being associated with at least one attribute; selecting a set of devices from the plurality of devices based at least in part on each device of a set of devices associated with the first attribute; selecting a subset of devices of the set of devices based at least in part on each device of a subset of devices located within the first viewing field; identifying a first display element associated with a first device of the subset of devices; displaying the first display element and at least one memory configured to cause the apparatus comprising. (Item A12) The apparatus according to any one of the above items, wherein the apparatus is a wearable extended reality (XR) device. (Item A13) The apparatus further comprises The steps include receiving a second input associated with a first display element, The steps include causing a first device to execute a command, at least partially, based on the step of receiving a second input, and A device capable of performing any one of the above items. (Item A14) This device further, The steps include identifying a second display element associated with a second device of a subset of devices, The step of displaying the second display element A device capable of performing any one of the above items. (Item A15) This device is the device described in any one of the above items, which displays a first display element and a second display element at the same time. (Item A16) This device further, The steps include receiving a second input associated with a first display element, The steps include causing a first device to execute a command, at least partially, based on the step of receiving a second input, and A device capable of performing any one of the above items. (Item A17) This device further, A step of processing a first input using a natural language processing (NLP) model to determine at least one characteristic associated with the first input, wherein the first input is a speech query containing at least one word, and A step of accessing a database containing multiple entries, wherein each entry of the multiple entries is associated with an attribute and one or more characteristics, A step of identifying a first entry of multiple entries, wherein the first entry associates at least one characteristic with a first attribute. The steps include, at least in part, determining that the first input corresponds to a first attribute based on the step of identifying a first entry, and A device capable of performing any one of the above items. (Item A18) This device further, Steps to register multiple devices, A step of identifying at least one attribute for each device among multiple devices, The data structure includes the step of storing at least one attribute for each device. A device capable of performing any one of the above items. (Item A19) This device further, The steps include receiving a second input associated with a first display element, A step of transmitting a request to a first device, wherein the request indicates a command associated with a second input, and A device capable of performing any one of the above items. (Item A20) The first display element is a device described in any one of the above items, corresponding to an action associated with the first attribute. (Item A21) A non-transient computer-readable medium having instructions encoded thereon, where the instructions, when executed by a control circuit network, A step of detecting a first input, A step of determining that the first input corresponds to the first attribute, A step of detecting the user orientation when a first input is detected, The process involves, at least partially, calculating a first field of view based on the user's orientation, A step of identifying multiple devices connected to a network, wherein each of the multiple devices is associated with at least one attribute, and The steps include, at least partially, selecting a set of devices from a set of devices based on each device in the set of devices associated with a first attribute, A step of selecting a subset of devices from a set of devices based on each device in a subset of devices located at least partially within a first field of view, The steps include identifying a first display element associated with a first device of a subset of devices, The steps include displaying the first display element and A non-transient, computer-readable medium that enables the following action. (Item A22) The control circuit network is a non-transient computer-readable medium as described in any one of the above items, which is part of a wearable extended reality (XR) device. (Item A23) The control network further, The steps include receiving a second input associated with a first display element, The steps include causing a first device to execute a command, at least partially, based on the step of receiving a second input, and A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A24) The control network further, The steps include identifying a second display element associated with a second device of a subset of devices, The step of displaying the second display element A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A25) The control circuit network is a non-transient computer-readable medium as described in any one of the above items, which displays the first display element and the second display element at the same time. (Item A26) The control network further, The steps include receiving a second input associated with a first display element, The steps include causing a first device to execute a command, at least partially, based on the step of receiving a second input, and A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A27) The control network further, A step of processing a first input using a natural language processing (NLP) model to determine at least one characteristic associated with the first input, wherein the first input is a speech query containing at least one word, and A step of accessing a database containing multiple entries, wherein each entry of the multiple entries is associated with an attribute and one or more characteristics, A step of identifying a first entry of multiple entries, wherein the first entry associates at least one characteristic with a first attribute. The steps include, at least in part, determining that the first input corresponds to a first attribute based on the step of identifying a first entry, and A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A28) The control network further, Steps to register multiple devices, A step of identifying at least one attribute for each device among multiple devices, The data structure includes the step of storing at least one attribute for each device. A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A29) The control network further, The steps include receiving a second input associated with a first display element, A step of transmitting a request to a first device, wherein the request indicates a command associated with a second input, and A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A30) The first display element is a non-transient, computer-readable medium described in any one of the above items, corresponding to the action associated with the first attribute. (Item A31) It is a system, The first device provides means for detecting a first input, A means for determining that a first input corresponds to a first attribute, A means for detecting the user's orientation when a first input is detected by a first device, Means for calculating a first field of view, at least in part, based on the user's orientation, A means for identifying multiple devices connected to a network, wherein each of the multiple devices is associated with at least one attribute, and the means is... A means for selecting a set of devices from a group of devices, at least in part, based on each device in the set of devices associated with a first attribute, Means for selecting a subset of devices from a set of devices based at least partially on each device in a subset of devices located within a first field of view, Means for identifying a first display element associated with a second device of a subset of devices, Means for displaying the first display element and A system equipped with these features. (Item A32) The first device is a wearable extended reality (XR) device, and the first device is a system according to any one of the above items, which displays the first display element. (Item A33) Means for receiving a second input associated with a first display element, Means for causing a second device to execute a command, at least in part, based on the step of receiving a second input. A system further comprising any one of the above items. (Item A34) Means for identifying a second display element associated with a third device of a subset of devices, A means for displaying a second display element by a first device and A system further comprising any one of the above items. (Item A35) The first device is a system according to any one of the above items, which displays a first display element and a second display element at the same time. (Item A36) Means for receiving a second input associated with a first display element, Means for causing a second device to execute a command, at least in part, based on the step of receiving a second input. A system further comprising any one of the above items. (Item A37) A means for processing a first input using a natural language processing (NLP) model to determine at least one characteristic associated with the first input, wherein the first input is a speech query containing at least one word; A means for accessing a database containing multiple entries, wherein each entry of the multiple entries is associated with an attribute and one or more characteristics, A means for identifying a first entry of multiple entries, wherein the first entry associates at least one characteristic with a first attribute, At least in part, based on the step of identifying a first entry, it is determined that the first input corresponds to a first attribute. A system further comprising any one of the above items. (Item A38) A means for registering multiple devices, A means for identifying at least one attribute for each device among multiple devices, Within the data structure, there is a means for storing at least one attribute per device. A system further comprising any one of the above items. (Item A39) Means for receiving a second input associated with a first display element, Means for transmitting a request to a second device, wherein the request indicates a command associated with a second input, and A system further comprising any one of the above items. (Item A40) The first display element is a system described in any one of the above items, corresponding to the action associated with the first attribute. (Item A41) It is a method, A step of detecting a first input, The steps include, at least partially, identifying a first device based on a first input, The steps include identifying a first set of attributes associated with a first device, A step of detecting a second input, The steps include, at least partially, identifying a second device based on a second input, Steps include identifying a second set of attributes and multiple actions associated with a second device, A step of selecting a first attribute, wherein the first set of attributes includes the first attribute, and the second set of attributes includes the first attribute, The steps include, at least partially, identifying the first action of a group of actions based on a first attribute, The second device performs the first action. Methods that include... (Item A42) A step of identifying a first display element associated with a first action, The steps include displaying the first display element, A step of receiving a selection of a first display element, wherein a second device is at least partially made to perform a first action based on the selection. The method described in any one of the above items, further including: (Item A43) The first display element is the method described in any one of the above items, as displayed by a wearable extended reality (XR) device. (Item A44) Steps include receiving multiple preferences, A step of identifying a first preference among multiple preferences, The first preference indicates that no confirmation is required to carry out the first action. The second device is made to perform a first action based at least in part on the step of identifying a first preference, and The method described in any one of the above items, further including: (Item A45) Steps include receiving multiple user interactions, A step of determining that a first user interaction of a plurality of user interactions is related to a second user interaction of a plurality of user interactions, wherein the first attribute is selected based on the first user interaction which is at least partially related to the second user interaction. The method described in any one of the above items, further including: (Item A46) The steps include receiving a first set of attributes associated with a first device, The steps include receiving a second set of attributes associated with a second device, A step of storing a first set of attributes associated with a first device, a second set of attributes associated with a second device, and multiple user interactions in a data structure. The method described in any one of the above items, further including: (Item A47) The method described in any one of the above items, wherein an artificial intelligence model determines that a first user interaction of multiple user interactions is related to a second user interaction of multiple user interactions. (Item A48) The first input is a voice query, as described in any one of the above items. (Item A49) The first input is the user's gaze, as described in any one of the above items. (Item A50) The first input is a tap on the touchscreen, as described in any one of the above items. (Item A51) It is a device, Control circuit network and A memory containing computer program code relating to one or more programs, wherein the memory and the computer program code are connected to the device using a control circuit network, at least as follows: A step of detecting a first input, The steps include, at least partially, identifying a first device based on a first input, The steps include identifying a first set of attributes associated with a first device, A step of detecting a second input, The steps include, at least partially, identifying a second device based on a second input, Steps include identifying a second set of attributes and multiple actions associated with a second device, A step of selecting a first attribute, wherein the first set of attributes includes the first attribute, and the second set of attributes includes the first attribute, The steps include, at least partially, identifying the first action of a group of actions based on a first attribute, The step of having the second device perform the first action and A memory and A device equipped with the following features. (Item A52) This device further, A step of identifying a first display element associated with a first action, The steps include displaying the first display element, A step of receiving a selection of a first display element, wherein a second device is at least partially made to perform a first action based on the selection. A device capable of performing any one of the above items. (Item A53) The first display element is a device as described in any one of the above items, displayed by a wearable extended reality (XR) device. (Item A54) This device further, Steps include receiving multiple preferences, A step of identifying a first preference among multiple preferences, The first preference indicates that no confirmation is required to carry out the first action. The second device is made to perform a first action based at least in part on the step of identifying a first preference, and A device capable of performing any one of the above items. (Item A55) This device further, Steps include receiving multiple user interactions, A step of determining that a first user interaction of a plurality of user interactions is related to a second user interaction of a plurality of user interactions, wherein the first attribute is selected based on the first user interaction which is at least partially related to the second user interaction. A device capable of performing any one of the above items. (Item A56) This device further, The steps include receiving a first set of attributes associated with a first device, The steps include receiving a second set of attributes associated with a second device, A step of storing a first set of attributes associated with a first device, a second set of attributes associated with a second device, and multiple user interactions in a data structure. A device capable of performing any one of the above items. (Item A57) The apparatus according to any one of the above items, wherein an artificial intelligence model determines that a first user interaction of multiple user interactions is related to a second user interaction of multiple user interactions. (Item A58) The first input is a voice query, and the device is one of the items described in any one of the above items. (Item A59) The first input is the user's line of sight, as described in one of the above items. (Item A60) The first input is a tap on a touchscreen, as described in one of the items above. (Item A61) A non-transient computer-readable medium having instructions encoded thereon, where the instructions, when executed by a control circuit network, A step of detecting a first input, The steps include, at least partially, identifying a first device based on a first input, The steps include identifying a first set of attributes associated with a first device, A step of detecting a second input, The steps include, at least partially, identifying a second device based on a second input, Steps include identifying a second set of attributes and multiple actions associated with a second device, A step of selecting a first attribute, wherein the first set of attributes includes the first attribute, and the second set of attributes includes the first attribute, The steps include, at least partially, identifying the first action of a group of actions based on a first attribute, The step of having the second device perform the first action and A non-transient, computer-readable medium that enables the following action. (Item A62) The control network further, A step of identifying a first display element associated with a first action, The steps include displaying the first display element, A step of receiving a selection of a first display element, wherein a second device is at least partially made to perform a first action based on the selection. A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A63) The first display element is a non-transient, computer-readable medium as described in any one of the above items, displayed by a wearable Extended Reality (XR) device. (Item A64) The control network further, Steps include receiving multiple preferences, A step of identifying a first preference among multiple preferences, The first preference indicates that no confirmation is required to carry out the first action. The second device is made to perform a first action based at least in part on the step of identifying a first preference, and A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A65) The control network further, Steps include receiving multiple user interactions, A step of determining that a first user interaction of a plurality of user interactions is related to a second user interaction of a plurality of user interactions, wherein the first attribute is selected based on the first user interaction which is at least partially related to the second user interaction. A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A66) The control network further, The steps include receiving a first set of attributes associated with a first device, The steps include receiving a second set of attributes associated with a second device, A step of storing a first set of attributes associated with a first device, a second set of attributes associated with a second device, and multiple user interactions in a data structure. A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A67) A non-transient computer-readable medium as described in any one of the above items, wherein an artificial intelligence model determines that the first user interaction of a group of user interactions is related to the second user interaction of a group of user interactions. (Item A68) The first input is a voice query, a non-transient, computer-readable medium as described in any one of the items above. (Item A69) The first input is the user's gaze, a non-transient computer-readable medium as described in any one of the above items. (Item A70) The first input is a tap on a touchscreen, a non-transient computer-readable medium as described in any one of the above items. (Item A71) It is a system, Means for detecting a first input, Means for identifying a first device based on a first input, at least in part, Means for identifying a first set of attributes associated with a first device, Means for detecting a second input, Means for identifying a second device based at least partially on a second input, Means for identifying a second set of attributes and multiple actions associated with a second device, A means for selecting a first attribute, wherein a first set of attributes includes the first attribute, and a second set of attributes includes the first attribute, A means for identifying a first action of multiple actions based at least partially on a first attribute, The second device provides a means for performing the first action and A system equipped with these features. (Item A72) Means for identifying a first display element associated with a first action, Means for displaying the first display element, Means for receiving the selection of a first display element, wherein a second device is at least partially caused to perform a first action based on the selection. The system described in any one of the above items, further including the system described in any one of the above items. (Item A73) The first display element is a system as described in any one of the above items, displayed by a wearable Extended Reality (XR) device. (Item A74) Means for receiving multiple preferences, A means for identifying a first preference among multiple preferences, The first preference indicates that no confirmation is required to carry out the first action. The second device is a means that causes a first action to be performed, at least in part, based on the step of identifying a first preference. A system further comprising any one of the above items. (Item A75) Means for receiving multiple user interactions, Means for determining that a first user interaction of a plurality of user interactions is related to a second user interaction of a plurality of user interactions, wherein the first attribute is selected, at least in part, based on the first user interaction that is related to the second user interaction. A system further comprising any one of the above items. (Item A76) Means for receiving a first set of attributes associated with a first device, Means for receiving a second set of attributes associated with a second device, A first set of attributes associated with a first device, a second set of attributes associated with a second device, and means for storing multiple user interactions within a data structure. A system further comprising any one of the above items. (Item A77) A system as described in any one of the above items, wherein an artificial intelligence model determines that the first user interaction of a group of user interactions is related to the second user interaction of a group of user interactions. (Item A78) The first input is a voice query, one of the systems described in any one of the above items. (Item A79) The first input is the user's gaze, as described in one of the items above. (Item A80) The first input is a tap on a touchscreen, as described in one of the items above. (Item A81) It is a method, The steps include receiving a first application and a first set of attributes associated with the first application, Steps include: storing the first set of attributes, A step of detecting a first input, The steps include, at least partially, identifying a first application based on a first input, The steps include selecting a first attribute of a first set of attributes associated with a first application, A step of identifying a first entry of multiple entries, Each entry in a set of multiple entries associates one or more attributes with one or more applications. The first entry involves the steps of associating the first attribute with the first application and one or more additional applications, A step of identifying a second application of one or more additional applications, based at least in part on a step of identifying a first entry of a plurality of entries, wherein the second application is associated with a plurality of actions, The steps include, at least partially, identifying the first action of a group of actions based on a first attribute, A step of displaying a first notification, wherein the first notification corresponds to a first action and Methods that include... (Item A82) The first step is to receive the selection of the notification, A step of causing the device to perform a first action, at least partially, based on the selection of a first notification. The method described in any one of the above items, further including: (Item A83) The first notification is displayed on a wearable Extended Reality (XR) device, as described in any one of the above items. (Item A84) The first set of attributes and the first application are stored in the device's memory, as described in any one of the above items. (Item A85) This device is a wearable Extended Reality (XR) device. This device displays the first notification. The method described in any one of the above items. (Item A86) A step of transmitting a first prompt associated with a first attribute to a second application, The second application receives a second prompt associated with the first action. The method described in any one of the above items, further including: (Item A87) A step of identifying a third application of one or more additional applications, based at least in part on a step of identifying a first entry of a plurality of entries, wherein the third application is associated with a second plurality of actions, The steps include, at least partially, identifying the second action of a second set of actions based on a first attribute, A step of displaying a second notification, the second notification corresponding to a second action and The method described in any one of the above items, further including: (Item A88) The first and second notices are to be displayed at the same time, in accordance with the method described in any one of the above items. (Item A89) The first input is the user's gaze, as described in any one of the above items. (Item A90) The first input is a tap on the touchscreen, as described in any one of the above items. (Item A91) It is a device, Control circuit network and A memory containing computer program code relating to one or more programs, wherein the memory and the computer program code are connected to the device using a control circuit network, at least as follows: The steps include receiving a first application and a first set of attributes associated with the first application, Steps include: storing the first set of attributes, A step of detecting a first input, The steps include, at least partially, identifying a first application based on a first input, The steps include selecting a first attribute of a first set of attributes associated with a first application, A step of identifying a first entry of multiple entries, Each entry in a set of multiple entries associates one or more attributes with one or more applications. The first entry involves the steps of associating the first attribute with the first application and one or more additional applications, A step of identifying a second application of one or more additional applications, based at least in part on a step of identifying a first entry of a plurality of entries, wherein the second application is associated with a plurality of actions, The steps include, at least partially, identifying the first action of a group of actions based on a first attribute, A step of displaying a first notification, wherein the first notification corresponds to a first action and A memory device configured to perform the following: A device equipped with the following features. (Item A92) This device further, The first step is to receive the selection of the notification, A step of causing the device to perform a first action, at least partially, based on the selection of a first notification. A device capable of performing any one of the above items. (Item A93) The first notification is a device described in any one of the above items, displayed on a wearable extended reality (XR) device. (Item A94) The first set of attributes and the first application are stored in the device's memory, as described in any one of the above items. (Item A95) This device is a wearable Extended Reality (XR) device. This device displays the first notification. The apparatus described in any one of the above items. (Item A96) This device further, A step of transmitting a first prompt associated with a first attribute to a second application, The second application receives a second prompt associated with the first action. A device capable of performing any one of the above items. (Item A97) This device further, A step of identifying a third application of one or more additional applications, based at least in part on a step of identifying a first entry of a plurality of entries, wherein the third application is associated with a second plurality of actions, The steps include, at least partially, identifying the second action of a second set of actions based on a first attribute, A step of displaying a second notification, the second notification corresponding to a second action and A device capable of performing any one of the above items. (Item A98) The first and second notices are displayed at the same time on the device described in either of the above items. (Item A99) The first input is the user's line of sight, as described in one of the above items. (Item A100) The first input is a tap on a touchscreen, as described in one of the items above. (Item A101) A non-transient computer-readable medium having instructions encoded thereon, where the instructions, when executed by a control circuit network, The steps include receiving a first application and a first set of attributes associated with the first application, Steps include: storing the first set of attributes, A step of detecting a first input, The steps include, at least partially, identifying a first application based on a first input, The steps include selecting a first attribute of a first set of attributes associated with a first application, A step of identifying a first entry of multiple entries, Each entry in a set of multiple entries associates one or more attributes with one or more applications. The first entry involves the steps of associating the first attribute with the first application and one or more additional applications, A step of identifying a second application of one or more additional applications, based at least in part on a step of identifying a first entry of a plurality of entries, wherein the second application is associated with a plurality of actions, The steps include, at least partially, identifying the first action of a group of actions based on a first attribute, A step of displaying a first notification, wherein the first notification corresponds to a first action and A non-transient, computer-readable medium that enables the following action. (Item A102) The control network further, The first step is to receive the selection of the notification, A step of causing the device to perform a first action, at least partially, based on the selection of a first notification. A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A103) The first notice is a non-transient, computer-readable medium displayed on a wearable Extended Reality (XR) device, as described in any one of the items above. (Item A104) The first set of attributes and the first application are stored in the device's memory in a non-transient, computer-readable medium as described in any one of the above items. (Item A105) This device is a wearable Extended Reality (XR) device. This device displays the first notification. A non-transient computer-readable medium as described in any one of the above items. (Item A106) The control network further, A step of transmitting a first prompt associated with a first attribute to a second application, The second application receives a second prompt associated with the first action. A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A107) The control network further, A step of identifying a third application of one or more additional applications, based at least in part on a step of identifying a first entry of a plurality of entries, wherein the third application is associated with a second plurality of actions, The steps include, at least partially, identifying the second action of a second set of actions based on a first attribute, A step of displaying a second notification, the second notification corresponding to a second action and A non-transient computer-readable medium described in any one of the above items, which can perform the following actions. (Item A108) The first and second notices shall be displayed at the same time in a non-transient, computer-readable medium as described in either of the above items. (Item A109) The first input is the user's gaze, a non-transient computer-readable medium as described in any one of the above items. (Item A110) The first input is a tap on a touchscreen, a non-transient computer-readable medium as described in any one of the above items. (Item A111) It is a system, Receiving a first application and a first set of attributes associated with the first application, A means for storing the first set of attributes, Means for detecting a first input, A means for identifying a first application based on a first input, at least in part, A means for selecting a first attribute of a first set of attributes associated with a first application, A means for identifying a first entry of a plurality of entries, Each entry in a set of multiple entries associates one or more attributes with one or more applications. The first entry includes means for associating the first attribute with the first application and one or more additional applications, A means for identifying a second application of one or more additional applications, based at least in part on the step of identifying a first entry of a plurality of entries, wherein the second application is associated with a plurality of actions, A means for identifying a first action of multiple actions based at least partially on a first attribute, A means for displaying a first notice, wherein the first notice corresponds to a first action and A system equipped with these features. (Item A112) Means for receiving the first notification selection, Means for causing the device to perform a first action based at least partially on the selection of a first notification and A system further comprising any one of the above items. (Item A113) The first notification is displayed on a wearable Extended Reality (XR) device and is one of the systems described in any one of the above items. (Item A114) The first set of attributes and the first application are stored in the device's memory, in the system described in any one of the above items. (Item A115) This device is a wearable Extended Reality (XR) device. This device displays the first notification. The system described in any one of the above items. (Item A116) Means for transmitting a first prompt associated with a first attribute to a second application, A means for receiving a second prompt associated with a first action from a second application, A system further comprising any one of the above items. (Item A117) A means for identifying a third application of one or more additional applications, based at least in part on the step of identifying a first entry of a plurality of entries, wherein the third application is associated with a second plurality of actions, A means for identifying a second action of a second set of actions, at least in part, based on a first attribute, A means for displaying a second notice, the second notice being a means corresponding to a second action and A system further comprising any one of the above items. (Item A118) The first and second notices are displayed at the same time on the system described in either of the above items. (Item A119) The first input is the user's gaze, as described in one of the items above. (Item A120) The first input is a tap on a touchscreen, as described in one of the items above. (Item B1) It is a method, Receiving a first application and a first set of attributes associated with the first application, To store the first set of the aforementioned attributes, To detect the first input, Identifying the first application based at least in part on the first input, Selecting a first attribute from a first set of attributes associated with the first application, Identifying the first entry among multiple entries, Each of the aforementioned entries associates one or more attributes with one or more applications. The first entry associates the first attribute with the first application and one or more additional applications, Identifying a second application among one or more additional applications, at least in part, based on identifying the first entry among the plurality of entries, wherein the second application is associated with a plurality of actions, Identifying a first action among the plurality of actions based at least in part on the first attribute, Displaying a first notification, wherein the first notification corresponds to the first action. Methods that include... (Item B2) Receiving the selection of the first notification mentioned above, To cause the device to perform the first action based at least in part on the selection of the first notification The method described in the above items, further including the method described in the above items. (Item B3) The first notification is displayed on a wearable extended reality (XR) device, and is provided in the manner described in any one of the above items. (Item B4) The first set of attributes and the first application are stored in the device's memory, as described in any one of the above items. (Item B5) The aforementioned device is a wearable extended reality (XR) device, The device displays the first notification as described in any one of the above items. (Item B6) Transmitting a first prompt associated with the first attribute to the second application, Receiving a second prompt from the second application that is associated with the first action The method described in any one of the above items, further including: (Item B7) Identifying a third application among the one or more additional applications, at least in part on identifying the first entry among the plurality of entries, wherein the third application is associated with the second plurality of actions, Identifying the second action among the second multiple actions based at least partially on the first attribute, Displaying a second notice, wherein the second notice corresponds to the second action. The method described in any one of the above items, further including: (Item B8) The first and second notices are displayed at the same time, as described in any one of the above items. (Item B9) The method according to any one of the above items, wherein the first input is at least one of the user's gaze or a tap on a touchscreen. (Item B10) It is a device, Control circuit network and A memory containing computer program code relating to one or more programs, wherein the at least one memory and the computer program code are transmitted to the device using the control circuit network, at least the following: Receiving a first application and a first set of attributes associated with the first application, To store the first set of the aforementioned attributes, To detect the first input, Identifying the first application based at least in part on the first input, Selecting a first attribute from a first set of attributes associated with the first application, Identifying the first entry among multiple entries, Each of the aforementioned entries associates one or more attributes with one or more applications. The first entry associates the first attribute with the first application and one or more additional applications, Identifying a second application among one or more additional applications, at least in part, based on identifying the first entry among the plurality of entries, wherein the second application is associated with a plurality of actions, Identifying a first action among the plurality of actions based at least in part on the first attribute, Displaying a first notification, wherein the first notification corresponds to the first action. A memory and A device equipped with the following features. (Summary) A system and method for displaying control elements for a device based on user input and / or past user interactions are provided herein. This can be accomplished by the system detecting a first input (e.g., a voice query) and then calculating the user's field of view when the first input was detected. The system may use both a first attribute associated with the first input and the user's field of view to identify a first device of a plurality of devices. The system may then identify an action corresponding to the first device and the first attribute. Display elements associated with the action are read and then displayed for the user. The system may then cause the first device to perform an action in accordance with the user's selection of the display elements. [Brief explanation of the drawing]

[0012] This disclosure will be described in detail with reference to the following drawings according to one or more different embodiments. The drawings are provided for illustrative purposes only and depict typical or exemplary embodiments. These drawings are provided to facilitate understanding of the concepts disclosed herein and should not be considered as an limitation of the scope, scope, or availability of these concepts. Note that for clarity and ease of illustration, these drawings are not necessarily made to scale.

[0013] [Figure 1] Figure 1 illustrates an illustrative diagram of a system for displaying control elements for a device based on one or more inputs, according to some embodiments of the present disclosure.

[0014] [Figure 2A] Figures 2A-2B show devices for displaying one or more display elements according to some embodiments of the present disclosure. [Figure 2B] Figures 2A-2B show devices for displaying one or more display elements according to some embodiments of the present disclosure.

[0015] [Figure 3] Figure 3 shows another illustrative diagram of a system for displaying control elements for a device based on one or more inputs, according to some embodiments of the present disclosure.

[0016] [Figure 4] Figure 4 shows another illustrative diagram of a system for displaying control elements for a device based on one or more inputs, according to some embodiments of the present disclosure.

[0017] [Figure 5] Figure 5 shows another illustrative diagram of a system for displaying control elements for a device based on one or more inputs, according to some embodiments of the present disclosure.

[0018] [Figure 6] Figure 6 shows an illustrative block diagram of a system according to an embodiment of the present disclosure.

[0019] [Figure 7] Figure 7 shows an illustrative block diagram of a media system according to an embodiment of the present disclosure.

[0020] [Figure 8] Figure 8 shows an illustrative block diagram of a user equipment device system according to several embodiments of the present disclosure.

[0021] [Figure 9] Figure 9 is an illustrative flowchart of a process for displaying control elements for a device based on one or more inputs, according to some embodiments of the present disclosure.

[0022] [Figure 10] Figure 10 is another illustrative flowchart of a process for displaying control elements for a device based on one or more inputs, according to some embodiments of the present disclosure.

[0023] [Figure 11] Figure 11 is an illustrative flowchart of a process for displaying a notification based on one or more inputs, according to some embodiments of the present disclosure. [Modes for carrying out the invention]

[0024] Detailed explanation Figure 1 illustrates a system 100 for displaying control elements for a device based on one or more inputs, according to several embodiments of the present disclosure. The system 100 includes a user 104 who interacts with a first device 102. In some embodiments, the first device 102 is an XR device worn by the user 104. In some embodiments, the first device 102 is a smartwatch, smartphone, laptop, and / or similar device. The system 100 may also include a second device 112, a third device 114, and a fourth device 116. Although four devices are shown, any number of devices may be used. For example, more or fewer devices than those shown may be present.

[0025] System 100 may identify a plurality of devices connected to a network. In some embodiments, the network is a home network, and the plurality of devices are smart devices connected to the home network. System 100 may identify each of the plurality of devices using one or more registration processes. During the registration process, System 100 may receive device information. System 100 may use the device information to determine attributes corresponding to each device. For example, a second device 112 may be associated with a first set of attributes (e.g., brightness, volume, media sharing, and / or similar attributes), and a third device 114 may be associated with a second attribute (e.g., brightness). System 100 may store the attributes associated with each device in one or more profiles. In some embodiments, a control hub (e.g., control hub 614) stores the device information and / or attributes associated with the plurality of devices. In some embodiments, the devices communicate with each other and / or with the control hub using one or more wired and / or wireless communication protocols. For example, the first device 102, the second device 112, the third device 114, and the fourth device 116 may communicate via a Wi-Fi network.

[0026] In some embodiments, system 100 detects a first input. The first input may be a voice query, gesture, gaze, and / or similar input. For example, the first input may be a voice query uttered by user 104 (e.g., "increase brightness"). In another embodiment, the first input may be a tap on the touch display of the first device 102. In another embodiment, the first input may be user 104 gazing at the second device 112. In another embodiment, the first input may be user 104 pointing to the second device 112 with their hand / finger. In another embodiment, the first input may be user 104 blinking and / or moving their eyes. System 100 may use one or more sensors to detect the first input. One or more sensors may include a camera, depth sensor, lidar sensor, position sensor, pressure sensor, accelerometer, audio sensor, and / or similar sensors.

[0027] In some embodiments, the first device 102 includes one or more sensors used to detect a first input. For example, the first device 102 may include a microphone that detects a voice query (e.g., "increase brightness") uttered by the user 104. In some embodiments, one or more other devices include one or more sensors used to detect a first input. For example, the second device 112 may include a camera that detects a gesture made by the user 104.

[0028] In some embodiments, the system 100 also detects the orientation of the user 104 when a first input is detected. For example, the first device 102 may be an XR device worn by the user 104, and the first device 102 detects the orientation of the user's head when a first input is detected. In another embodiment, one or more devices may use one or more image detection models to process images of the user 104 captured by one or more cameras to determine the orientation of the user 104's head. In yet another embodiment, one or more devices may use one or more image detection models to process images of the user 104 captured by one or more cameras to determine the location of one or more of the user 104's eyes and / or pupils. The system 100 may then calculate the user 104's field of view 110 based on the user 104's orientation. In some embodiments, the system 100 uses any of the sensors described herein to detect and / or determine the user 104's field of view 110.

[0029] In response to a first input, system 100 may identify one or more devices among the devices connected to the network, at least in part, based on one or more devices associated with the first input. In some embodiments, system 100 uses one or more attributes of the devices to determine one or more devices associated with the first input. For example, the first input may be a voice query such as "increase brightness," and the set of devices may include a second device 112, a third device 114, and a fourth device 116, since the second device 112, a third device 114, and a fourth device 116 all have adjustable brightness settings. In another embodiment, the first input may be a voice query such as "decrease volume," and the set of devices may include a second device 112 and a fourth device 116, since the second device 112 and a fourth device 116 all have adjustable volume settings.

[0030] The system 100 may select a subset of devices associated with the first input based at least partially on each device in a subset of devices located within the user 104's field of view 110. For example, the first input may be a voice query such as "increase brightness," and the set of devices may include the second device 112, the third device 114, and the fourth device 116, since the second device 112, the third device 114, and the fourth device 116 all have adjustable brightness settings. The system 100 may select the second device 112 and the third device 114 as a subset of devices because the second device 112 and the third device 114 are within the user 104's field of view 110. In some embodiments, the system 100 selects a single device associated with the first input based at least partially on a single device located within the first field of view. For example, the first input may be a voice query such as "decrease volume," and the set of devices may include the second device 112 and the fourth device 116, since the second device 112 and the fourth device 116 have adjustable volume. The system may select the second device 112 as a single device because the second device 112 is associated with the first input and is within the user 104's field of view 110.

[0031] In some embodiments, system 100 uses one or more additional inputs to confirm one or more identified devices. For example, a first input may be user 104 looking at a media file displayed on a fourth device 116. The fourth device 116 may highlight the media file in response to the first input. System 100 may receive additional input when user 104 makes a hand gesture (e.g., pinch, swipe, thumbs up, point, etc.). System 100 may use the additional input to confirm that the media file has been selected by the user. System 100 may then receive a voice query (e.g., "send to the TV") that identifies a second device 112. System 100 may then transfer the selected media file to the second device 112 in response to the voice query. In some embodiments, system 100 reformats the media file for optimal display on the second device 112.

[0032] System 100 may identify one or more display elements for a subset of devices. The display elements may be associated with one or more actions of the devices. For example, a display element may be associated with brightness setting control, temperature setting control, volume setting control, media output control, media transfer control, activation control, and / or similar controls. In some embodiments, one or more display elements are associated with a first input. For example, if the first input is a voice command "increase brightness" and the subset of devices includes a second device 112 and a third device 114, the display element may be a brightness setting control for the second device 112 and / or the third device 114. In some embodiments, System 100 stores the display elements in one or more profiles. For example, a control hub may store one or more display elements during a device registration process. In some embodiments, System 100 receives display elements from one or more devices. For example, system 100 may request a first display element associated with a first input from a second device 112, and may request a second display element associated with a first input from a third device 114. The second device 112 may transmit the first display element to the first device 102 for display, and the third device 114 may transmit the second display element to the first device 102 for display.

[0033] In some embodiments, one or more devices display one or more identified display elements for the user. For example, if the first input is the voice command "increase brightness," the device may identify a first display element related to brightness setting control for a second device 112. The first device 102 may then display the brightness setting control for the second device 112 on the display of the first device 102. In some embodiments, the system may display more than one display element. For example, if the first input is the voice command "increase brightness," the device may identify a first display element related to brightness setting control for a second device 112 and a second display element related to brightness setting control for a third device 114. The first device 102 may then display the brightness setting control for the second device 112 and the brightness setting control for the third device 114 on the display of the first device 102.

[0034] The display elements may be selectable by user 104. For example, the first device 102 may display a display element associated with brightness setting control for the third device 114. User 104 may select the display element and adjust the brightness setting of the third device 114. In another embodiment, the first device 102 may display a display element associated with volume setting control for the second device 112. User 104 may select the display element and adjust the volume setting of the second device 112.

[0035] System 100 may cause one or more devices to perform an action in response to a selection by user 104. For example, if user 104's selection corresponds to reducing the brightness of a third device 114, the first device 102 may send a request to the third device 114 to reduce the brightness setting according to user 104's selection. In another embodiment, if user 104's selection corresponds to reducing the brightness of a third device 114, the first device 102 may send a first request to the control hub. The control hub may then send a second request to the third device 114 to reduce the brightness setting according to user 104's selection.

[0036] Figures 2A-2B show a device that displays one or more display elements according to some embodiments of the present disclosure. In some embodiments, the device is the first device 102 from Figure 1. Although the first device 102 is presented as an XR device, other devices may also display one or more display elements. For example, the first device 102 may be a smartwatch, a smartphone, a laptop, and / or similar devices.

[0037] In some embodiments, the first device 102 displays one or more identified display elements for the user. For example, in response to a voice command such as "increase brightness," the first device 102 may identify a first display element 202 related to brightness setting control for a second device 112. The first device 102 may then display the first display element 202 related to brightness setting control for the second device 112, as shown in Figure 2A. In another embodiment, in response to a voice command such as "increase brightness," the first device 102 may identify a first display element 202 related to brightness setting control for the second device 112 and a second display element 204 related to brightness setting control for a third device 114. The first device 102 may then display the first display element 202 and the second display element 204, as shown in Figure 2B.

[0038] In some embodiments, the display elements may be selectable. In some embodiments, the display elements include one or more selectable options. The user may interact with one or more display elements using voice queries, gestures, gaze, and / or similar inputs. For example, the user selection may be a voice query associated with the first display element 202. In another embodiment, the user selection may be a tap on the touch display of the first device 102. In another embodiment, the user selection may be the user gazing at the first display element 202. In another embodiment, the user selection may be the user making a gesture with their hand. In another embodiment, the user selection may be the user blinking and / or moving their eyes. The system 100 may use any of the sensors described herein to detect the selection.

[0039] Figure 3 shows another illustrative diagram of a system for displaying control elements for a device based on one or more inputs, according to some embodiments of the present disclosure. Figure 3 shows an XR device 302. In some embodiments, the XR device 302 is the first device 102 from Figure 1. Although the XR device 302 is shown, other devices may also practice the methodology described herein. For example, the methodology described may be implemented using a smartwatch, smartphone, laptop, and / or similar devices. One of the many functions described in relation to Figures 3-5 is described as being implemented by the XR device 302. Although one of the many functions is described as being implemented by the XR device 302, in some embodiments, one or more other devices in the system may also practice the function.

[0040] In some embodiments, the XR device 302 detects a first input. The first input may be a voice query, gesture, gaze, and / or similar input. For example, the XR device 302 may detect a first input when a user gazes at the first device 304. In another embodiment, the XR device 302 may detect a first input when a user launches an application 306 on the first device 304. The XR device 302 may use one or more sensors to detect a first input. One or more sensors may include a camera, depth sensor, lidar sensor, position sensor, pressure sensor, accelerometer, audio sensor, and / or similar sensors. In some embodiments, the XR device 302 uses one or more other devices (e.g., the second device 112 in Figure 1) to detect an input.

[0041] In some embodiments, the XR device 302 identifies one or more devices associated with a first input. For example, if the first input is a user looking at a first device 304, the first device 304 may be associated with the first input. In another embodiment, if the first input is a user selection of an application 306 on the first device 304, the first device 304 may be associated with the first input. The XR device 302 may use one or more sensors described herein to identify one or more devices associated with the first input.

[0042] The XR device 302 may also identify a first set of attributes associated with the first device 304. For example, the first set of attributes may include brightness, health, notifications, media sharing, and / or similar attributes. In some embodiments, the XR device 302 uses one or more databases to identify the first set of attributes. For example, the first database may store the attributes associated with the device in one or more profiles. In some embodiments, a control hub (e.g., control hub 614) stores device information and / or attributes associated with the devices. In some embodiments, one or more devices communicate with each other and / or with the control hub using one or more wired and / or wireless communication protocols.

[0043] In some embodiments, the XR device 302 detects a second input. For example, the XR device 302 may detect a second input when a user gazes at a second device 308, as shown in Figure 4. The XR device 302 may identify one or more devices associated with the second input. For example, if the second input is a user gazing at the second device 308, the second device 308 may be associated with the second input. In some embodiments, the XR device 302 uses one or more sensors described herein to identify one or more devices associated with the second input. The XR device 302 may also identify a second set of attributes associated with the second device 308. For example, the second set of attributes may include comfort, health, energy, and / or similar attributes. In some embodiments, the XR device 302 uses one or more databases to identify the second set of attributes.

[0044] In some embodiments, the XR device 302 identifies one or more actions based on a first input, a second input, a first set of attributes, and / or a second set of attributes. For example, the XR device 302 may determine that both the first set of attributes and the second set of attributes contain "health". Since both the first set of attributes and the second set of attributes contain "health", the XR device 302 may determine one or more actions related to the second device 308 (e.g., turning on the air conditioning).

[0045] In some embodiments, the XR device 302 uses one or more databases to identify one or more actions associated with one or more devices. For example, the database may include multiple entries that link a device to one or more actions. A first entry may link a first set of actions (e.g., turning on the air conditioning, turning off the air conditioning, turning on the heating, turning off the heating, etc.) to a second device 308. A second entry may link a second set of actions (e.g., increasing the brightness, decreasing the brightness, activating silent mode, transmitting a message, playing a portion of the content, launching an application, setting an alarm, turning off the device, turning on the device, etc.) to a first device 304. An entry may also associate one or more attributes with multiple actions. For example, a first entry may link a first action (e.g., turning on the air conditioning) to a first attribute (e.g., health, temperature, etc.). In another embodiment, the second entry may link the second action (e.g., launching a fitness-related application 306) to the first attribute (e.g., health).

[0046] In some embodiments, the XR device 302 also displays one or more display elements associated with one or more of the identified actions. For example, the XR device 302 may display a first display element 310 in response to a step of identifying a first action (e.g., turning on the air conditioning). In some embodiments, the first display element 310 is selectable. In some embodiments, one or more display elements include one or more selectable options. The user may interact with one or more display elements using voice queries, gestures, gaze, and / or similar inputs. For example, the user selection may be a voice query associated with the first display element 310. In another embodiment, the user selection may be a tap on the touch display of the XR device 302. In another embodiment, the user selection may be the user gazing at the first display element 310. In another embodiment, the user selection may be the user making a gesture with their hand. In another embodiment, the user selection may be the user blinking and / or moving their eyes. The XR device 302 may use any of the sensors described herein to detect the selection. The XR device 302 may cause the second device 308 to perform one or more actions in response to the user interacting with one or more display elements.

[0047] In some embodiments, the XR device 302 displays a first display element 310 based on one or more attributes (e.g., health) of the second device 308 that match one or more attributes (e.g., health) of the first device 304. In some embodiments, different display elements are associated with different attributes. For example, the first display element 310 may be associated with a first attribute (e.g., health), and a second display element (not shown) may be associated with a second attribute (e.g., energy).

[0048] In some embodiments, the XR device 302 causes the second device 308 to perform one or more actions without displaying the first display element 310. For example, the XR device 302 may automatically cause the second device 308 to turn on the air conditioning after detecting a second input. In some embodiments, the XR device 302 automatically causes the second device 308 to perform one or more actions based at least partially on a second input and past user interactions. For example, the XR device 302 may have access to one or more databases that store user interactions. The XR device 302 may use one or more databases and one or more artificial intelligence (AI) models to identify user patterns. For example, the XR device 302 may use the AI ​​models and databases to determine that the user consistently uses the second device 308 and turns on the air conditioning while the user is using an application 306 (e.g., a fitness application) on the first device 304. Therefore, when a user launches application 306 on the first device 304 and then looks at the second device 308, the XR device 302 may automatically turn on the air conditioning on the second device 308 based on past user interactions.

[0049] In some embodiments, the XR device 302 detects a first input when a user gazes at a television (e.g., the second device 112 shown in Figure 1). The XR device 302 may use one or more sensors to detect the first input. The XR device 302 may then identify one or more devices associated with the first input. For example, if the first input is a user gazing at a television, the television may be associated with the first input. The XR device 302 may use one or more sensors to identify the television. The XR device 302 may also identify a first set of attributes associated with the television. For example, the first set of attributes may include brightness, audio, media sharing, and / or similar attributes. In some embodiments, the XR device 302 uses one or more databases to identify the first set of attributes.

[0050] The XR device 302 may then detect a second input. For example, the XR device 302 may detect a second input when a user gazes at a smart light (e.g., the third device 114 shown in Figure 1). The XR device 302 may use one or more sensors to detect the second input. The XR device 302 may then identify one or more devices associated with the second input. For example, if the second input is a user gazing at a light, the light may be associated with the second input. The XR device 302 may use one or more sensors to identify the light. The XR device 302 may also identify a second set of attributes associated with the light. For example, the second set of attributes may include brightness, energy, and / or similar attributes. In some embodiments, the XR device 302 uses one or more databases to identify the second set of attributes.

[0051] The XR device 302 may also identify one or more actions based on a first input, a second input, a first set of attributes, and / or a second set of attributes. For example, the XR device 302 may determine that both the first set of attributes and the second set of attributes include "brightness". Because both the first set of attributes and the second set of attributes include "brightness", the XR device 302 may determine one or more actions (e.g., brightness control related to a light). In some embodiments, the XR device 302 displays one or more display elements related to one or more of the identified actions. For example, the XR device 302 may display a display element in response to identifying a first action (e.g., brightness control). In another embodiment, the XR device 302 may cause another device (e.g., a television) to display a display element in response to identifying a first action (e.g., brightness control). In some embodiments, the XR device 302 causes the light to perform one or more actions (e.g., dimming, dimming, etc.) in response to a user interacting with one or more display elements. In some embodiments, the XR device 302 causes the light to perform one or more actions (e.g., dimming, dimming, etc.) without displaying the display elements. For example, the XR device 302 may automatically cause the light to dim after detecting a second input (e.g., a user looking at the light).

[0052] In some embodiments, the XR device 302 detects a first input when a user issues a voice query (e.g., "transfer"). The XR device 302 may use one or more sensors to detect the first input. The XR device 302 may then identify one or more devices associated with the first input. For example, if a user issues a voice query while looking at a laptop, the laptop may be associated with the first input. The XR device 302 may use one or more sensors to identify the laptop. The XR device 302 may also identify a first set of attributes associated with the laptop. For example, the first set of attributes may include brightness, audio, notifications, media sharing, and / or similar attributes. In some embodiments, the XR device 302 uses one or more databases to identify the first set of attributes.

[0053] The XR device 302 may then detect a second input. For example, the XR device 302 may detect a second input when a user gazes at a smartphone (e.g., the fourth device 116 shown in Figure 1). The XR device 302 may use one or more sensors to detect the second input. The XR device 302 may then identify one or more devices associated with the second input. For example, if the second input is a user gazing at a smartphone, the smartphone may be associated with the second input. The XR device 302 may use one or more sensors to identify the smartphone. The XR device 302 may also identify a second set of attributes associated with the smartphone. For example, the second set of attributes may be brightness, audio, notifications, media sharing, and / or similar attributes. In some embodiments, the XR device 302 uses one or more databases to identify the second set of attributes.

[0054] The XR device 302 may also identify one or more actions based on a first input, a second input, a first set of attributes, and / or a second set of attributes. For example, the XR device 302 may decide to respond to a plurality of actions related to media transfer between identified devices based on a first input (e.g., a voice query "transfer"). A first action of the plurality of actions may relate to identifying a portion of the content to be transferred. The XR device 302 may identify a portion of the content based on the first input. For example, the XR device 302 may identify a portion of the content to be displayed by the laptop within a threshold time when the first input is received. In another embodiment, the XR device 302 may identify a portion of the content to be selected by the user within a threshold time when the first input is received. A second action of the plurality of actions may be to format the identified portion of the content. For example, the XR device 302 may cause the laptop to resize the identified portion of the content according to one or more specifications of the receiving device (e.g., a smartphone). In another embodiment, the XR device 302 may cause the laptop to convert an identified portion of the content from a first format to a second format and facilitate the transfer of that portion of the content to a receiving device (e.g., a smartphone). A third action of the multiple actions may cause the formatted portion of the content to be transferred to the receiving device. For example, the XR device 302 may cause the laptop to transmit the formatted portion of the content to a smartphone. In another embodiment, the XR device 302 may cause the laptop to transmit a link to the formatted portion of the content and / or the identified portion of the content so that the smartphone can download that portion of the content.

[0055] In some embodiments, the XR device 302 displays one or more display elements related to one or more of the identified actions. For example, the XR device 302 may display a display element requesting authorization from a laptop to transfer a portion of media to a smartphone. In another embodiment, the XR device 302 may cause another device (e.g., a laptop, smartphone, etc.) to display a display element in response to identifying one or more actions.

[0056] In some embodiments, the XR device 302 causes the laptop and / or smartphone to perform one or more actions in response to the user interacting with one or more display elements. In some embodiments, the XR device 302 causes the laptop and / or smartphone to perform one or more actions without displaying any display elements. For example, the XR device 302 may automatically detect a second input (e.g., the user looking at the smartphone) and then trigger a series of actions related to media transmission between identified devices.

[0057] Figure 6-8 illustrates exemplary devices, systems, servers, and associated hardware for displaying control elements for a device based on one or more inputs, according to several embodiments.

[0058] Figure 6 shows a system 600 comprising an XR device 601, a smartphone 602, a thermostat 604, a speaker 606, a television 608, a light 610, and a laptop 612, all coupled to a control hub 614. In some embodiments, one or more devices communicate with the control hub 614 using wired and / or wireless communication paths. Seven devices are shown, but any number of devices may be used. For example, more or fewer devices than those shown may be present. In some embodiments, one or more devices are identical to or similar to other devices described herein.

[0059] System 600 may identify one or more devices using one or more registration processes. During the registration process, System 600 may receive device information. System 600 may use the device information to determine attributes for each device. For example, a smartphone 602 may be associated with a first set of attributes (e.g., brightness, volume, media sharing, and / or similar attributes), and a light 610 may be associated with a second attribute (e.g., brightness). System 600 may store the attributes associated with each device in one or more databases. In some embodiments, a control hub 614 stores device information and / or attributes associated with multiple devices. In some embodiments, devices communicate with each other and / or with the control hub 614 using one or more wired and / or wireless communication protocols. For example, multiple devices may communicate over a Wi-Fi network.

[0060] In some embodiments, device information and / or attributes associated with multiple devices are received from multiple devices. For example, a thermostat 604 may store a first Extended Markup Language (XML) file in the thermostat 604's memory, and a television 608 may store a second XML file in the television 608's memory. The thermostat 604 may broadcast the first XML data contained in the first XML file network over a network (e.g., a Wi-Fi network). The television 608 may broadcast the second XML data contained in the second XML file over a network. In some embodiments, the thermostat 604 and / or the television 608 broadcast the XML data using one or more discovery protocols (e.g., Universal Plug and Play (UPnP), Simple Services Discovery Protocol (SSDP), etc.). One or more devices (e.g., a control hub 614) may receive the first XML data and / or the second XML data. One or more devices may then store the first XML data and / or the second XML data in one or more databases. The stored XML data may be used to generate one or more entries as described herein.

[0061] In some embodiments, device information includes one or more actions associated with one or more devices. System 600 may store one or more actions associated with one or more devices in one or more databases. One or more databases may include multiple entries that link a device to one or more actions. For example, an entry may link a thermostat 604 to a first set of actions (e.g., turning on the air conditioner, turning off the air conditioner, turning on the heating, turning off the heating, etc.). In another embodiment, an entry may link a smartphone 602 to a second set of actions (e.g., increasing brightness, decreasing brightness, activating silent mode, sending a message, playing a portion of content, launching an application, setting an alarm, turning off the device, turning on the device, etc.). An entry may also associate one or more actions with one or more attributes. For example, an entry may link a first action (e.g., turning on the air conditioner) with a first attribute (e.g., health). In another embodiment, the entry may link a second action (e.g., launching a fitness-related application) to a first attribute (e.g., health).

[0062] In some embodiments, one or more databases described herein are data structures. In some embodiments, the data structure includes device information and / or similar information. The data structure may store one or more device identifiers. The device identifier may be a unique identifier for each device. For example, the device identifier associated with XR device 601 may be XR_01, and the device identifier associated with television may be TV_01. The data structure may also store a plurality of attributes relating to one or more devices. For example, the data structure may store a first set of attributes associated with smartphone 602 (e.g., brightness, volume, media sharing, and / or similar attributes), and a second set of attributes associated with light 610 (e.g., brightness, energy, and / or similar attributes). The data structure may also store user interactions associated with one or more devices. For example, the data structure may store one or more inputs associated with one or more devices (e.g., how often a user turns off light 610 when turning on television 608). The data structure may also store user preferences associated with one or more devices. For example, the data structure may store a first preference corresponding to a temperature setting for a thermostat 604, and a second preference corresponding to a brightness setting for a light 610. The data structure may also store one or more actions associated with one or more devices. For example, the data structure may store one or more actions associated with a thermostat 604 (e.g., turning on the air conditioning, turning off the air conditioning, turning on the heating, turning off the heating, etc.). In some embodiments, the data structure is lightweight and optimized for efficient storage on one or more devices and / or cloud services. In some embodiments, the data structure is distributed across one or more memories, control hubs, and / or cloud servers.

[0063] In system 700, there may be more than one user device 702, but in Figure 7, only one is shown to avoid overcomplicating the drawing. In addition, a user may utilize more than one type of user device 702 and more than one of each type of user device. As described above, user device 702 may be an XR device, a smartphone, a laptop computer, a desktop computer, a television, a speaker, and / or any similar such device. In some embodiments, paths may exist between user device devices so that the devices can communicate directly with each other via a communication path and other short-range two-point communication paths such as a USB cable, an IEEE 1394 cable, a wireless path (e.g., Bluetooth®, infrared, IEEE 802-11x, etc.), or other short-range communication via a wired or wireless path. In some embodiments, user device devices may also communicate directly with each other via an indirect path through a communication network 706.

[0064] The user equipment device 702, media content source 712, and server 714 may be coupled to the communication network 706. That is, the user equipment device 702 is coupled to the communication network 706 via a first communication path 704, the media content source 712 is coupled to the communication network 706 via a second communication path 708, and the server 714 is coupled to the communication network 706 via a third communication path 710. The communication network 706 may be one or more networks, including the Internet, a mobile phone network, a mobile voice or data network (e.g., a 4G, 5G, or LTE network), a cable network, a public switched telephone network, or other types of communication networks or combinations of communication networks. One or more communication paths may, separately or together with other paths, include one or more communication paths such as a satellite path, a fiber optic path, a cable path, a path supporting Internet communications (e.g., IPTV), a free-space connection (e.g., for broadcast or other radio signals), or any other suitable wired or wireless communication path or combination of such paths. In some embodiments, one or more communication paths may be wireless. Communication between devices can be provided by one or more communication paths, but in Figure 7, it is shown as a single path to avoid unnecessarily complicating the diagram.

[0065] The media content source 712 and server 714 can be linked to any number of databases that provide information to user equipment devices. For example, the media content source 712 and / or server 714 may have access to device attributes, display elements, and / or similar information. The media content source 712 represents any computer-accessible content source, such as a storage device for media content, audio content, metadata, and / or similar information. The media content source 712 and / or server 714 may store and / or run various software modules to enable the display of control elements for a device based on the functionality of one or more inputs. In system 700, there may be more than one media content source 712 and / or server 714, but in Figure 7, only one of each is shown to avoid overcomplicating the diagram. In addition, system 700 may utilize more than one type of media content source 712 and / or server 714, and more than one of each type of media content source 712 and / or server 714.

[0066] Figure 8 shows a generalized embodiment of the user equipment device 800 according to several embodiments. In some embodiments, the user equipment device 800 is an embodiment of the user equipment device (e.g., the first device 102) described in Figure 1-7. The user equipment device 800 may receive content and data via an input / output (I / O) path 802. The I / O path 802 may provide audio content (e.g., broadcast programming, on-demand programming, Internet content, content available via a local area network (LAN) or wide area network (WAN), and / or other content) and data to a control circuit network 804, which includes a processing circuit 806 and a storage device 808. The control circuit network 804 may use the I / O path 802 to send and receive commands, requests, and other suitable data. The I / O path 802 may connect the control circuit network 804 (specifically, the processing circuit 806) to one or more communication paths. I / O functionality can be provided by one or more of these communication paths, but in Figure 8, it is shown as a single path to avoid overcomplicating the diagram.

[0067] The control circuit network 804 may be based on any suitable processing network, such as the processing network 806. As referred to herein, the processing network should be understood to mean a network based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays ("FPGA"), application-specific integrated circuits ("ASIC"), etc., and may include multi-core processors (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputers. In some embodiments, the processing network may be distributed across multiple separate processors or processing units, for example, multiple processing units of the same type (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor). The representation of a control element for a device based on the functionality of one or more inputs can be implemented at least in part using the control circuit network 804. The representation of a control element for a device based on the functionality of one or more inputs described herein may be implemented in or supported by any suitable software, hardware, or combination thereof.

[0068] In a client-server based embodiment, the control circuit network 804 may include a communication network suitable for communicating with one or more servers, which can implement at least a described representation of control elements for a device based on the functionality of one or more inputs. Instructions for performing the functionality described above may be stored on one or more servers. The communication network may include a cable modem, an Integrated Services Digital Network ("ISDN") modem, a Digital Subscriber Line ("DSL") modem, a telephone modem, an Ethernet® card, or a wireless modem for communication with other equipment, or any other suitable communication network. Such communication may involve the Internet or any other suitable communication network or route. In addition, the communication network may include a network that enables peer-to-peer communication between user equipment devices or communication between user equipment devices at remote locations (as described in more detail below).

[0069] The memory may be an electronic storage device provided as a storage device 808, which is part of the control circuit network 804. As used herein, the terms “electronic storage device” or “storage device” should be understood to mean any device for storing electronic data, computer software, or firmware, such as random access memory, read-only memory, hard drives, optical drives, digital video disc ("DVD") recorders, compact disc ("CD") recorders, Blu-ray® disc ("BD") recorders, Blu-ray® 3D disc recorders, digital video recorders (sometimes called personal video recorders, i.e., “PVRs,” or “DVRs”), solid-state devices, quantum memory devices, game consoles, game media, or any other suitable fixed or removable storage device, and / or any combination thereof. The storage device 808 may be used to store various types of content as described herein. Non-volatile memory may also be used (for example, to start up boot routines and other instructions). In some embodiments, cloud-based storage may also be used to complement or in place of storage 808.

[0070] The control circuit network 804 may include a video generation network such as one or more analog tuners, one or more MPEG-2 decoders, or MPEG-2 decoders or multiple decoders, or HEVC decoders, or any other suitable digital decoding network, a high-resolution tuner, or any other suitable tuning or video circuit, or a combination of such circuits, and a tuning network. An encoding network (for example, for converting terrestrial, analog, or digital signals to MPEG or HEVC or any other suitable signal for storage) may also be provided. The control circuit network 804 may also include a scaler network for converting content up and down to a preferred output format. The control circuit network 804 may also include a digital-to-analog converter network and an analog-to-digital converter network for converting between digital and analog signals. The tuning and encoding networks may be used by the user equipment device 800 to receive and display, play back, or record content. The tuning and encoding networks may also be used to receive video communication session data. For example, the circuits described herein, including tuning, video generation, encoding, decoding, encryption, decryption, scaling, and analog / digital circuits, may be implemented using software running on one or more general-purpose or specialized processors. Multiple tuners may be provided to handle simultaneous tuning functions (e.g., viewing and recording functions, picture-in-picture (PIP) functions, multiple tuner recording, etc.). If the storage device 808 is provided as a separate device from the user equipment device 800, the tuning and encoding circuits (including multiple tuners) may be associated with the storage device 808.

[0071] The user may issue commands to the control circuit network 804, which are received by the microphone 816. The microphone 816 may be any microphone (or a set of microphones) capable of detecting human speech. The microphone 816 is connected to the processing circuit network 806 and transmits the detected voice commands and other utterances to it for processing. In some embodiments, a voice assistant (e.g., Siri, Alexa, Google Home, and similar voice assistants) receives and processes the voice commands and other utterances.

[0072] The user equipment device 800 may optionally include an interface 810. The interface 810 may be any suitable user interface, such as a remote control device, mouse, trackball, keypad, keyboard, touchscreen, touchpad, stylus input, joystick, or other user input interface. The display 812 may be provided as a standalone device or integrated with other elements of the user equipment device 800. For example, the display 812 may be a touchscreen or touch-sensitive display. In such a scenario, the interface 810 may be integrated with or combined with a microphone 816. If the interface 810 is configured with a screen, such a screen may be one or more of the following: a monitor, television, liquid crystal display ("LCD") for mobile devices, active matrix display, cathode ray tube display, light-emitting diode display, organic light-emitting diode display, quantum dot display, or any other suitable device for displaying visual images. In some embodiments, the interface 810 may be HDTV-compatible. In some embodiments, the display 812 may be a 3D display. The speaker 814 may be controlled by the control circuit network 804. The speaker (or multiple speakers) 814 may be provided as integrated with other elements of the user equipment device 800, or it may be a standalone unit. In some embodiments, the display 812 may output through the speaker 814.

[0073] In some embodiments, the user device 800 may optionally include sensors 818. Only one sensor 818 is shown, but any number of sensors may be used. In some embodiments, the sensor 818 is a camera, depth sensor, lidar sensor, position sensor, pressure sensor, accelerometer, audio sensor, and / or any similar such sensor. In some embodiments, the sensor 818 of the user device 800 (e.g., an image sensor or camera) may capture the real-world environment around the user device 800. The user device 800 may then render the captured real-world scene on the display 812. The user device 800 may display virtual or augmented objects that will be displayed on the display 812, thereby generating to extend the real-world scene to be visible on the display 812.

[0074] Figure 9 is an illustrative flowchart of process 900 for displaying control elements for a device based on one or more inputs, according to some embodiments of the present disclosure. Process 900 and any of the following processes may be executed by a control circuit network 804 on a user equipment device 800 and / or by a control circuit network on one or more servers. In some embodiments, the control circuit network may be part of a remote server that is isolated from the user equipment device 800 via a communication network, or distributed via a combination of both. In some embodiments, instructions for executing process 900 and any of the following processes may be encoded on a non-transient storage medium (e.g., storage device 808) as a set of instructions to be decoded and executed by a processing network (e.g., processing network 806). The processing network may, in turn, provide instructions to other subcircuits included within the control circuit network, such as encoding, decoding, encryption, decryption, scaling, analog / digital conversion networks, and equivalents. Note that any of the processes or any step thereof may be performed on or provided by any of the devices described in Figure 1-8. The process is illustrated and described as a sequence of steps, but it is assumed that various embodiments of the process may be performed in any order or combination and that it is not necessary to include all of the illustrated steps. In addition, several preferred steps may be implemented in parallel for further purposes of the disclosure. For example, several preferred steps may be performed in any order, in parallel, or substantially simultaneously to reduce the delay of the system or method or to increase its speed.

[0075] In 902, the control network detects a first input. In some embodiments, the first input may be a voice query, gesture, gaze, and / or similar input. For example, the first input may be a voice query (e.g., "increase brightness"). In another embodiment, the first input may be a tap on a touch display. In another embodiment, the first input may be the user gazing at one or more devices. In another embodiment, the first input may be the user pointing at one or more devices. In another embodiment, the first input may be the user blinking and / or moving their eyes. The control network may use one or more sensors to detect the first input. One or more sensors may include a camera, depth sensor, lidar sensor, position sensor, pressure sensor, accelerometer, audio sensor, and / or similar sensors.

[0076] In 904, the control network determines that a first input corresponds to a first attribute. In some embodiments, the control network uses one or more machine learning models to determine the attribute associated with the first input. In some embodiments, the control network uses one or more natural language processing (NLP) models to determine the attribute associated with the first input. For example, the control network may use NLP to determine that a voice query (e.g., "increase brightness") corresponds to a first attribute (e.g., brightness). In some embodiments, the control network has access to a database that links inputs to attributes. For example, a first input (e.g., a user staring at a lamp) may correspond to a first attribute (e.g., brightness). In another embodiment, a second input (e.g., a user making a gesture on a laptop) may correspond to a second attribute (e.g., media sharing).

[0077] In 906, the control network detects the user's orientation when a first input is detected. For example, the control network may determine that the user is wearing an XR device and that the XR device detects the user's head orientation when a first input is detected. In another embodiment, one or more cameras may capture one or more images of the user when a first input is detected. The control network may use one or more image detection models to process one or more images of the user and determine the user's head orientation when a first input is detected. In another embodiment, one or more cameras may capture one or more images of the user when a first input is detected. The control network may use one or more image detection models to process one or more images of the user and determine the location of one or more of the user's eyes and / or pupils. In some embodiments, the user's orientation is detected within a threshold (e.g., 1 second, 5 seconds, 30 seconds, etc.) when a first input is detected.

[0078] In 908, the control network calculates a first field of view based at least partially on the user's orientation. In some embodiments, the control network uses one of the sensors and / or models described herein to detect and / or determine the user's first field of view.

[0079] In 910, the control network identifies a plurality of devices connected to the network, and each of the plurality of devices is associated with at least one attribute. In some embodiments, the network is a home network, and the plurality of devices are smart devices connected to the home network. In some embodiments, the control network identifies each of the plurality of devices using one or more registration processes. During the registration process, the control network may receive device information. The control network may use the device information to determine attributes and / or actions associated with each device. For example, a first device may be associated with a first set of attributes (e.g., brightness, volume, media sharing, and / or similar attributes), and a second device may be associated with a second attribute (e.g., brightness). The control network may store the attributes and / or actions associated with each device in one or more databases. In some embodiments, the control network communicates with one or more of the plurality of devices using one or more wired and / or wireless communication protocols.

[0080] In 912, the control network selects a set of devices from a plurality of devices based at least partially on each device in a set of devices associated with a first attribute. In some embodiments, the control network uses one or more attributes of devices to determine one or more devices associated with a first input. For example, the first input may be a voice query, “increase brightness,” and the first attribute may be “brightness.” The control network may identify a first device (e.g., a television), a second device (e.g., a light), and a third device (e.g., a smartphone) as a set of devices, since each device in the set of devices is associated with an attribute that is identical to or similar to the first attribute (e.g., brightness). In another embodiment, the first input may be a voice query, “decrease volume,” and the first attribute may be “volume.” The control circuit network may identify a first device (e.g., a television) and a third device (e.g., a smartphone) as a set of devices, since each device in the set of devices is associated with an attribute that is identical to or similar to a first attribute (e.g., volume).

[0081] In step 914, the control network selects a subset of devices from a set of devices based at least partially on each device in a subset of devices located within a first field of view. For example, the first input may be a voice query such as "increase brightness," and the first attribute may be "brightness." The control network may identify a first device (e.g., a television), a second device (e.g., a light), and a third device (e.g., a smartphone) as a set of devices, since each device in the set of devices is associated with an attribute that is identical to or similar to the first attribute (e.g., brightness). The control network may then select the first and second devices as a subset of devices because both are within the field of view calculated in step 908. In some embodiments, the control network selects a single device. For example, the first input may be a voice query such as "decrease volume," and the first attribute may be "volume." The control network may identify a first device (e.g., a television) and a third device (e.g., a smartphone) as a set of devices, since each device in the set of devices is associated with an attribute that is identical to or similar to a first attribute (e.g., volume). The control network may then select only the first device because the first device is within the field of view calculated in step 908 and the third device is not within the field of view.

[0082] In 916, the control circuit network identifies a first display element associated with a first device of a subset of devices. The first display element may be associated with one or more actions of the first device. For example, the display element may be associated with brightness setting control, temperature setting control, volume setting control, media output control, media transfer control, activation control, and / or similar controls. In some embodiments, one or more display elements are associated with a first attribute of step 904. For example, the first input may be a voice query such as “increase brightness,” and the first attribute may be “brightness.” The display element may be a brightness setting control for the first device, since the brightness setting control corresponds to the first attribute.

[0083] In 918, the control network displays the first display element. In some embodiments, the control network accesses the first display element from one or more databases. For example, device information received and stored in a database during one or more device registration processes may include the first display element. In some embodiments, the control network receives the first display element from one or more devices. For example, the control network may request the first display element from a first device, and the first device transmits the first display element to the control network for display. In another embodiment, the control network may request the first display element from a first device, and the first device transmits the first display element to an additional device for display.

[0084] In some embodiments, the first display element and / or a portion of the display element is selectable. For example, a control circuit network may display the first display element (e.g., brightness setting control for a first device). A user may select the first display element and adjust the brightness setting of the first device.

[0085] The control circuit network may cause one or more devices to perform an action in response to the selection of a first display element. For example, if the selection of the first display element corresponds to decreasing the brightness of a first device, the control circuit network may send a request to the first device to decrease the brightness setting according to the selection of the first display element. In another embodiment, if the selection of the first display element corresponds to decreasing the brightness of a first device, the control circuit network may send a first request to an additional device (e.g., a control hub). The additional device may then send a second request to the first device to decrease the brightness setting according to the selection of the first display element.

[0086] FIG. 10 is an illustrative flowchart of 1000 of a process for displaying control elements for a device based on one or more inputs, according to some embodiments of the present disclosure.

[0087] At 1002, the control circuit network detects a first input. In some embodiments, the control circuit network uses the same or similar methodology described at step 902 to detect the first input.

[0088] At 1004, the control circuit network identifies a first device, at least in part, based on the first input. For example, if the first input is that the user gazes at the first device, the first device may be associated with the first input. In another example, if the first input is a user selection of an application on the first device, the first device may be associated with the first input. The control circuit network may use one or more sensors described herein to identify one or more devices associated with the first input.

[0089] In 1006, the control network identifies a first set of attributes associated with a first device. For example, the first device may be a smartphone, and the first set of attributes may include brightness, health, notifications, media sharing, and / or similar attributes. In some embodiments, the control network uses one or more databases to identify the first set of attributes. For example, the databases may store the attributes associated with the device in one or more profiles. In some embodiments, an additional device (e.g., a control hub) stores device information and / or attributes associated with the device, and the control network requests the first set of attributes from the additional device.

[0090] In step 1008, the control network detects a second input. In some embodiments, the control network detects a second input using the same or similar methodology described in step 902.

[0091] In step 1010, the control network identifies the second device, at least partially, based on the second input. In some embodiments, the control network identifies the second device, at least partially, based on the second input, using the same or similar methodology described in step 1004.

[0092] In 1012, the control network identifies a second set of attributes and a number of actions associated with a second device. For example, the second device may be a thermostat, and the second set of attributes may include comfort, health, energy, and / or similar attributes. In some embodiments, the control network uses one or more databases to identify the second set of attributes. For example, the databases may store the attributes associated with the device in one or more profiles. In some embodiments, an additional device (e.g., a control hub) stores device information and / or attributes associated with the device, and the control network requests the second set of attributes from the additional device.

[0093] In 1014, the control network selects a first attribute, the first set of attributes includes the first attribute, and the second set of attributes includes the first attribute. For example, the control network may decide that both the first set of attributes and the second set of attributes include "health". In some embodiments, the control network also uses a first input and / or a second input to select the first input. For example, if the first set of attributes and the second set of attributes have more than one attribute in common, the control network may use the first input and / or the second input to select the first attribute of the multiple common attributes.

[0094] In 1016, the control network identifies a first action of a plurality of actions based at least partially on a first attribute. For example, if both a first set of attributes and a second set of attributes include "health", the control network may determine one or more actions (e.g., turning on the air conditioning) associated with a first device. In some embodiments, the control network uses one or more databases to identify one or more actions associated with one or more devices. For example, the databases may contain multiple entries that link a device to one or more actions. A first entry may link a first plurality of actions (e.g., increasing brightness, decreasing brightness, activating silent mode, transmitting a message, playing a portion of content, launching an application, setting an alarm, turning the device off, turning the device on, etc.) to a first device. A second entry may link a second plurality of actions (e.g., turning on the air conditioning, turning off the air conditioning, turning on the heating, turning off the heating, etc.) to a second device. An entry may also associate one or more attributes with a plurality of actions. For example, the first entry may link a first action (e.g., launching a fitness-related application) to a first attribute (e.g., health). In another embodiment, the second entry may link a second action (e.g., turning on the air conditioning) to a first attribute (e.g., health).

[0095] In 1018, the control network causes the second device to perform a first action. For example, the control network may automatically cause the second device (e.g., a thermostat) to turn on the air conditioning after detecting a second input (e.g., looking at the thermostat). In some embodiments, the control network automatically causes the second device to perform one or more actions based at least partially on the second input and past user interactions. For example, the control network may have access to one or more databases that store user interactions. Using the databases, the control network may determine that the user has consistently used the second device and turned on the air conditioning after a first input (e.g., launching a fitness application on the first device). Thus, when the control network detects the first input (e.g., the first device launching a fitness application) and the second input (e.g., the user looking at the second device), the control network may automatically cause the second device to perform a first action (e.g., turning on the air conditioning) based on past user interactions.

[0096] In some embodiments, the control network causes a second device to perform a first action in response to interaction with one or more indicator elements. For example, the control network may display an indicator element corresponding to the first action (e.g., turning on the air conditioning). When the control network receives a selection of an indicator element, it may cause the second device to turn on the air conditioning. In some embodiments, the control network may cause the second device to perform the first action by transmitting a request to the second device. For example, the control network may send a request to the second device to turn on the air conditioning. In some embodiments, the control network may cause the second device to perform the first action by transmitting a request to an additional device. For example, the control network may send a first request to an additional device (e.g., a control hub) which indicates a request for the second device to turn on the air conditioning. The additional device may then send a second request to the second device to turn on the air conditioning in accordance with the first request.

[0097] Figure 11 is an illustrative flowchart of a process 1100 for displaying a notification based on one or more inputs, according to some embodiments of the present disclosure.

[0098] In 1102, the control network receives a first application and a first set of attributes associated with the first application. In some embodiments, the control network receives the first application when it is downloaded onto the storage device (e.g., storage device 808) of a device (e.g., an XR device, a smartphone, a laptop, etc.). For example, a smartphone may download a fitness application hosted on an application store (e.g., Apple's App Store). The control network may also receive a first set of attributes associated with the first application. For example, the set of attributes associated with a fitness application may include health, entertainment, and / or similar attributes. In another embodiment, the set of attributes associated with a music playlist application (e.g., Spotify, Apple Music, YouTube® Music, Pandora, etc.) may include relaxation, health, entertainment, and / or similar attributes. The first set of attributes may be received in addition to the first application and / or determined by the control network.

[0099] In some embodiments, one or more applications are associated with one or more configuration files. For example, a first application may be associated with a first configuration file. The first configuration file may contain metadata related to the first application. For example, the metadata may indicate a first set of attributes associated with the first application, application compatibility associated with the first application, and / or similar application information. In some embodiments, a control network uses the application information to determine compatibility between one or more applications. For example, the application information includes a uniform resource locator (URL) that links compatible applications to the first application. The control network may use the URL to identify applications compatible with the first application. In another embodiment, the application information includes an application extension that links compatible applications to the first application. The control network may use the application extension to identify applications compatible with the first application. In yet another embodiment, the application information includes one or more universal links that link compatible applications to the first application. The control network may use one or more universal links to identify applications compatible with the first application.

[0100] In 1104, the control network stores a first set of attributes. In some embodiments, the control network stores the first set of attributes in one or more databases. For example, a database may store attributes associated with one or more applications and / or devices. In some embodiments, one or more databases include multiple entries that associate attributes with applications. For example, a first entry may associate the health attribute with a fitness application, a music playlist application, and a thermostat application. In another embodiment, a first entry may associate the sleep attribute with a light control application, an email application, a messaging application, and a thermostat application. The control network may update one or more databases when the control network receives additional applications and / or attributes.

[0101] In 1106, the control network detects a first input. In some embodiments, the first input may be a voice query, gesture, gaze, and / or similar input. For example, the first input may be a user selecting a first application using a touchscreen display. In another embodiment, the first input may be a user selecting a first application using a mouse. In another embodiment, the first input may be a voice query (e.g., "open fitness application"). In another embodiment, the first input may be a tap on a touch display. In another embodiment, the first input may be a user gazing at one or more devices and / or applications. In another embodiment, the first input may be a user pointing at one or more devices and / or applications. In another embodiment, the first input may be a user blinking and / or moving their eyes. The control network may use one or more sensors to detect the first input. One or more sensors may include a camera, depth sensor, lidar sensor, position sensor, pressure sensor, accelerometer, audio sensor, and / or similar sensors.

[0102] In 1108, the control network identifies a first application based at least partially on a first input. For example, if the first input is a user selecting a first application using a touchscreen display, the first application may be associated with the first input. In another embodiment, if the first input is a user gazing at a first application, the first application may be associated with the first input. The control network may use one or more sensors described herein to identify one or more applications associated with the first input.

[0103] In 1110, the control network selects a first attribute of a first set of attributes associated with a first application. In 1112, the control network identifies a first entry of a plurality of entries, the first entry associating the first attribute with the first application and one or more additional applications. For example, the first set of attributes associated with a fitness application may include health, entertainment, and / or similar attributes. The control network may select "health" as the first attribute. The control network may then identify that the first entry associates the first attribute (health) with a first application (e.g., a fitness application), a second application (e.g., a music playlist application), and a third application (e.g., a thermostat application).

[0104] In 1114, the control network identifies a second application of one or more additional applications based at least partially on the step of identifying a first entry. For example, one or more additional applications associated with the first entry may be a second application (e.g., a music playlist application) and / or a third application (e.g., a thermostat application). The control network may select the second application (e.g., a music playlist application).

[0105] In step 1116, the control network identifies a first action of a plurality of actions, at least partially, based on a first attribute, and the plurality of actions are associated with a second application. In some embodiments, the control network identifies a first action when a second application transmits the first action to the control network. For example, the control network may issue a first prompt to one or more applications (including a second application) identified in step 1112. The first prompt may indicate that an input related to a first attribute (e.g., health) has been detected. The second application may then issue a second prompt indicating a first action (e.g., playing a training playlist). In some embodiments, the control network receives a plurality of prompts from applications that receive a first prompt from the control network.

[0106] In some embodiments, the control network uses one or more databases to identify one or more actions associated with a second application. For example, the database may include multiple entries that link the application to one or more actions. A first entry may link a second application to a first group of actions (e.g., playing a playlist, increasing the volume, decreasing the volume, skipping a song, and / or similar actions). The entries may also associate one or more attributes with the group of actions. For example, a first entry may link a first action (e.g., playing a training playlist) to a first attribute (e.g., health, temperature, weather, etc.). In another embodiment, a second entry may link a second action (e.g., turning on the air conditioning) to a first attribute (e.g., health). The control network may use the database and / or the first attribute to identify the first action (e.g., playing a training playlist).

[0107] In 1118, the control network displays a notification, which corresponds to a first action. In some embodiments, the notification and / or part of the notification is selectable. For example, the control network may display a first notification (e.g., "Do you want me to play your workout playlist?"). In response to the user selecting the first notification, the control network may cause a second application (e.g., a music playlist application) to perform the first action (e.g., playing the workout playlist). In some embodiments, the control network receives a notification from the second application.

[0108] In some embodiments, the control network causes more than one notification to be displayed. In some embodiments, the notification is associated with more than one application. For example, the control network may identify multiple applications that are associated with the first attribute in step 1114. The control network may transmit the first prompt to multiple applications. The first prompt may indicate that input related to the first attribute (e.g., health or fitness) has been detected. A second application (e.g., a music playlist application) may issue a second prompt indicating a first action (e.g., playing a workout playlist), and a third application (e.g., a thermostat application) may issue a third prompt indicating a second action (e.g., turning on the air conditioning). The control circuit network may then display both a first notification (e.g., "Do you want me to play your workout playlist?") and a second notification (e.g., "Do you want me to turn on the air conditioner?", "Do you want me to adjust the temperature to 78 degrees?", etc.). The second notification may require a response from the user, such as "78 degrees". In some embodiments, the prompt is conditional and associated with a criterion that must be satisfied before the action associated with the prompt can be performed. An embodiment with a conditional prompt may include "should I play your workout playlist when you start working out?". The condition in this scenario is starting a workout session.For example, when a fitness application determines that a user has started a workout (e.g., based on receiving such a signal or information from a smart treadmill), the control circuitry can notify the relevant application of the start of such an activity.

[0109] In some embodiments, the control circuitry uses one or more AI models and / or stored user interactions to present notifications and / or surface applications. Surfacing an application can relate to presenting application information, rearranging the applications to be displayed, prioritizing applications (e.g., during a search), and / or similar such functionality. The control circuitry may have access to one or more databases that store user interactions. The control circuitry may use one or more databases and AI models to identify user patterns. For example, the control circuitry may use an AI model and a database to determine that a user consistently turns on the vehicle's headlights at night. Thus, when the user gazes at the display inside the vehicle at night, the control circuitry may automatically cause a first notification (e.g., "Do you want me to turn on the headlights?") to be displayed by the display. In response to the user selecting the notification, the control circuitry may cause the vehicle's headlights to be turned on low beam.

[0110] The processes discussed above are illustrative and not intended to be limiting. Those skilled in the art will understand that the steps of the processes discussed herein may be omitted, modified, combined, and / or rearranged, and that any additional steps may be performed without departing from the scope of the invention. More generally, the above disclosures are illustrative and not intended to be limiting. Only the subsequent claims are intended to define the boundaries with respect to what the invention encompasses. Furthermore, it should be noted that features and limitations described within any one embodiment may also apply to any other embodiment herein, and that flowcharts or examples relating to one embodiment may be performed in different orders or in parallel, and may be combined with any other embodiment in a preferred manner. In addition, the systems and methods described herein may be performed in real time. Also, it should be noted that the systems and / or methods described above may be applied to or used in accordance with other systems and / or methods.

Claims

1. It is a method, Receiving a first application and a first set of attributes associated with the first application, To store the first set of the aforementioned attributes, To detect the first input, Identifying the first application based at least in part on the first input, Selecting a first attribute from a first set of attributes associated with the first application, Identifying the first entry among multiple entries, Each of the aforementioned entries associates one or more attributes with one or more applications. The first entry associates the first attribute with the first application and one or more additional applications, Identifying a second application among one or more additional applications, at least in part, based on identifying the first entry among the plurality of entries, wherein the second application is associated with a plurality of actions, Identifying a first action among the plurality of actions based at least in part on the first attribute, Displaying a first notification, wherein the first notification corresponds to the first action. Methods that include...

2. Receiving the selection of the first notification mentioned above, To cause the device to perform the first action based at least in part on the selection of the first notification The method according to claim 1, further comprising:

3. The method according to claim 1, wherein the first notification is displayed on a wearable extended reality (XR) device.

4. The method according to claim 1, wherein the first set of attributes and the first application are stored in the memory of the device.

5. The aforementioned device is a wearable extended reality (XR) device, The method according to claim 4, wherein the device displays the first notification.

6. Transmitting a first prompt associated with the first attribute to the second application, Receiving a second prompt associated with the first action from the second application: The method according to claim 1, further comprising:

7. Identifying a third application among the one or more additional applications, at least in part on identifying the first entry among the plurality of entries, wherein the third application is associated with the second plurality of actions, Identifying the second action among the second plurality of actions based at least in part on the first attribute, Displaying a second notice, wherein the second notice corresponds to the second action. The method according to claim 1, further comprising:

8. The method according to claim 7, wherein the first notice and the second notice are displayed at the same time.

9. The method according to claim 1, wherein the first input is at least one of the user's gaze or a tap on a touchscreen.

10. It is a device, Control circuit network and A memory containing computer program code relating to one or more programs, wherein the at least one memory and the computer program code are transmitted to the device using the control circuit network, at least the following: Receiving a first application and a first set of attributes associated with the first application, To store the first set of the aforementioned attributes, To detect the first input, Identifying the first application based at least in part on the first input, Selecting a first attribute from a first set of attributes associated with the first application, Identifying the first entry among multiple entries, Each of the aforementioned entries associates one or more attributes with one or more applications. The first entry associates the first attribute with the first application and one or more additional applications, Identifying a second application among one or more additional applications, at least in part, based on identifying the first entry among the plurality of entries, wherein the second application is associated with a plurality of actions, Identifying a first action among the plurality of actions based at least in part on the first attribute, Displaying a first notification, wherein the first notification corresponds to the first action. A memory and A device equipped with the following features.

Images