Method and apparatus for managing displayed content on a wearable device using a session graph

Abstract

According to one aspect, a method includes: detecting facial features of an entity by at least one imaging sensor of a wearable device; detecting interactive communication between a user of the wearable device and the entity based at least on image data from the at least one imaging sensor; updating a conversation graph in response to detecting the interactive communication between the user and the entity; and managing content for display on the wearable device based on the conversation graph.

Description

Technical Field

[0001] The present disclosure relates to using a session graph to manage display content on a wearable device. Background Art

[0002] Unless the number and type are appropriate, notifications can feel disruptive on wearable devices such as smart glasses, which visually display notifications overlapping the user's view of the world. The user's attention is a limited resource, so it is the device's responsibility to ensure that the user is exposed to relevant and meaningful notifications. Similar to email, sending notifications to users is effectively free, so as the number of apps on a device increases, the number of unwanted or less frequent user notifications also increases. Summary of the invention

[0003] According to one aspect, a method includes: detecting facial features of an entity by at least one imaging sensor of a wearable device; detecting interactive communication between a user of the wearable device and the entity based at least on image data from the at least one imaging sensor; updating a conversation graph in response to detecting interactive communication between the user and the entity; and managing content for display on the wearable device based on the conversation graph.

[0004] According to some aspects, the method may include one or more (e.g., all) of the following features (or any combination thereof). The method may include: receiving a voice command from a user via a microphone of a wearable device; identifying an entity associated with the voice command based on a conversation graph; and performing an action using the identified entity. In some examples, the method may include: determining that a facial feature corresponds to an existing contact associated with a user of the wearable device. The method may include: obtaining first image data by a first imaging sensor; determining whether a facial feature is detected in the first image data; and obtaining second image data by a second imaging sensor in response to the facial feature being determined to be detected in the first image data, wherein the second image data has a higher resolution than the first image data. The second image data is used to determine whether the facial feature corresponds to an existing contact. Interaction communication is detected based on the image data and audio data received via a microphone of the wearable device. The conversation graph includes a plurality of nodes including a first node and a second node, the first node representing a user of the wearable device, the second node representing an entity, and the conversation graph including a link connected between the first node and the second node. The method may include: calculating a familiarity metric based at least on a frequency of detected interaction communication between the user and the entity; and annotating the link or the second node with the familiarity metric. The method may include: adjusting a familiarity metric in response to detecting an interactive communication between a user and an entity. The method may include: creating a new node in a conversation graph in response to detecting an interactive communication between a user and an entity, the new node representing the entity; and creating a new link in the conversation graph between the new node and a first node, the new link or new node being annotated with a familiarity metric representing a familiarity level between the user and the entity. The method may include: detecting an entity cluster group among a plurality of entities based on information included in the conversation graph; identifying that content is associated with the entity; and determining whether to display the content on a wearable device based on whether the entity associated with the content is included in the entity cluster group. The method may include: detecting a notification generated by an application executable by the wearable device; assigning a priority to the notification based on the conversation graph; and determining whether to display the notification on the wearable device based on the assigned priority. The method may include: determining that the notification is related to an entity included in the conversation graph, wherein assigning includes assigning a priority to the notification that is higher than a priority assigned to a notification related to an entity not included in the conversation graph. The method may include: receiving information from a user via an application executable by the wearable device; identifying at least one entity from a conversation graph based on the information; and providing the at least one entity as a suggested entity for display on an interface of the application. The method may include: determining a ranking of the plurality of entities from the conversation graph based on a familiarity metric associated with one or more of the plurality of entities, wherein one or more of the plurality of entities are displayed as suggested entities based on the ranking.The method may include: detecting a message to be delivered to the wearable device; assigning a priority to the message based on the conversation graph; delaying delivery of the message in response to the priority being below a threshold level; and delivering the message in response to the priority being above the threshold level. The wearable device may include smart glasses.

[0005] According to one aspect, a non-transitory computer-readable medium storing executable instructions, the executable instructions being configured to cause the at least one processor to perform the following operations when executed by at least one processor: detecting facial features of an entity by at least one imaging sensor of a wearable device; determining that the facial features correspond to an existing contact associated with a user of the wearable device; detecting interactive communications between the user and the entity based on image data from the at least one imaging sensor and audio data received via a microphone of the wearable device; updating a conversation graph in response to detecting interactive communications between the user and the entity, wherein the conversation graph includes a plurality of nodes representing the entity and links connected between the plurality of nodes. The executable instructions for updating the conversation graph include instructions for calculating a familiarity metric associated with the entity and annotating nodes or links with the familiarity metric, and managing content for display on the wearable device based on the conversation graph.

[0006] According to some aspects, the non-transitory computer-readable medium may include one or more (e.g., all) of the following features (or any combination thereof). The executable instructions include instructions that, when executed by the at least one processor, cause the at least one processor to: obtain first image data by a first imaging sensor; determine whether a facial feature is detected in the first image data; and obtain second image data by a second imaging sensor in response to the facial feature being determined to be detected in the first image data. The second image data has a higher resolution than the first image data. The second image data is used to determine whether the facial feature corresponds to an existing contact. The executable instructions include instructions that, when executed by the at least one processor, cause the at least one processor to: receive a voice command from a user via a microphone of a wearable device; identify an entity associated with the voice command based on a conversation graph; and perform an action using the identified entity.

[0007] According to one aspect, a wearable device includes: a first imaging sensor configured to receive first image data; a second imaging sensor configured to receive second image data, wherein the second image data has a higher resolution than the first image data; and an image analyzer configured to detect the presence of a facial feature of an entity in the first image data, and activate the second imaging sensor to obtain the second image data in response to detecting the facial feature in the first image data. The image analyzer is configured to use the second image data to determine that the facial feature corresponds to an existing contact associated with a user of the wearable device. The wearable device includes: an interactive communication detector configured to detect interactive communication between the user and the entity based on at least one of the first image data or the second image data; a conversation graph builder configured to update a conversation graph in response to detecting interactive communication between the user and the entity; and a display manager configured to manage content displayed on a display of the wearable device based on the conversation graph.

[0008] According to some aspects, the wearable device may include one or more (e.g., all) of the following features (or any combination thereof). The conversation graph includes a plurality of nodes representing entities and links connecting between the plurality of nodes, wherein the conversation graph builder is configured to calculate a familiarity metric associated with the entity and annotate the node or link with the familiarity metric. The display manager includes a notification manager configured to manage display of notifications based on the conversation graph, wherein the wearable device further includes a cluster detector configured to detect an entity cluster group among the plurality of entities based on information included in the conversation graph. The notification manager is configured to identify that the notification is associated with an entity and assign a priority to the notification based on whether the entity associated with the notification is included in the entity cluster group.

[0009] According to one aspect, a non-transitory computer-readable medium storing executable instructions, the executable instructions being configured to cause the at least one processor to perform the following operations when executed by at least one processor: receiving image data from at least one imaging sensor of a wearable device, the image data including facial data of an entity; determining that facial features correspond to existing contacts associated with a user of the wearable device; detecting interactive communications between the user and the entity based on the image data from the at least one imaging sensor and audio data received via a microphone of the wearable device; updating a conversation graph in response to detecting interactive communications between the user and the entity, wherein the conversation graph includes a plurality of nodes representing the entity and links connected between the plurality of nodes. The executable instructions for updating the conversation graph include instructions for calculating a familiarity metric associated with the entity and annotating a node or link with the familiarity metric. The executable instructions include instructions for managing content for display on the wearable device based on the conversation graph. In some examples, the executable instructions include instructions that, when executed by the at least one processor, cause the at least one processor to: receive first image data from a first imaging sensor on the wearable device; determine whether a facial feature is detected in the first image data; and in response to the facial feature being determined to be detected in the first image data, receive second image data from a second imaging sensor, the second image data having a higher resolution than the first image data, wherein the second image data is used to determine whether the facial feature corresponds to an existing contact. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A

[0013] A computing system for managing content using a session graph according to an aspect is shown.

[0011] Figure 1B A session diagram according to an aspect is shown.

[0012] Figure 1C An example of a notification manager for managing notifications using a session graph is shown according to an aspect.

[0013] Figure 1D An example of a notification manager for managing notifications using cluster groups of a session graph according to an aspect is shown.

[0014] Figure 1E An example of a messaging manager for managing messaging using a session graph according to an aspect is shown.

[0015] Figure 1F An example of an application that displays suggested entities from a conversation graph according to an aspect is shown.

[0016] Figure 1GAn example of an auto-completion manager for ranking entities in a conversation graph according to an aspect is shown.

[0017] Figure 2

[0013] A computing system for managing content using a session graph according to an aspect is shown.

[0018] Figure 3 An example of a head mounted display device according to an aspect is shown.

[0019] Figure 4 A flowchart depicting example operation of a computing system according to an aspect is shown.

[0020] Figure 5 A flowchart depicting example operation of a computing system according to another aspect is shown.

[0021] Figure 6 A flow diagram depicting example operations for managing notifications using a session graph according to an aspect is shown.

[0022] Figure 7 A flow diagram depicting example operations for ranking entities in a conversation graph according to an aspect is shown.

[0023] Figure 8 A flow diagram depicting example operations for managing messages using a conversation graph according to an aspect is shown.

[0024] Fig. 9

[0013] A flow diagram depicting example operations for identifying entities in voice commands using a conversation graph according to an aspect is shown. DETAILED DESCRIPTION

[0025] The present disclosure relates to a computing system that uses a conversation graph to manage content to be displayed. The computing system may include a head-mounted display device such as smart glasses. The conversation graph is constructed based on detected real-life interactions between a user and other entities. For example, the conversation graph may include multiple nodes and multiple links connecting the multiple nodes, where each node may represent an entity that has a conversation with the user in real life. For example, a first node may represent a user of the computing system, and a second node may represent entity A. The first node is connected to the second node via a link, where the link (or the second node) is annotated with a familiarity metric. The familiarity metric may represent a familiarity level between the user and entity A (e.g., the higher the familiarity metric, the higher the familiarity level between the user and entity B). The familiarity metric may be calculated based on the frequency and / or duration of the interaction between the user and entity A.

[0026] The computing system may include one or more imaging sensors. The imaging sensor may capture image data, wherein at least a portion of a person is included in the image data (e.g., person A enters the field of view of the imaging sensor(s)). The computing system (e.g., a wearable device) may analyze the image data to detect facial features of person A and determine whether the facial features correspond to an existing contact associated with the user. In some examples, the computing system may detect that the user should recognize the entity, but is unsure of the identity of the entity. For example, the computing system may not necessarily recognize the entity, but may use the fact that the entity exists as a clue.

[0027] In some examples, the computing system can detect a face in the image data and compare the detected face to the image data associated with its contacts to determine (e.g., by accessing a database storing existing contact data) whether the newly captured face corresponds to one of the user's existing contacts. In some examples, the computing system includes a first imaging sensor (e.g., a low-power camera) and a second imaging sensor (e.g., a high-power camera). In some examples, the first imaging sensor is activated (e.g., continuously or periodically) to detect the presence of facial features within the image data. If the facial features are detected, the second imaging sensor is activated to capture higher resolution image data, which is then used to determine whether the newly captured face corresponds to one of the user's existing contacts. When the computing system detects facial features of an existing contact (e.g., person A is one of the user's contacts), the computing system can determine whether the user and person A are in interactive communication (e.g., a conversation) based on the image data (and / or audio data captured from a microphone on the wearable device). In response to detecting interactive communication between the user and person A, the computing system can update the conversation graph. In some examples, the data (or a portion of the data) is stored locally.

[0028] The computing system may use a session graph to manage content (e.g., notifications, messages, etc.) displayed on a display of the system. This allows sensor-based solutions to reduce the content displayed in an efficient manner. In some examples, the content is a notification generated by an application executable by the computing system. In some examples, the content is information provided via an auto-complete interface of the application. In some examples, the content is a message (e.g., an instant message) to be delivered to the display of the system. In some examples, instead of (or in addition to) managing the content displayed on the display, the computing system may use the session graph to identify an entity specified (or partially specified) in a voice command.

[0029] In more detail, a computing system may detect a notification from an application executable by the computing system and determine whether to display the notification on a display using a conversation graph. For example, if the notification is associated with an entity included in the conversation graph, the computing system may determine to display the notification on the display. If the notification is not associated with an entity included in the conversation graph, the computing system may determine not to display the notification at this time. In some examples, the computing system assigns priorities to notifications based on the conversation graph. For example, if a notification is associated with an entity included in the conversation graph, the notification may be assigned a higher priority than a notification associated with an entity not included in the conversation graph. A higher priority notification may be displayed to a user before a lower priority notification. In some examples, the computing system uses a familiarity metric to assign different priorities to entities within the conversation graph. For example, if a user regularly converses with entity B (but not regularly with entity A), the computing system may assign a priority to notifications associated with entity B that is higher than the priority assigned to notifications associated with entity A. In this way, the computing system may reduce the amount of unwanted or less useful notifications presented to the user.

[0030] In some examples, the computing system may detect one or more cluster groups based on information included in the conversation graph. Each cluster group includes two or more nodes (or entities) having data corresponding to common characteristics. If entity A has a conversation with the user, the conversation graph may be updated to incorporate event metadata about the interaction. Over time, the computing system may use the event metadata and the connections between the nodes in the conversation graph to detect one or more entity cluster groups (e.g., work entities, home entities, social entities, etc.). The computing system may then use the cluster groups to determine which notifications to display on the display (e.g., assigning priority to work notifications when the user is physically with work contacts, assigning priority to social notifications when the user is physically with social contacts, etc.).

[0031] In some examples, the conversation graph is used to create a smarter auto-complete interface. A computing system can receive information from a user via an application executable by the computing system. For example, a user can be using an email application to create a new message, where the user begins typing a name in an address bar. The computing system can identify one or more entities from the conversation graph and then display these entities as suggested entities on the auto-complete interface.

[0032] Figures 1A to 1GA computing system 100 is shown for managing content 111 on a display 114 using a conversation graph 138, according to an aspect. For example, based on image data 113 received via one or more imaging sensors 104, the computing system 100 builds a conversation graph 138 that identifies entities with which a user of the computing system 100 interacts in real life (and, in some examples, virtually). Using the conversation graph 138, the computing system 100 manages content 111 displayed on the display 114, such as prioritizing notifications 132 and / or messages 133 from entities included within the conversation graph 138 and / or identifying one or more suggested entities 135 on an auto-complete interface of an application 116.

[0033] In some examples, computing system 100 may identify an entity associated with a voice command based on conversation graph 138 (e.g., a voice command of “ok device, call Tristan” may identify an intent entity when there are multiple existing contacts 130 for “Tristan”). In some examples, conversation graph 138 may be analyzed to determine one or more cluster groups 160 (e.g., work entities, social entities, etc.), and computing system 100 may use cluster groups 160 to determine which notifications 132 are to be prioritized for display on display 114 (e.g., assigning priority to work notifications when the user is physically with work contacts, assigning priority to social notifications when the user is physically with social contacts, etc.).

[0034] The computing system 100 may include one or more devices, at least one of which is a display device that can be worn on a person's skin. In some examples, the computing system 100 is or includes a wearable device. The wearable device may include a head-mounted display (HMD) device such as an optical head-mounted display (OHMD) device, a transparent head-up display (HUD) device, an augmented reality (AR) device, or other devices such as Google devices or headsets with sensors, displays, and computing capabilities. In some examples, the wearable device includes smart glasses. Smart glasses are optical head-mounted display devices designed in the shape of a pair of glasses. For example, smart glasses are glasses that add information (e.g., projecting display 114) next to the content viewed by the wearer through the glasses. In some examples, the computing system 100 includes a wearable device (e.g., smart glasses) and a computing device (e.g., a mobile computing device such as a smartphone, or a server computer). The wearable device can be connected to the computing device via a wireless connection such as a short-range connection (e.g., a Bluetooth connection or a near field communication (NFC) connection) or an Internet connection (e.g., Wi-Fi or a mobile network). In some examples, some of the components of computing system 100 are included in a wearable device, and some of the components of computing system 100 are included in a computing device. In some examples, all of the components of computing system 100 are included in a wearable device.

[0035] The computing system 100 includes one or more processors 103, which may be formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processor 103 may be semiconductor-based, i.e., the processor may include semiconductor materials that may execute digital logic. The computing system 100 may also include one or more memory devices 105. The memory device 105 may include any type of storage device that stores information in a format that may be read and / or executed by the processor(s) 103. The memory device 105 may store applications 116 and modules (e.g., image analyzer 106, interactive communication detector 108, conversation graph builder 110, cluster detector 112, display manager 140, voice action manager 142, etc.) that perform certain operations when executed by the processor(s) 103. In some examples, the applications 116 and modules may be stored in an external storage device and loaded into the memory device 105.

[0036] The computing system 102 includes applications 116, which may be stored in the memory device 105 and which perform certain operations when executed by the processor 103. The applications 116 may be any type of computer program that may be executed by the computing system 100. The applications 116 may enable the computing system 100 to perform certain functions, such as browsing the Internet, sending emails or text messages, capturing images, recording videos, getting directions, etc. Although the applications 116 may vary widely, they may include an email application 118, a calendar application 120, a browser application 122, a storage application 124, a voice call application 126, and / or a messaging application 128.

[0037] The computing system 100 includes a display 114 that is projected onto a user's field of view. The display 114 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting display (OLED), an electrophoretic display (EPD), or a micro-projection display using an LED light source. In some examples, in the case of smart glasses, the display 114 may provide a transparent or semi-transparent display so that a user wearing the glasses can see the image provided by the display 114 and can also see information in the field of view of the smart glasses that is behind the projected image. Although in Figures 1A to 1G 1, but computing system 100 may include a touchpad that allows a user to control computing system 100 (eg, the touchpad may allow swiping across an interface displayed on display 114). Figures 1A to 1G Not shown, but computing system 100 includes a battery configured to provide power to circuit components, a battery charger for controlling charging of the battery, and one or more antennas for enabling communication with other computing device(s).

[0038] The computing system 100 includes a plurality of sensors, such as a microphone 136 configured to capture audio data 134, and one or more imaging sensors 104 configured to capture image data 113. Figure 1A 1H , but computing system 100 may include other types of sensors, such as light condition sensors (e.g., ambient light sensors), and motion sensors configured to detect motion of computing system 100. Microphone 136 is a transducer device that converts sound into electrical signals represented by audio data 134. Imaging sensor(s) 104 are cameras that detect and transmit information used to make an image represented by image data 113. Imaging sensor(s) 104 can take pictures and record video. In some examples, computing system 100 includes a single imaging sensor 104. In some examples, computing system 100 includes multiple imaging sensors 104.

[0039] In some examples, computing system 100 includes imaging sensor 104a and imaging sensor 104b. In some examples, computing system 100 includes more than two imaging sensors 104. Imaging sensor 104a can be considered a low power low resolution (LPLR) image sensor. Imaging sensor 104b can be considered a high power high resolution (HPLR) image sensor. The image captured by imaging sensor 104b has a higher resolution than the image captured by imaging sensor 104a. In some examples, imaging sensor 104a is configured to obtain image data 113 when computing system 100 is activated and computing system 100 is coupled to a user (e.g., continuously or periodically capture image data 113 when computing system 100 is activated). In some examples, imaging sensor 104a is configured to operate as a normally-on sensor. In some examples, imaging sensor 104b is activated (e.g., for a short duration) in response to detection of an object of interest, as discussed further below.

[0040] When the user is wearing the computing system 100 (or a portion of the computing system 100), the imaging sensor 104 captures image data 113. The computing system 100 includes an image analyzer 106 that receives the image data 113 captured by the imaging sensor 104. In some examples, the imaging sensor 104a (e.g., an LPLR image sensor) is activated when the user is wearing the computing system 100, and the image analyzer 106 receives the image data 113 from the imaging sensor 104a. The image analyzer 106 analyzes the image data 113 to determine whether the image data 113 includes an object of interest (e.g., facial features (or face) of an entity).

[0041] The image analyzer 106 may include a machine learning (ML) model (e.g., one or more classifiers) configured to perform an object recognition algorithm (e.g., a facial recognition algorithm). In some examples, the ML model includes a neural network. A neural network may be a set of interconnected nodes, each node representing an artificial neuron. The nodes are connected to each other in layers, where the output of one layer becomes the input of the next layer. The neural network transforms the input (e.g., image data 113) received by the input layer, transforms the input through a series of hidden layers, and produces an output via an output layer. Each layer consists of a subset of a set of nodes. The nodes in the hidden layer are fully connected to all nodes in the previous layer, and the output of the nodes in the hidden layer is provided to all nodes in the next layer. The nodes in a single layer act independently of each other (i.e., do not share connections). The nodes in the output provide the transformed input to the requesting process. In some examples, the neural network is a convolutional neural network, which is a neural network that is not fully connected. Therefore, a convolutional neural network has a lower complexity than a fully connected neural network. Convolutional neural networks can also use pooling or maximum pooling to reduce the dimensionality of the data flowing through the neural network (thereby reducing the complexity), and therefore this can reduce the required level of calculation. This makes the computation of the output in a convolutional neural network faster than in a neural network.

[0042] Image analyzer 106 may receive image data 113 and use the ML model to predict whether image data 113 includes facial features of an entity. If image analyzer 106 does not detect facial features of an entity within image data 113, image analyzer 106 may continuously (e.g., periodically) receive image data 113 from imaging sensor 104 (e.g., imaging sensor 104a) to continue determining whether image data 113 includes facial features of an entity that enters a field of view of imaging sensor 104. In some examples, because imaging sensor 104a (e.g., LPLR camera) is used to search for facial features, power of computing system 100 may be reduced, which may extend the life of a battery of computing system 100.

[0043] If the image analyzer 106 detects the presence of facial features within the image data 113, the image analyzer 106 determines whether the facial features are associated with the user's existing contacts 130. The existing contacts 130 may be obtained from one or more of the applications 116. In some examples, the existing contacts 130 may be obtained from a contact service executable by a server computer over a network (e.g., the Internet). In some examples, one of the applications 116 is a contact management application, and the existing contacts 130 may be accessed from the contact management application. In some examples, the existing contacts 130 are determined from an email application 118, a messaging application 128, a browser application 122, and / or any application 116 that is generally used to communicate virtually with other entities. In some examples, the existing contacts 130 are determined from a plurality of applications 116 (e.g., stored contacts from the user's email, text, and / or calendar). The existing contacts 130 may include information about each entity, such as the entity's name, address, phone number, and / or email address, etc. Existing contacts 130 may be associated with image data (e.g., contact information for person A may be stored with a picture of person A, and information for person B may be stored with a picture of person B). Image analyzer 106 may compare facial features of image data 113 with image data of existing contacts 130 to determine whether the face of the entity captured by imaging sensor 104 is associated with one of existing contacts 130.

[0044] The resolution of the image data 113 captured by the imaging sensor 104a (e.g., an LPLR camera) may be too low to perform a comparison with the image data associated with the existing contacts 130. In some examples, if the image analyzer 106 detects the presence of facial features within the image data 113 captured by the imaging sensor 104a, the image analyzer 106 may activate the imaging sensor 104b (e.g., an HPHR camera) to capture a higher resolution image of the facial features of the entity, and the higher resolution image data is used to compare the facial features with the image data of the existing contacts 130 to determine whether the face of the entity captured by the imaging sensor 104b is associated with one of the existing contacts 130.

[0045] If the face of the entity captured by the imaging sensor 104 is associated with one of the existing contacts 130, the interactive communication detector 108 can detect whether interactive communication 109 is occurring between the user and the entity based on at least the image data 113. In some examples, the interactive communication 109 is a real-life communication (e.g., one-way communication or two-way communication) between the user and the entity. In some examples, the interactive communication 109 can be referred to as a conversation between two or more entities. Using the image data 113, the interactive communication detector 108 can estimate the location of the entity relative to the user of the computing system 100, and / or the gaze direction of the entity. Based on the estimated location and / or gaze direction, the interactive communication detector 108 can determine whether the user and the entity are performing interactive communication 109 (e.g., the entity can be located in front of the user, and the gaze direction can indicate that the entity is looking at the user). In addition to the image data 113, in some examples, the interactive communication detector 108 also uses audio data 134 received via a microphone 136 on the computing system 100 to detect whether interactive communication 109 is occurring between the user and the entity. For example, the interactive communication detector 108 may determine that the audio data 134 includes speech, and the detection of the speech in combination with the information derived from the image data 113 may indicate that an interactive communication 109 is occurring between the entity and the user. In some examples, the interactive communication detector 108 detects the volume (or loudness) of the speech, and if the volume of the speech in combination with the information derived from the image data 113 is above a threshold level, the interactive communication detector 108 may detect an interactive communication 109 between the user and the entity.

[0046] In response to detecting an interactive communication 109 between the user and the entity, the conversation graph builder 110 can update the conversation graph 138 associated with the user. In some examples, the conversation graph builder 110 updates the conversation graph 138 in response to the interactive communication 109 being detected as starting. In some examples, the conversation graph builder 110 updates the conversation graph 138 in response to the interactive communication 109 being detected as ending (e.g., this may be based on the entity having moved to a point outside the field of view of the imaging sensor 104 and / or the audio data 134 being determined to not include speech). In some examples, even if the face of the entity captured by the imaging sensor 104 is not associated with one of the existing contacts 130, and the interactive communication detector 108 detects an interactive communication 109 between the entity and an unknown contact, the conversation graph builder 110 may update the conversation graph 138 to merge the unknown contact.

[0047] The conversation graph 138 is constructed based on detected real-world interactions between users and other entities. Figure 1B, the conversation graph 138 may include a plurality of nodes 156, and a plurality of links 158 connecting the plurality of nodes 156, wherein each node 156 may represent an entity with which the user is having a conversation in real life. In some examples, the conversation graph 138 includes nodes 156 representing entities with which the user has virtually communicated, e.g., via email, text messaging, instant messaging, etc.

[0048] In some examples, such as Figure 1B As shown, the plurality of nodes 156 may include a first node 156-1 representing a user of the computing system 100, a second node 156-2 representing an entity A, a third node 156-3 representing an entity B, a fourth node 156-4 representing an entity C, a fifth node 156-5 representing an entity D, and a sixth node 156-6 representing an entity E. Entities A through E may represent different persons, wherein the computing system 100 has detected interactive communications 109 with the user of the computing system 100. Each of the second node 156-2 through the sixth node 156-6 is connected to the first node 156-1 via a separate link 158. The conversation graph 138 includes a familiarity metric 154 associated with each entity included within the conversation graph 138. In some examples, each link 158 is annotated with a corresponding familiarity metric 154. The familiarity metric 154 may represent a familiarity level between the user and the entity (e.g., the higher the familiarity metric 154, the higher the familiarity level between the user and the entity). The conversation graph builder 110 can calculate the familiarity metric 154 based on the frequency and / or length of interactions between the user and the corresponding entity. For example, if entity A has a plurality of detected interaction communications 109 with the user of the computing system 100 (and / or one or some of the detected interaction communications 109 are relatively long (e.g., compared to detected interaction communications with other entities)), then the familiarity metric 154 associated with entity A (or the link 158 connected between the first node 156-1 and the second node 156-2) can be relatively high (e.g., compared to the familiarity metrics of other entities).

[0049] In some examples, in response to detecting interactive communication 109 between the user and entity A, the session graph builder 110 may determine whether a node 156 representing entity A is already included in the session graph 138. If entity A is not included in the session graph 138, the session graph builder 110 may add a new node 156 (e.g., a second node 156-2) to the session graph 138, wherein a new link 158 is connected to the first node 156-1 representing the user of the computing system 100. If entity A is determined to be an existing contact 130, the second node 156-2 may be annotated with information about entity A, such as name and other contact details. If entity A is not determined to be an existing contact 130, the second node 156-2 may be annotated with information indicating that entity A is unknown.

[0050] In some examples, the conversation graph builder 110 can update the conversation graph 138 to include event metadata 152 about the conversation event. The conversation event can be created in response to the detection of the interactive communication 109. The event metadata 152 can include the time length of the interactive communication 109, the start and end points of the detected interactive communication 109, and / or the location where the interactive communication 109 occurred. The event metadata 152 can also store the frequency of the interactive communication 109 between the user and the corresponding entity over a period of time.

[0051] With respect to entity A, the conversation graph builder 110 can calculate a familiarity metric 154 regarding the strength of the connection between the user and entity A based on the event metadata 152, and annotate a link 158 between the first node 156-1 and the second node 156-2 with the familiarity metric 154 associated with entity A. For example, if the interaction communication 109 between entity A and the user is relatively short (e.g., compared to detected interaction communications of other entities), and it is the first detected interaction communication 109 between entity A and the user, the familiarity metric 154 can be relatively low (e.g., compared to familiarity metrics of other entities). In response to detecting the interaction communication 109 between the user and entity A, and determining that entity A has been included in the conversation graph 138, the conversation graph builder 110 can adjust (e.g., increase) the familiarity metric 154, and update the event metadata 152 to include the new conversation event details.

[0052] In some examples, over time, one or more cluster groups 160 may be detected. For example, the cluster detector 112 may analyze the conversation graph 138 to detect one or more cluster groups 160 based on information included within the conversation graph 138 (e.g., nodes 156, links 158, event metadata 152, familiarity metrics 154, etc.), where each cluster group 160 includes: two or more nodes 156 having data corresponding to a common characteristic (e.g., work, home, social, etc.). As the conversation graph 138 is built over time (e.g., entities A through E are merged into the conversation graph 138), the cluster detector 112 may use the event metadata 152 and connections between the various nodes 156 of the conversation graph 138 to detect one or more cluster groups 160 of entities (e.g., work entities, home entities, social entities, etc.). Then, as more interactive communications 109 are detected, the cluster detector 112 may assign the new entity to a cluster group 160 based on the characteristics of the new entity. In some examples, other signals include the type of environment in which the communication occurs.

[0053] In some examples, the cluster detector 112 is configured to execute a clustering algorithm to detect one or more cluster groups 160 of entities. In some examples, the cluster detector 112 may use k-means or other unsupervised clustering algorithms to detect cluster groups 160. In some examples, the clustering algorithm includes a community-based clustering algorithm. In some examples, the clustering algorithm includes a sim clustering algorithm. In some examples, the clustering algorithm includes a modularity-based graph partitioning algorithm. In some examples, the clustering algorithm includes a Louvain clustering algorithm. In some examples, the clustering algorithm includes a machine learning-based algorithm that defines a neural network. In some examples, the clustering algorithm includes any type of connectivity-based clustering algorithm (e.g., a hierarchical clustering algorithm), a centroid-based clustering algorithm, a distribution-based clustering algorithm, and / or a density-based clustering algorithm.

[0054] For example, the cluster detector 112 may determine that entity B and entity C are work contacts, and create a first cluster group 160-1 using entities B and C. The cluster detector 112 may determine that entity E and entity D are social contacts, and create a second cluster group 160-2 using entities E and D. As discussed further below, the display manager 140 may use the cluster groups 160 to determine which notifications 132 are to be displayed on the display 114 (e.g., assigning priority to work notifications when the user is physically with work contacts, assigning priority to social notifications when the user is physically with social contacts, etc.).

[0055] refer to Figure 1A, display manager 140 may use session graph 138 to manage content 111 displayed on display 114 of computing system 100. Content 111 may be generated by application 116 or an operating system associated with computing system 100. In some examples, content 111 may be any type of data that may be displayed on display 114, which may originate from or be associated with an entity. In some examples, display manager 140 may allow display of content 111 if content 111 is associated with an entity (e.g., entity A through entity E) included within session graph 138. For example, if content 111 is generated by or associated with entity A, and entity A is included within session graph 138, display manager 140 may allow display of content 111, or assign a higher priority to display of content 111. In some examples, display manager 140 may allow display of content 111, or assign a higher priority to display of content 111, if the entity has a familiarity metric 154 that exceeds a threshold amount.

[0056] In some examples, if content 111 is associated with an entity that is not included in conversation graph 138, display manager 140 may delay display of content 111 or assign a lower priority to display of content 111. In some examples, content 111 may be delayed until computing system 100 is activated from a power saving state, or at some other point later after the point at which content 111 is initially set to be displayed. If content 111 is associated with entity F (e.g., entity F is not included in conversation graph 138), display manager 140 may delay display of content 111 or assign a lower priority to display of content 111. In some examples, if content 111 is associated with an entity included in conversation graph 138 and the familiarity metric 154 of the entity is below a threshold amount, display manager 140 may delay display of content 111 or assign a lower priority to display of content 111.

[0057] In some examples, content 111 includes notifications 132 generated by application 116. For example, an online streaming application may generate notifications 132 about new shows (where the online streaming application is considered an entity), or an email application 118 may generate notifications 132 about new emails received from an entity. In some examples, content 111 includes messages 133. Messages 133 may be text messages or other types of instant messaging. In some examples, content 111 includes one or more suggested entities 135 from a conversation map 138 for display on an auto-complete interface of application 116.

[0058] Display manager 140 may include notification manager 141 configured to manage notifications 132 based on conversation graph 138. In some examples, display manager 140 includes messaging manager 143 configured to manage delivery of messages 133 based on conversation graph 138. In some examples, display manager 140 may include auto-completion manager 145 configured to use conversation graph 138 to identify one or more suggested entities 135 for display on an auto-completion interface of application 116.

[0059] like Figure 1C As shown, notification manager 141 can detect notification 132 to be delivered to display 114 and assign priority 162 to notification 132 based on conversation map 138. Priority 162 assigned to notification 132 can indicate the timing (and / or manner) at which notification 132 is notified to a user of computing system 100. Priority 162 can include multiple priority levels, such as a first priority level, a second priority level, and a third priority level. The first priority level can be the highest priority level (e.g., most disruptive to the user), and the third priority level can be the lowest priority level (e.g., least disruptive to the user). In some examples, priority 162 includes more than three priority levels.

[0060] Priority 162 may include a variety of notification priority levels, such as immediately displaying notification 132 and redisplaying notification 132 if notification 132 is not checked after a certain period of time, immediately displaying notification 132 without redisplaying notification 132 after a period of time, immediately displaying notification 132 with an audible and / or vibration alert, not displaying notification 132 but providing an audible and / or vibration alert, delaying display of notification 132 until another notification 132 from the same entity is detected (and then displaying notifications 132 as a group), displaying display of notification 132 until multiple similar notifications 132 are detected (and then displaying similar notifications 132 as a group), not displaying notification 132 and not providing an audible and / or vibration alert, etc. In addition, if notification 132 is to be displayed, priority 162 may provide different priority levels for the size and / or location of the notification (e.g., a higher priority may indicate being displayed in a larger size or at a more visible location on display 114 to the user).

[0061] Notification manager 141 may determine whether to display notification 132 on display 114 based on assigned priority 162. In some examples, notification manager 141 may assign a higher priority 162 to notification 132 if the notification is associated with an entity included within conversation map 138. In some examples, notification 132 from email application 118 may have a default priority notification level (or a priority notification level set by a user for all notifications 132 generated by email application 118). In some examples, notification manager 141 may assign a priority 162 to notification 132 that is higher than the default or set priority notification level if the notification 132 is associated with an entity included within conversation map 138.

[0062] like Figure 1C As shown in , notification manager 141 can detect a first notification 132-1 (wherein the first notification 132-1 is associated with entity E), a second notification 132-2 (wherein the second notification 132-2 is associated with entity B), and a third notification 132-3 (wherein the third notification 132-3 is associated with entity H). Notification manager 141 can detect that the first notification 132-1, the second notification 132-2, and the third notification 132-3 are at different times or at approximately the same time. Figure 1B , a node representing entity E (e.g., the sixth node 156-6) is included in the session graph 138, and a node representing entity B (e.g., the third node 156-3) is included in the session graph 138. However, a node representing entity H is not included in the session graph 138.

[0063] Return to reference Figure 1C , since entity E and entity B are included in conversation graph 138, notification manager 141 may assign a higher priority 162 to first notification 132-1 and second notification 132-2 than to third notification 132-3. In some examples, notification manager 141 may assign priority 162 using familiarity metric 154. For example, if entity B has a higher familiarity metric 154 than entity E, notification manager 141 may assign a higher priority 162 to second notification 132-2 than to first notification 132-1.

[0064] In some examples, reference Figure 1D, the notification manager 141 uses the detected cluster groups 160 to manage the display of notifications 132. For example, over time, the cluster detector 112 can detect cluster groups 160 such as a first cluster group 160-1 and a second cluster group 160-2. The first cluster group 160-1 can relate to a first type of contact, and the second cluster group 160-2 can relate to a second type of contact. The first type of contact can include work contacts. The second type of contact can include social contacts. However, it should be noted that work contacts and social contacts are merely examples, where the first cluster group 160-1 and the second cluster group 160-2 can be related to other types of groups. Figure 1B As shown, the first cluster group 160 - 1 may include entity B and entity C, and the second cluster group 160 - 2 may include entity D and entity E.

[0065] In some examples, notification manager 141 may determine whether to display notification 132 based on whether an entity associated with notification 132 is included in cluster group 160. For example, notification manager 141 may receive an indication that a user has work contacts. For example, notification manager 141 may operate in conjunction with image analyzer 106, and image analyzer 106 may use image data 113 to determine that a user has one or more work contacts. In some examples, notification manager 141 may use time and / or location information to determine whether a user is at work (and / or at another predetermined location, for example). If the user is at work (and / or has work contacts) and the detected notification 132 is related to entity B, notification manager 141 may assign a higher priority 162 to notification 132 because entity B is within first cluster group 160-1 related to work contacts. On the other hand, if the detected notification 132 is related to entity E, notification manager 141 may assign a lower priority 162 to notification 132 because entity E is not a work contact.

[0066] like Figure 1EAs shown, the messaging manager 143 can detect a message 133 to be delivered to the display 114 and assign a priority 163 to the message 133 based on the conversation map 138. The message 133 can relate to a message delivered via the messaging application 128. In some examples, the message 133 can come from one of a plurality of different messaging applications 128. The priority 163 assigned to the message 133 can indicate the timing (and / or manner) at which the message 133 is delivered to the user of the computing system 100. The priority 163 can include a plurality of priority levels, such as a first priority level, a second priority level, and a third priority level. The first priority level can be the highest priority level (e.g., most disruptive to the user), and the third priority level can be the lowest priority level (e.g., least disruptive to the user). In some examples, the priority 163 includes more than three priority levels. The priority 163 can include a plurality of notification priority levels, such as displaying the message 133 immediately, delaying the message 133 until a later time, and / or blocking the message 133. In some examples, the priority 163 includes any of the priorities discussed with reference to the priority 162.

[0067] Messaging manager 143 may determine whether to display message 133 on display 114 based on assigned priority 163. In some examples, in response to priority 163 being greater than a threshold level, message 133 is displayed on display 114. In some examples, messaging manager 143 may assign a higher priority 162 to message 133 if message 133 is associated with an entity included within conversation graph 138. For example, messaging manager 143 may detect first message 133-1 (where first message 132-1 is associated with entity E), second message 133-2 (where second message 133-2 is associated with entity B), and third message 133-3 (where third message 133-3 is associated with entity H). Since entity E and entity B are included in conversation graph 138 and entity H is not included in conversation graph 138, messaging manager 143 may assign a higher priority 163 to first message 133-1 and second message 133-2 than to third message 133-3, with higher priority messages 133 being displayed first. In some examples, messaging manager 143 may use familiarity metric 154 to assign priority 163. For example, if entity B has a higher familiarity metric 154 than entity E, messaging manager 143 may assign a higher priority 162 to second message 133-2 than to first message 133-1.

[0068] In some examples, session graph 138 is used to create a smarter auto-complete interface. For example, auto-complete manager 145 can use session graph 138 to identify one or more suggested entities 135 to be displayed on the auto-complete interface of application 116. In some examples, computing system 100 can receive information from a user via application 116. For example, referring to Figure 1F , a user may be using email application 118 to create a new message 170, where the user begins typing a name in address bar 172. Auto-completion manager 145 may identify one or more entities (e.g., entity A, entity E, and entity F) from conversation graph 138, and then display these entities as suggested entities 135 on the auto-completion interface. For example, if the user types "K" in address bar 172, auto-completion manager 145 may identify nodes 156 in conversation graph 138 that begin with the letter "K" (e.g., Kellen, Katie, Kevin), and then display these entities as suggested entities 135 on the auto-completion interface of application 116. After selecting the intended entity, the user may select Send 174 to send the email. Although Figure 1F An email application 118 is used as an example, but the auto-completion manager 145 may operate in conjunction with any type of application 116 , such as a calendar application 120 , a storage application 124 , and / or a browser application 122 .

[0069] In some examples, such as Figure 1G As shown, the auto-completion manager 145 can rank the nodes 156 (entities) in the conversation graph 138 based on the familiarity metric 154 of each node, and then provide the related entities as a ranked list 185 on the auto-completion interface of the application 116. For example, entity A can be associated with familiarity metric A, entity B can be associated with familiarity metric B, entity C can be associated with familiarity metric C, and entities D through N can be associated with familiarity metrics D through N, respectively. For example, if Kellen (entity A) has the highest familiarity metric 154 and Kevin (entity N) has the lowest familiarity metric 154, the auto-completion manager 145 can provide a list of suggested entities 135 as Kellen, Katie, and Kevin. In some examples, instead of ranking according to the familiarity metric 154, the auto-completion manager 145 can use the event metadata 152 to rank the suggested entities 135. For example, if the event metadata 152 indicates that the user recently interacted with Kevin 109 , the auto-completion manager 145 may identify Kevin as a suggested entity 135 in the address bar 172 .

[0070] In some examples, the reference Figure 1AInstead of (or in addition to) managing the content 111 displayed on the display 114, the voice action manager 142 can use the conversation graph 138 to identify the entity specified (or partially specified) in the voice command 144. For example, the microphone 136 can receive the audio data 134, and the audio data 134 can specify the voice command 144. For example, the voice command 144 can specify a hot word (e.g., "ok device"), followed by a command such as composing a specific message to the entity, calling a specific entity, etc. In some examples, the voice action manager 142 can use the conversation graph 138 to identify possible entities subject to the command. For example, a particular voice command 144 can specify "ok device, call Tristan". However, there may be several entities named Tristan in the existing contacts 130 (e.g., Tristan A, Tristan B). The conversation graph 138 can be fully utilized to determine which entity (e.g., Tristan A or Tristan B) is the possible entity that the user wants to call. For example, if Tristan A is a close friend of the user, and Tristan B is an acquaintance of the user, the conversation graph 138 may have a node 156 representing an entity named Tristan A that has a relatively high familiarity metric 154. The conversation graph 138 may or may not have a node 156 representing an entity named Tristan B, and if it does, the conversation graph 138 may have a relatively low familiarity metric 154. In some examples, the voice action manager 142 may use the conversation graph 138 to identify that Tristan A is the subject of the voice command 144 because Tristan A is included in the conversation graph 138 and / or has a higher familiarity metric 154 than the familiarity metric 154 associated with Tristan B.

[0071] Figure 2 FIG. 2 shows a computing system 200 according to another aspect. The computing system 200 may include a reference Figures 1A to 1G In some examples, computing system 200 includes head mounted display device 202. In some examples, head mounted display device 202 includes Figures 1A to 1G For example, the head mounted display device 202 may include (one or more) processors 103, (one or more) memory devices 105, imaging sensors 104, applications 116, image analyzers 106, existing contacts 130, interactive communication detectors 108, conversation graph builders 110, conversation graphs 138, cluster detectors 112, displays 114, display managers 140, and Figures 1A to 1GThe voice action manager 142 is configured to:

[0072] In some examples, computing system 200 includes head mounted display device 202 and computing device 201. Computing device 201 can be connected to head mounted display device 202 via wireless connection 275. In some examples, computing device 201 includes a mobile computing device, such as a smart phone, a tablet computer, a laptop computer, or other wearable device. In some examples, computing device 201 includes a server computer. The server computer can be a computing device in the form of multiple different devices, such as a standard server, a group of such servers, or a rack server system. In some examples, the server computer can be a single system that shares components such as a processor and memory. In some examples, wireless connection 275 is a short-range communication link such as a near field communication (NFC) connection or a Bluetooth connection. In some examples, wireless connection 275 is a network connection such as a local area network (LAN), a wide area network (WAN), a cellular network, a satellite network, or other types of data networks. In some examples, wireless connection 275 may include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and / or send data within a network.

[0073] In some examples, Figures 1A to 1G Some components of the computing system 100 are included in the head mounted display device 202, and Figures 1A to 1G Some components of computer system 100 are included in computing device 201. In some examples, sensor components are included in head mounted display device 202, such as Figure 1A The head mounted display device 202 may include an imaging sensor 104 and a microphone 136. Figures 1A to 1G display 114 , processor(s) 103 , memory(s) 105 , and applications 116 .

[0074] In some examples, Figure 1AThe image analyzer 106, the interactive communication detector 108, and the conversation graph builder 110 are included in the head mounted display device 202. In some examples, one or more of the image analyzer 106, the interactive communication detector 108, or the conversation graph builder 110 are included in the computing device 201, where the sensor data (e.g., the image data 113 and the audio data 134) are sent to the computing device 201 via the wireless connection 275, and the computing device 201 builds the conversation graph 138. In some examples, the cluster detector 112 is included in the head mounted display device 202. In some examples, the cluster detector 112 is included in the computing device 201, where the conversation graph 138 can be periodically sent to the computing device 201 via the wireless connection 275 to detect the cluster groups 160. In some examples, Figure 1A The display manager 140 is included in the head mounted display device 202. In some examples, Figure 1A The display manager 140 is included in the computing device 201. In some examples, the voice action manager 142 is included in the head mounted display device 202. In some examples, the voice action manager 142 is included in the computing device 201.

[0075] Figure 3 An example of a head mounted display device 302 according to an aspect is shown. The head mounted display device 302 may be Figures 1A to 1G The computing system 100 and / or Figure 2 An example of a head mounted display device 202. The head mounted display device 302 includes smart glasses 387. Smart glasses 387 are glasses that add information (e.g., projecting a display 314) alongside what the wearer is viewing through the glasses. In some examples, instead of projecting information, the display 314 is an in-lens microdisplay. In some examples, the smart glasses 387 (e.g., spectacles or ophthalmic glasses) are visual aids that include a lens 390 (e.g., a glass or hard plastic lens) mounted in a frame 389 that typically holds the lens 390 in front of a person's eye with a bridge 393 on the nose and legs 391 (e.g., temples or temple pieces) that rest on the ears. The smart glasses 387 include an electronic assembly 395 that includes the circuitry of the smart glasses 387. In some examples, the electronic assembly 395 includes a housing that surrounds the smart glasses 387. Figures 1A to 1G Components of computing system 100 and / or Figure 2 Components of the head mounted display device 202. In some examples, the electronic component 395 is included or integrated into one of the legs 391 of the smart glasses 387 (or both of the legs 391).

[0076] Figure 4 It is a depiction Figures 1A to 1G400 of an example operation of the computing system 100. Figure 4 The flowchart 400 is relative to Figures 1A to 1G 100, but flowchart 400 may be applicable to any embodiment discussed herein, including Figure 2 The computing system 200 and / or Figure 3 head mounted display device 302. Although Figure 4 The flowchart 400 of FIG. 4 shows the operations in a sequential order, but it will be understood that this is merely an example and that additional or alternative operations may be included. In addition, Figure 4 The operations and related operations may be performed in an order different from that shown, or in a parallel or overlapping manner.

[0077] Operation 402 includes detecting, by at least one imaging sensor 104 of computing system 100, a facial feature of an entity. Operation 404 includes determining that the facial feature corresponds to an existing contact 130 associated with a user of computing system 100. In some examples, flowchart 400 does not include operation 404, but instead, after detecting the facial feature, operations proceed to operation 406.

[0078] Operation 406 includes detecting an interactive communication 109 between the user and the entity based at least on the image data 113 from the at least one imaging sensor 104. Operation 408 includes updating the conversation map 138 in response to detecting the interactive communication 109 between the user and the entity.

[0079] Operation 410 includes detecting one or more cluster groups 160 of entities over time based on information included within conversation map 138. Operation 412 includes managing content 111 displayed on display 114 of computing system 100 based on conversation map 138.

[0080] Figure 5 It is a depiction Figures 1A to 1G 500 of an example operation of the computing system 100. Figure 5 The flowchart 500 is relative to Figures 1A to 1G 100, but flowchart 500 may be applicable to any embodiment discussed herein, including Figure 2 The computing system 200 and / or Figure 3 head mounted display device 302. Although Figure 5 The flowchart 500 of FIG. 5 shows the operations in a sequential order, but it will be understood that this is merely an example and that additional or alternative operations may be included. In addition, Figure 5 The operations and related operations may be performed in an order different from that shown, or in a parallel or overlapping manner.

[0081] Operation 502 includes obtaining first image data by a first imaging sensor 104a. Operation 504 includes determining whether a facial feature is detected in the first image data.

[0082] Operation 506 includes obtaining, by the second imaging sensor 104b, second image data in response to the facial feature being determined to be detected in the first image data, wherein the second image data has a higher resolution than the first image data. Operation 508 includes determining that the facial feature corresponds to an existing contact 130 based on the second image data.

[0083] Operation 510 includes detecting interactive communication 109 between a user and an entity based on at least one of the first image data or the second image data. Operation 512 includes updating a conversation map 138 in response to detecting interactive communication 109 between the user and the entity. Operation 514 includes managing content 111 displayed on a display 114 of the computing system 100 based on the conversation map 138.

[0084] Figure 6 It is a depiction Figures 1A to 1G 600 of an example operation of the computing system 100. Figure 6 The flowchart 600 corresponds to Figures 1A to 1G 100, but flowchart 600 may be applicable to any embodiment discussed herein, including Figure 2 The computing system 200 and / or Figure 3 head mounted display device 302. Although Figure 6 Flowchart 600 of the embodiment shows the operations in a sequential order, but it will be understood that this is merely an example and additional or alternative operations may be included. Figure 6 The operations and related operations may be performed in an order different from that shown, or in a parallel or overlapping manner.

[0085] Operation 602 includes detecting a notification 132. Operation 604 includes assigning a priority 162 to the notification 132 based on the conversation map 138. Operation 606 includes determining whether to display the notification 132 based on the priority 162. Operation 608 includes displaying the notification 132 on a display 114 of the computing system 100 in response to the priority 162 being greater than a threshold level.

[0086] Figure 7 It is a depiction Figures 1A to 1G Flowchart 700 of example operations of computing system 100. Figure 7 Flowchart 700 is relative to Figures 1A to 1G 100, but flowchart 700 may be applicable to any embodiment discussed herein, including Figure 2 The computing system 200 and / or Figure 3 head mounted display device 302. Although Figure 7 Flowchart 700 of the embodiment shows the operations in a sequential order, but it will be understood that this is merely an example and additional or alternative operations may be included. Figure 7 The operations and related operations may be performed in an order different from that shown, or in a parallel or overlapping manner.

[0087] Operation 702 includes receiving information from a user via an application 116 executable by the computing system 100. Operation 704 includes identifying one or more entities from the conversation graph 138 based on the information. Operation 706 includes determining a ranking of the entities from the conversation graph 138 based on the information included within the conversation graph 138. Operation 708 includes displaying the one or more entities as suggested entities 135 on an auto-complete interface of the application 116 according to the ranking.

[0088] Figure 8 It is a depiction Figures 1A to 1G 800 of an example operation of the computing system 100. Figure 8 Flowchart 800 is relative to Figures 1A to 1G 100, but flowchart 800 may be applicable to any embodiment discussed herein, including Figure 2 The computing system 200 and / or Figure 3 head mounted display device 302. Although Figure 8 Flowchart 800 of FIG. 8 shows operations in sequential order, but it will be understood that this is merely an example and additional or alternative operations may be included. In addition, Figure 8 The operations and related operations may be performed in an order different from that shown, or in a parallel or overlapping manner.

[0089] Operation 802 includes detecting a message 133 to be delivered to the computing system 100. Operation 804 includes assigning a priority 163 to the message 133 based on the conversation map 138. Operation 806 includes delaying delivery of the message 133 in response to the priority 163 being below a threshold level. Operation 808 includes delivering the message 133 in response to the priority 163 being above the threshold level.

[0090] Fig. 9 It is a depiction Figures 1A to 1G 900 of an example operation of the computing system 100. Fig. 9 Flowchart 900 is relative to Figures 1A to 1G 100, but flowchart 900 may be applicable to any embodiment discussed herein, including Figure 2 The computing system 200 and / or Figure 3 head mounted display device 302. Although Fig. 9Flowchart 900 of the embodiment shows the operations in a sequential order, but it will be understood that this is merely an example and additional or alternative operations may be included. Fig. 9 The operations and related operations may be performed in an order different from that shown, or in a parallel or overlapping manner.

[0091] Operation 902 includes receiving a voice command 144 via the microphone 136. Operation 904 includes identifying an entity associated with the voice command 144 based on the conversation graph 138. Operation 906 includes performing an action using the identified entity.

[0092] In addition to the above description, controls may be provided to the user, allowing the user to choose whether and when the systems, programs, or features described herein enable the collection of user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current location), and whether to send content or communications from a server to the user. In addition, certain data may be processed in one or more ways before being stored or used to remove personally identifiable information. For example, the user's identification may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be summarized (such as to a city, zip code, or state level) where location information is obtained so that the user's specific location cannot be determined. Thus, the user may have control over what information is collected about the user, how the information is used, and what information is provided to the user.

[0093] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0094] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0096] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0097] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0098] In this specification and the appended claims, unless the context clearly dictates otherwise, the singular forms "a", "an", and "the" do not exclude plural references. In addition, unless the context clearly dictates otherwise, conjunctions such as "and", "or", and "and / or" are inclusive. For example, "A and / or B" includes A alone, B alone, and A and B. In addition, the connecting lines or connectors shown in the various figures presented are intended to represent exemplary functional relationships and / or physical or logical couplings between various elements. Many alternative or additional functional relationships, physical connections, or logical connections may exist in actual devices. In addition, unless an element is specifically described as "essential" or "critical", no item or component is essential for the practice of the embodiments disclosed herein.

[0099] Terms such as, but not limited to, approximately, substantially, generally, etc. are used herein to indicate that precise values ​​or ranges of specified terms are not required and need not be specified. As used herein, the terms discussed above will have their ready and immediate meanings to those of ordinary skill in the art.

[0100] Furthermore, the use of terms such as upward, downward, top, bottom, side, end, front, rear, etc. herein is used with reference to the orientation currently considered or shown. If they are considered relative to another orientation, it should be understood that these terms must be modified accordingly.

[0101] Moreover, in this specification and the appended claims, the singular forms "a", "an", and "the" do not exclude plural references unless the context clearly dictates otherwise. Moreover, conjunctions such as "and", "or", and "and / or" are inclusive unless the context clearly dictates otherwise. For example, "A and / or B" includes A alone, B alone, and A and B.

[0102] Although certain example methods, devices, and articles of manufacture have been described herein, the scope of coverage of this patent is not limited thereto. It should be understood that the terminology employed herein is for the purpose of describing particular aspects and is not intended to be limiting. On the contrary, this patent covers all methods, devices, and articles of manufacture that fall fully within the scope of the claims of this patent.

Claims

1. A method for managing display content, comprising: Obtaining first image data by a first imaging sensor of the wearable device; In response to detecting a facial feature of an entity in the first image data, obtaining second image data by a second imaging sensor, the second image data having a higher resolution than the first image data; detecting interactive communication between a user of the wearable device and the entity based on the second image data; updating a conversation graph in response to detecting the interactive communication between the user and the entity; and Content for display on the wearable device is managed based on the session graph.

2. The method according to claim 1, wherein: The entity is a first entity, and the method further comprises: receiving a voice command from the user via a microphone of the wearable device; identifying a second entity associated with the voice command based on the conversation graph; and An action using the second entity is performed.

3. The method according to claim 1, further comprising: It is determined that the facial feature corresponds to an existing contact associated with a user of the wearable device.

4. The method according to claim 1, wherein: The interactive communication is detected based on the first image data or the second image data and audio data received via a microphone of the wearable device.

5. The method according to claim 1, wherein: The conversation graph includes a plurality of nodes, the plurality of nodes including a first node and a second node, the first node represents the user of the wearable device, the second node represents the entity, and the conversation graph includes a link connected between the first node and the second node.

6. The method according to claim 5, further comprising: calculating a familiarity metric based at least on a frequency of detected interactive communications between the user and the entity; as well as The link or the second node is annotated with the familiarity metric.

7. The method according to claim 5, wherein: The updates include: A familiarity metric is adjusted in response to detecting the interactive communication between the user and the entity.

8. The method according to claim 7, wherein: The familiarity metric is also adjusted based on detected virtual communications between the user and the entity.

9. The method according to claim 5, wherein: The updates include: In response to detecting the interactive communication between the user and the entity, creating a new node in the conversation graph, the new node representing the entity; and A new link is created in the conversation graph between the new node and the first node, the new link or the new node being annotated with a familiarity metric representing a familiarity level between the user and the entity.

10. The method according to claim 5, further comprising: detecting a cluster group of entities among a plurality of entities based on information included in the conversation graph; identifying that the content is associated with an entity; as well as Whether to display the content on the wearable device is determined based on whether the entity associated with the content is included in the entity cluster group.

11. The method according to claim 1, wherein: The management includes: detecting a notification generated by an application executable by the wearable device; assigning priorities to the notifications based on the conversation graph; and A determination is made based on the priority whether to display the notification on the wearable device.

12. The method according to claim 11, further comprising: determining that the notification is related to an entity included in the session graph, Wherein the assigning comprises assigning a priority to the notification that is higher than a priority assigned to a notification associated with an entity not included in the conversation graph.

13. The method according to claim 1, wherein: The management includes: receiving information from the user via an application executable by the wearable device; identifying at least one entity from the conversation graph based on the information; and The at least one entity is provided as a suggested entity for display on an interface of the application.

14. The method according to claim 13, further comprising: A ranking of the plurality of entities is determined from the conversation graph based on a familiarity metric associated with one or more of the plurality of entities, wherein one or more of the plurality of entities are displayed as suggested entities based on the ranking.

15. The method according to claim 1, further comprising: detecting a message to be delivered to the wearable device; assigning a priority to the message based on the conversation graph; delaying the message or not delivering the message in response to the priority being below a threshold level; and The message is delivered in response to the priority being above the threshold level.

16. The method according to any one of claims 1 to 15, wherein: The wearable device includes smart glasses.

17. A non-transitory computer-readable medium storing executable instructions, the executable instructions, when executed by at least one processor, being configured to cause the at least one processor to perform operations comprising: Obtaining first image data by a first imaging sensor of the wearable device; In response to detecting a facial feature of an entity in the first image data, obtaining second image data by a second imaging sensor, the second image data having a higher resolution than the first image data; detecting interactive communication between a user of the wearable device and the entity based on the second image data; updating a conversation graph in response to detecting the interactive communication between the user and the entity, the conversation graph comprising a plurality of nodes representing entities and links connecting between the plurality of nodes, wherein the executable instructions for updating the conversation graph include instructions for calculating a familiarity metric associated with the entity and annotating a node or a link with the familiarity metric; and Content for display on the wearable device is managed based on the session graph.

18. The non-transitory computer readable medium of claim 17, wherein: The operations further include: Determine that the facial feature corresponds to an existing contact associated with a user of the wearable device, wherein, in response to the facial feature being determined to correspond to the existing contact, the second image data and audio data received via a microphone of the wearable device are used to detect the presence of the interactive communication.

19. The non-transitory computer readable medium according to any one of claims 17 to 18, wherein: The operations further include: receiving a voice command from the user via a microphone of the wearable device; identifying an entity associated with the voice command based on the conversation graph; and Perform an action using the entity.

20. A wearable device comprising: a first imaging sensor configured to receive first image data; a second imaging sensor configured to receive second image data having a higher resolution than a resolution of the first image data; an image analyzer configured to detect the presence of a facial feature of an entity in the first image data and, in response to the facial feature being detected in the first image data, activate the second imaging sensor to obtain the second image data, the image analyzer configured to use the second image data to determine that the facial feature corresponds to an existing contact associated with a user of the wearable device; an interactive communication detector configured to detect interactive communication between the user and the entity based on at least one of the first image data or the second image data; a conversation graph builder configured to update a conversation graph in response to detecting the interactive communication between the user and the entity; as well as A display manager is configured to manage content displayed on a display of the wearable device based on the session graph.

21. The wearable device according to claim 20, wherein: The conversation graph includes a plurality of nodes representing entities and links connecting the plurality of nodes, wherein the conversation graph builder is configured to calculate a familiarity metric associated with the entity and annotate the node or the link with the familiarity metric.

22. The wearable device according to claim 20, wherein: The display manager includes a notification manager configured to manage display of notifications based on the session graph, wherein the wearable device further includes: a cluster detector configured to detect a cluster group of entities among a plurality of entities based on information included in the conversation graph; The notification manager is configured to identify that a notification is associated with an entity and assign a priority to the notification based on whether the entity associated with the notification is included in the entity cluster group.

23. A non-transitory computer-readable medium storing executable instructions, the executable instructions, when executed by at least one processor, being configured to cause the at least one processor to perform operations comprising: receiving first image data from a first imaging sensor of the wearable device; In response to detecting a facial feature of an entity in the first image data, receiving second image data from a second imaging sensor, the second image data having a higher resolution than the first image data; detecting interactive communication between the user and the entity based on the second image data and audio data received via a microphone of the wearable device; updating a conversation graph in response to detecting the interactive communication between the user and the entity, the conversation graph comprising a plurality of nodes representing entities and links connecting between the plurality of nodes, wherein the executable instructions for updating the conversation graph include instructions for calculating a familiarity metric associated with the entity and annotating a node or a link with the familiarity metric; and Content for display on the wearable device is managed based on the session graph.

Images