User-centric contextual information for browsers
By using user-centric graphs to identify context information in the browser and providing auxiliary information, the problem of the browser failing to effectively identify context is solved, and the user task completion efficiency and browser performance are improved.
Patent Information
- Application Number
- CN201980064675.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-10-04
- Filing Date
- 2019-09-04
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-09-04
AI Technical Summary
A typical browser fails to effectively identify and present contextual information related to the user's current task, making it difficult for users to quickly find the information they need in a browsing session, increasing the time for task completion and network traffic.
By maintaining a user-centric graph, identifying the context in which the user interacts with the browser application, and providing auxiliary information to the user based on the facts in the graph, reducing the query needs for multiple computer services, and achieving intelligent and automatic presentation of information .
It improves the efficiency of user tasks and browser computing efficiency, reduces the steps of user interaction with the browser and network traffic, and presents relevant information in advance by predicting user needs, improving the user experience.
Smart Images

Figure CN112840335B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to providing contextual information to a user during a browsing session. Background Art
[0002] Users often use browser software to access information sources (e.g., networks, computer file systems, databases, repositories, etc.) to complete various tasks and engage in various activities. However, typical browsers are not configured to recognize usage context (e.g., details about the current task or activity) at a rich level of granularity in order to present information relevant to the task of interest. Instead, typical browsers present information relevant to the current document (e.g., web page, computer file, or database entry) being accessed without particular consideration of the usage context. Summary of the invention
[0003] This disclosure is provided to introduce in simplified form a selection of design concepts further described below in the detailed description. This disclosure is not intended to identify key features or important features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. In addition, the claimed subject matter is not limited to implementations that address any or all of the shortcomings noted in any part of this disclosure.
[0004] A method for providing contextual information to a user during a browsing session, comprising: maintaining a user-centric graph, the graph comprising a plurality of user-centric facts, the facts being associated with the user and derived from the user's interactions with a plurality of different computer services. The method further comprises: identifying a context in which the user interacts with a browser application. The method further comprises: identifying auxiliary information related to the context, and displaying the auxiliary information to the user. The auxiliary information may be based at least on one or more user-centric facts in the user-centric graph. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] Figure 1A An exemplary computer system for providing user-centric contextual information in a browser is shown. Figure 1B Another example of a user context in which the browser provides user-centric context information is shown.
[0006] Figure 2 A method of providing user-centric contextual information in a browser is shown.
[0007] Figure 3A-3B An exemplary user-centric graph data structure for storing a collection of user-centric facts is shown.
[0008] Figure 4An exemplary computer system is shown. DETAILED DESCRIPTION
[0009] The present disclosure relates to a method for a browser program to provide contextual information to a user, such as to assist the user during a browsing session. Non-limiting examples of browser programs include desktop computers, networks, mobile phones, and / or other computer services that are configured to browse and access multiple content items, such as multimedia content, web pages, or any other data. Non-limiting examples of browsing may include: 1) viewing a list of content organized in any suitable manner (e.g., organized by file location or URL), 2) searching by content and / or metadata (e.g., by content type, by natural language content, by text strings and / or styles, by image content, etc.), 3) accessing files in a list and / or search (e.g., displaying images in search results), and / or 4) accessing / viewing curated, suggested, or otherwise retrieved content. For example, browser programs include web browsers, file system browsers, and browsing software configured to access library records, medical records, financial data, news archives, and / or any other suitable information that can be stored / organized in a browsable manner.
[0010] The present disclosure improves the technology of browser programs, including improvements to browser UI and improvements to browser backends. Specifically, the present disclosure includes a method for causing a browser to present auxiliary information related to the user's current interactive context to a user, and the auxiliary information may be useful for various reasons, such as helping the user find information of interest, engaging in activities of interest, and / or using the browser to complete a task. By presenting relevant information that can promote task completion to the user, the efficiency of the user's task completion and the computing efficiency of the browser can be improved. For example, by using the context information presented by the browser to replace extensive additional search / browsing, the user may be able to complete the task faster and / or interact with the browser less. In some examples, the auxiliary information may be related to the potential future steps of the task, so the user may be able to plan ahead and complete the task more efficiently by considering the information related to the future steps. Similarly, fewer computing steps and / or less network communications may be required by the time the user completes the task, for example, the browser may be able to batch related information from multiple different sources together and efficiently transmit the batched information to the user, instead of issuing multiple requests by the user to obtain each individual information fragment. In addition, according to the present disclosure, information related to a user's interaction with different computer services can be aggregated in a common user-centric graph, thereby reducing the need to access relevant information for multiple different queries (e.g., associated with different computer services). Aggregation of relevant information from multiple different computer services can enable intelligent, automatic presentation of auxiliary information, which the user can use instead of performing multiple manual queries associated with multiple different computer services. For example, if a user's task can be assisted by receiving the top search results for each of 3 different web searches, the browser may be able to anticipate that the user may perform each of the 3 web searches in the near future in order to present the 3 top search results, rather than waiting for the user to perform each of the 3 different web searches and presenting other less relevant results for each search. Figure 1A An exemplary computing system 100 including a client computer 102 is shown. The client computer 102 may be any suitable computing device, such as a desktop computer or a mobile phone device. The client computer 102 is configured to instantiate a browser 110. Figure 1AAs shown, the browser 110 can be provided as a GUI program that includes visual elements for viewing / navigating information sources. When the browser 110 is to be provided as a GUI, the client computer 102 may include and / or be operably coupled to one or more display devices configured to visually present the browser 110. Although examples are discussed in terms of a browser including a GUI for display, in other examples, the browser 110 may include other user interface elements (e.g., text, natural language, voice audio, and / or a user interface configured to interact with a user via any other suitable state-of-the-art and / or future input device).
[0011] For example, in Figure 1A , browser 110 is a web browser displaying two content tabs 111 (titled to indicate a page related to hotels) and 112 (titled to indicate a page related to flights). Although the example is discussed in terms of a web browser, browser 110 may alternatively or additionally include any other kind of browser, such as a file system browser. Content area 113 shows visual content associated with the first content tab 111, namely, reservation information for a hotel, including a reservation date calendar 114 and a "Book Now!" button 115. Browser 110 also includes an auxiliary information button 120, which is configured to indicate when contextual information is available to assist the user, and can be selected to display an information pane 121 configured to display the contextual information. The contextual information displayed in information pane 121 may include any suitable information for assisting the user in performing tasks and / or activities that they may be engaged in. For example, in Figure 1A , information pane 121 shows contextual information related to the user's travel plans, which can assist the user in determining available booking dates using calendar 114. Information pane 121 may additionally or alternatively show Figure 1A For example, when the context is related to travel, the information pane 121 can display more information related to future and / or historical trips. In an example, the user may have visited the same place on a previous trip last year. Therefore, the information pane 121 can present relevant information from the previous trip. For example, the information pane 121 can include automatic suggestions to go to a website associated with a hotel where the user stayed during the previous trip. For example, the information pane 121 can display one or more photos and / or links to photo / video albums associated with the user's previous trips.
[0012] In order to provide relevant auxiliary information to facilitate easier task completion, the computing system 100 intelligently identifies the context of user activities. Non-limiting examples of context include 1) tasks and / or activities (e.g., current tasks that the user is working on, recent tasks that the user has completed, and / or future tasks that the user has scheduled and / or committed to); 2) the status of ongoing and / or long-term commitments (e.g., work, courses, and / or projects); 3) a collection of related tasks, activities, websites, information summaries, web searches, collaborators' contact information, documents, etc. related to a particular work (e.g., R&D, academic or scholarly research, etc.); and / or 4) predefined topics of interest to the user and / or topics identified based on the user's recent, current, and / or historical activities (e.g., favorite TV shows, current research topics).
[0013] The computing system 100 can use the user's personal information generated by the user's interaction with the browser and / or with any of a plurality of other computer services to identify the user's current context. For example, the user's current context can be identified based on a collection of user-centric facts derived from the user's interaction with the browser and / or other computer services. Similarly, the listed auxiliary information provided to the user in response to the identified context can be extracted from any combination of the user's interaction with the browser and / or other computer services. "Computer service" herein may refer to software applications, hardware and / or software products executable on a computer system (e.g., including a networked computing system, cloud computing, etc.), and / or any other machine or process configured to provide interactive and / or non-interactive services to one or more users to assist users, edit, convert, store and / or maintain data, interact with other computer services, and the like. Other computer services may include web-based, desktop and / or mobile phone applications, smart home devices / "internet of things" devices, smart assistant devices, wearable devices, background services, and the like. Non-limiting examples of computer services from which personal information may be derived include: 1) intelligent assistant programs; 2) email programs; 3) office suite programs (e.g., word processors, slideshows, spreadsheet programs); 4) calendar programs; 5) teleconferencing programs; 6) task tracking programs; 7) health / activity / nutrition tracking programs; 8) multimedia programs (e.g., television / video, music, and / or video game content delivery programs); 9) mapping / publishing programs; 10) integrated development environments for computer programming; 11) social networking services; 12) workplace collaboration environments; 13) cloud data storage and file synchronization programs.
[0014] As noted above, the user browser context and the auxiliary information provided in response to the context can be derived from user-centric facts from various sources, for example, from data sources associated with one or more different computer services. Therefore, the computer system 100 may include a user-centric graph server 103, which is configured to instantiate, maintain and / or store a collection of user-centric facts as a user-centric graph data structure 300. The user-centric graph structure 300 represents multiple user-centric facts in the form of nodes and edges in a graph data structure. The structural features of the user-centric graph structure 300 can indicate facts and / or possible relationships between facts. The user-centric graph structure 300 can represent each fact as a subject-predicate-object triple, which includes a subject graph node representing the subject (e.g., entity, event, or any other noun) of the fact, an object graph node representing the object of the fact, and an edge representing a predicate describing the relationship between the subject and the object. In addition, the user-centric graph structure can represent the relationship between facts, for example, two or more facts can share a common subject and / or object. Relationships between subjects, objects, and facts in the graph may be encoded as nodes, edges, and / or paths in the graph, which may represent common entities, temporal and / or topical relationships indicating facts, and any other suitable relationships. The user-centric graph structure may be used to draw inferences from one or more facts, e.g., by traversing the graph along paths formed by edges of user-centric facts.
[0015] The user-centric graph can be used by any computer service, for example, to store user-centric facts generated by a user's interaction with a computer service, and / or to query the user-centric graph to identify one or more user-centric facts related to the interaction with the computer service or any other computer service. A computer service that uses the user-centric graph can be a first-party computer service created and / or managed by an organization or entity that manages the user-centric graph, or can be a third-party service created and / or managed by a different organization or entity. A computer service can use the user-centric graph via one or more APIs (e.g., an update API and a query API) of the user-centric graph, where each API can be used by multiple different computer services, including first-party computer services and third-party services.
[0016] A user may interact with a computer service using more than one different device, e.g., via a user interface of a mobile device, a desktop computer, a smart assistant speaker device, etc. User-centric facts may be aggregated from the computer service at each different device with which the user interacts. For example, a user may interact with a particular computer service (e.g., an email program) at more than one different device (e.g., via a mobile phone with a mobile phone email application, and via a personal computer device via a web browser-based email client). Thus, various computer services may use and / or contribute to the user-centric graph data structure 300 regardless of which particular device the user may use to interact with the computer service. For example, although in Figure 1A 10 is shown in association with the client computer 102, but it should be understood that the browser 110 can be instantiated on any suitable device, such as a personal computer, a mobile phone, an intelligent assistant device, etc. Therefore, the browser 110 can present auxiliary information based on user-centric facts regardless of which device instantiates the browser 110. Because any suitable computer service running on any suitable device can use the user-centric graph data structure 300 to derive auxiliary information, the presentation of the auxiliary information is independent of the specific device on which the browser 110 and / or any other computer service is instantiated. Therefore, the auxiliary information can be presented to the user in a consistent manner on different devices running the browser 110 or any other browser program.
[0017] The user-centric graph server 103 and / or the client computer 102 may be connected via the network 104 to one or more user-centric data storage units, e.g., user-centric data 105A, 105B, and optionally to any suitable number of other additional user-centric data storage units via user-centric data 105N. Each user-centric data storage unit may store any suitable data related to the user, e.g., data generated as a result of interaction with one or more computer services. For example, each user-centric data storage unit may be an application-specific data storage unit that stores data related to the application, e.g., user data, configuration settings, etc.
[0018] The user-centric graph data structure 300 includes a plurality of different constituent graph structures, e.g., constituent graph structure 302A, constituent graph structure 302B, etc. Each constituent graph structure may be an application-specific constituent graph structure associated with a different computer service, e.g., graph structure 302A associated with user-centric data from a first computer service stored in user-centric data 105A. For example, application-specific constituent graph structure A may be associated with a scheduling program, while application-specific constituent graph structure B may be associated with an email program. Multiple constituent graphs may be viewed as a single graph by viewing the edges between constituent graphs as edges in a single larger graph data structure 300. To simplify the description, the depicted graph includes only twelve nodes. In an actual implementation, a user-centric graph will include more (e.g., hundreds, thousands, millions, or more) nodes distributed between more constituent graphs.
[0019] The user-centric graph server 103 may also be connected to any other suitable computer device and / or computer network via the network 104, such as a computer on the Internet, such as a web search provider, a website host, etc. In some examples, the user-centric graph data structure 300 may be configured to include one or more external facts obtained from such other computer devices in addition to user-centric facts specifically related to the user. For example, the user-centric graph data structure 300 may include one or more facts derived from web search results and / or websites. When the user-centric graph data structure 300 includes external facts, the external facts may be related to the user-centric facts in the user-centric graph in any suitable manner, for example, the user-centric graph may include one or more nodes and / or edges representing global data derived from the Internet, which are connected via paths in the graph to one or more nodes and / or edges representing user-centric data generated due to the user's interaction with the computer service. In some examples, the connection between the user-centric facts and the external facts may create additional / enhanced user-centric facts.
[0020] Client computers 102 may communicate with user-centric graph server 103 via network 104 to utilize information represented in the user-centric graph to assist the user. For example, such assistance may include answering questions directed to and / or about the user by retrieving user-centric facts and / or external facts, finding historical data related to interactions between the user and one or more computer services, and / or inferring new facts to add to the user-centric graph.
[0021] like Figure 1AAs shown, the browser 110 can use the user's personal information from one or more emails and / or calendar / schedule items, as shown in the information pane 121. The information pane 121 can show information related to any one or more suitable contexts (e.g., information related to tasks that the user may wish to complete). The context can be based on one or more user-centric facts in the user-centric graph. For example, the context can be based on a cluster of user-centric facts that are interconnected by edges within the user-centric graph. Alternatively or additionally, the context can be based on one or more user-centric facts within the user-centric graph that comply with specific constraints, such as matching a specific topic, relating to a specific entity or event, or having a specific temporality (e.g., facts describing events that occur within a specific period or schedule or any other suitable temporal relationship within a specific time period). Non-limiting examples of user-centric facts that may be included in the context are user-centric facts describing: 1) emails sent to or from the user; 2) the user's contacts; 3) the user's calendar data; 4) documents (e.g., multimedia documents including photos, videos, or any other content that the user is curating, editing, and / or initiating); 5) web pages; and / or 6) location data associated with the user (e.g., location data measured by sensors of a device belonging to the user, and / or location data inferred from the user's interactions with a computer service); and / or 7) application usage data associated with the user (e.g., login data, usage statistics regarding frequency and duration of usage, etc.; for example, application usage data may include 110's browser usage data, including recent page visits, search history, etc.).
[0022] The context in the user-centric graph can indicate other more user-centric facts in the user-centric graph data structure 300, for example, user-centric facts that are connected to one or more user-centric facts in the context via edges / paths in the user-centric graph data structure 300. For example, a first user-centric fact in the context can be the fact that the user received a first email from an airline reservation confirmation email address. A second user-centric fact in the context can be the fact that the user received a second email from the same airline (e.g., as indicated by the second email being from the same or similar email address as the first email). Thus, the two user-centric facts can be connected in the user-centric graph based on the two emails being from the same airline. In addition to connections in the user-centric graph based on common identifiers (such as email addresses), other more examples of connections in the user-centric graph will be discussed below based on subject, time, and / or other constraints.
[0023] The user-centric graph data structure 300 may be traversed and / or processed to retrieve one or more user-centric facts related to the context. The traversal and / or processing of the user-centric graph data structure 300 may be based on the structure of the user-centric graph data structure 300 and / or the details of the user-centric facts within the context and / or related to the context.
[0024] For example, the user-centric graph data structure 300 can be traversed by starting with an initial set of one or more user-centric facts in context, performing a random walk through the user-centric graph data structure 300, and further traversing user-centric facts connected to those in the initial set to build a larger set of potentially related user-centric facts. The random walk through the graph can be weighted based on the relationships between the user-centric facts (e.g., details associated with subject graph nodes, object graph nodes, and / or edges of user-centric facts). For example, the user-centric facts can be annotated with confidence scores, which can be used to weight the random walk.
[0025] In addition to performing random walks based on confidence values associated with nodes and / or edges in the user-centric graph, the user-centric graph may be traversed and / or processed in any other suitable manner. Non-limiting examples of processing may include: 1) "ranking" user-centric facts in the graph according to a scoring function and retrieving relatively higher ranked user-centric facts (e.g., finding user-centric facts about the most relevant entities for a context; and 2) "slicing" the user-centric graph to filter user-centric facts that satisfy specific constraints (e.g., finding user-centric facts related to emails that are connected to a context). "Ranking," "slicing," and other ways to traverse / process a user-centric graph are described in more detail below.
[0026] The auxiliary information displayed in pane 121 can be based on user-centric facts in the context, and / or user-centric facts retrieved from user-centric graph data structure 300 based on the context. For example, the auxiliary information in pane 121 can include a natural language representation of each of one or more user-centric facts, and / or a natural language summary of one or more user-centric facts. In some examples, the auxiliary information can include user-centric facts related to completing tasks and / or topics related to the context. In some examples, the auxiliary information can include a summary of the user's historical activities related to the context.
[0027] In addition to being based on user-centric facts, the displayed auxiliary information may also be based on external facts and / or derived facts. Non-limiting examples of external facts and / or derived facts that the auxiliary information may be based on include: 1) web search results related to user-centric facts; 2) dictionary / encyclopedia / reference information related to user-centric facts; 3) website content related to user-centric facts; and / or 4) the results of processing user-centric facts according to programmatic, mathematical, statistical and / or machine learning functions (e.g., calculating dates or durations based on dates associated with user-centric facts in context, for example, to suggest possible arrangements for future tasks; using machine learning models to calculate natural language summaries or responses to queries; etc.). Non-limiting examples of information that may be present in the auxiliary information include information based on (e.g., summarized, inferred and / or otherwise determined from) one or more of the following: 1) emails sent to or from the user; 2) contacts of the user; 3) calendar data of the user; 4) documents; 5) web pages; 6) location data associated with the user; and / or 7) application usage data associated with the user.
[0028] The browser 110 may be configured to present information related to one or more of a plurality of predefined contexts, such as "travel" and "upcoming tasks." For example, Figure 1A As shown in , information pane 121 displays auxiliary information related to travel itineraries and upcoming tasks in the user's calendar. Alternatively or additionally, browser 110 may be configured to intelligently identify one or more contexts that are particularly relevant to the user based on the user's personal information. For example, even when browser 110 may not be configured to present contextual information about the topic of "travel", browser 110 may still identify "travel itinerary" as a relevant context for the user, for example, based on the user's email and / or calendar data.
[0029] By presenting relevant auxiliary information (including relevant flight dates sent to the user from an email, and upcoming scheduled tasks in the user's calendar) in response to the determined context, the browser can help the user determine a suitable hotel reservation date from the available dates in calendar 114 without the user having to separately look up the user's personal email and calendar data. Therefore, the user may be able to complete the selection in calendar 114 and confirm the selection through button 115 without having to divert attention to intermediate steps, such as opening an email program and finding the relevant email, and / or opening a calendar program and navigating to the relevant date. In this example, presenting flight dates and upcoming tasks can save the user several steps. For example, the user may know that they have a video conference scheduled for some time during their trip, and may need to plan to be in a hotel with wireless Internet during the scheduled time, but may not remember the exact date of the trip. Therefore, in order to plan a hotel reservation, the user may need to figure out the flight date (e.g., from the relevant email) and then check the calendar during the relevant date. As Figure 1A As shown in , computing system 100 can intelligently determine that a user may need relevant information from email / calendar and present contextually relevant auxiliary information in information pane 121, thereby allowing the user to avoid the steps of searching email for relevant flight reservations and then searching the calendar for the relevant duration.
[0030] In some examples, the context for triggering auxiliary information can be defined by a collection of user-centric facts related to the context, such as tasks, emails, contact information, websites, web searches, file / database searches, historical browser activities, historical activities in other applications, etc. In some examples, the collection of user-centric facts can include semantically related information content, for example, a collection of user-centric facts that define a context for a predefined topic can include user-centric facts that are each semantically related to the topic. For example, when the predefined topic is "travel", the collection of user-centric facts can include emails, web searches, etc., each of which includes semantic data and / or metadata related to travel. Non-limiting examples of semantic data include natural language content, image content, text, digital data, etc. Non-limiting examples of metadata include file / network / database location, publication date, access time, author information, sender / recipient information, and the like.
[0031] In some examples, context can be defined by a collection of user-centric facts related by temporal relationships, based on user-centric facts associated with relevant date / time information. A collection of user-centric facts that define context based on temporal relationships can be identified even if the collection of user-centric facts would not otherwise be identified as defining context based on tasks / activities, ongoing projects, predefined topics, and / or topics based on recent activities. Non-limiting examples of context based on temporal relationships include: 1) a co-occurring collection of websites, emails, etc. that users visit simultaneously or within a short duration of each other; 2) an activity with an associated scheduled time (e.g., a deadline for a project or scheduled appointment) together with one or more other activities that have occurred or are scheduled to occur soon before or after the scheduled time; 3) a recurring temporal context, for example, every evening, every Tuesday at 5:00 PM, or any other suitable scheduled and / or regularly occurring event.
[0032] For example, when a user is traveling, they may also need to attend a scheduled conference call. Therefore, the "travel" context may also include time-related information for the conference call schedule, because the scheduled dates for the conference call occur within the travel dates, even though the conference call schedule would not otherwise be considered semantically related to the travel itinerary.
[0033] Identifying context for triggering auxiliary information can include combining multiple different signals, such as metadata, semantic content, and temporal relationships, to assess the strength of relationships between different user-centric facts. If a collection of user-centric facts is related in more than one different way (e.g., similar topics as well as temporal relationships), it may represent a more important context for the user. In some examples, the presence of multiple signals can enable the identification of context that would not otherwise be identified.
[0034] Figure 1B Another example of a browser session is shown in which a user is currently editing a document in a browser 110 via an in-browser editor, as shown in tab 111' displaying "Document 1." In the exemplary session, the user has also recently interacted with a web search tool labeled "Web Search" in tab 112'. The currently displayed tab 111' includes page content 113' in the form of an editable interface. The editable interface may allow a user to edit and save the displayed content, for example, using a word processing GUI including a cursor 116 for selecting a location and / or range in the text. As shown, the "Document 1" content displayed in page content 113' includes user-defined text describing a "R&D Project" and includes various notes about the project logistics, for example, NOTE: Alice, Bob, and Carol are assigned to a project due on a date of 9 / 7, according to David's email.
[0035] Based on the page content 113' and the user's recent interactions with one or more other applications, the information button 120 may indicate that an information pane 121' of auxiliary information is available for the current context. The current context may be defined by user-centric facts that relate to any appropriate information about the user's interaction with the browser, such as content edited by the user in the page content 113', previous interactions between the user and the browser (e.g., web searches conducted via tabs 112', other historical website visits and / or web searches), and / or recent interactions between the user and other applications (e.g., other document editing applications, email applications, and / or calendaring applications). Specifically, the current context may be defined by one or more user-centric facts related to "R&D Project". As shown in the information pane 121', auxiliary information related to "R&D Project" is displayed based on such context, for example, to potentially help the user complete one or more tasks related to the R&D Project.
[0036] The auxiliary information displayed in the information pane 121' may include any appropriate information related to the context. For example, the information pane 121' includes: 1) a list of relevant documents that the user may want to view and / or edit to complete the R&D project; 2) relevant websites describing competitor methods; 3) relevant web searches related to competitor methods; 4) relevant tasks that the user may want to complete in order to complete the R&D project; 5) relevant emails related to the project; and 6) people related to the project that the user may want to contact.
[0037] For example, relevant emails may be determined by one or more user-centric facts that define context based on the user's interaction with an email application. Relevant documents may be determined by one or more user-centric facts in the context based on the user's interaction with a browser (e.g., interaction with a document editor that may be displayed in the browser, such as in page content 113') and / or interaction with any other computer service (e.g., with a dedicated word processor or slideshow program). Similarly, listed persons may be determined based on emails to and / or from the user, collaborators who also edit relevant documents, and / or persons listed in the user's contact list. By identifying context and relevant documents, tasks, websites, web searches, tasks, emails, and persons, the browser 110 may be able to facilitate easier completion of R&D projects because the user may be able to access relevant information from the information pane 121' rather than explicitly querying various information sources (e.g., documents, contact lists, emails, etc.). Therefore, the user is able to find and use relevant information from multiple different sources without having to divert attention from working on "Document 1" in the tab 111'.
[0038] Similarly, a set of user-centric facts that define a context of a topic based on recent activity may include user-centric facts that are each related to one or more common topics. Determining that a user-centric fact belongs in a set that defines a particular context may also be based on any other suitable information related to the user-centric facts, such as metadata and semantic content of the user-centric facts.
[0039] Figure 2 An exemplary method 200 for providing contextual information in a browser to assist a user is shown. The method 200 includes maintaining a user-centric graph including a plurality of user-centric facts associated with a user at 210. The user-centric facts may be derived from the user's interactions with one or more different computer services.
[0040] Method 200 may be performed by Figure 1A The method 200 may be performed by any suitable combination of devices. Figure 1AAny device that executes on one or more processors a program stored in Figure 1A The code is executed on one or more storage devices associated with any device. In addition, the operation may be performed in any suitable combination. Figure 1A 100 and / or the browser 110. For example, the browser 110 may include a graph user interface that is visually presented on a display device of the client computer 102 and a graphical user interface that is displayed by the client computer 102 and / or the browser 110. Figure 1A The operation of the browser 110 on the client computer 102 may generally include 1) collecting information about the interaction between the user and the browser 110, which information may be used to define and / or infer the context in the user-centric graph data structure 300; 2) sending the information to the user-centric graph data structure 300; Figure 1A one or more other devices (e.g., to a user-centric graph server 103); 3) receiving a response from the one or more other devices, wherein the response includes context-based auxiliary information; and 4) visually presenting the response in a GUI of the browser 110.
[0041] At 230, method 200 also includes: identifying the context of the user's interaction with the browser application. Optionally, at 232, the context is defined by at least one or more contextualized user-centric facts, which are derived from the user's interaction with one or more different computer services. Optionally, at 234, the identification of the context is triggered by identifying a triggering fact. The triggering fact can be any user-centric fact or any other fact (e.g., a fact derived from a website that the user is visiting). The context can be based on the triggering fact in any suitable manner. For example, the context can be defined by a single user-centric fact, namely the triggering fact. Alternatively or additionally, the context can be defined by one or more facts related to the triggering fact in any suitable manner (e.g., via a structural relationship of the graph, a common theme / task / entity / event, etc., as described above). For example, the context can be defined by a temporal relationship between the triggering fact and one or more other user-centric facts in the graph (e.g., based on a temporal relationship between a date associated with a user interaction that defines the triggering fact and / or other user-centric facts and an expiration date associated with the user-centric fact, etc.).
[0042] The triggering fact may be based on one or more previous interactions (e.g., recent and / or historical interactions) of the user with the browser or any other computer service. For example, the user-centric graph server 103 may be configured to receive, from the browser 110, a computer-readable description of one or more previous interactions of the user with the browser. Thus, the triggering fact may include such a computer-readable description of the previous interactions. Non-limiting examples of interactions between a user and a browser include: 1) metadata and / or content corresponding to recent file locations and / or websites visited by the user; 2) recent files and / or web search queries submitted by the user; 3) content entered by the user in a web form, editable document, or via any other input mechanism provided by the browser; 4) closing one or more tabs, windows, or other interface elements of the browser; 5) clicking on an information button to request information (e.g., clicking on button 120 of browser 110); 6) viewing location / search history in the browser; 7) adding a new bookmark page (e.g., "Favorites") and / or viewing a list of bookmarked pages; 8) interacting with a website or any other computer service (e.g., placing an order to purchase a product, making a reservation, sending and / or reading email via an email client accessible through the browser, etc.); and / or 9) previous interactions of the user with computer services other than a browser (e.g., an email or calendar program).
[0043] In some examples, the browser can continuously identify potential triggering facts that arise from user interactions with the browser in order to determine whether auxiliary information should be presented for each candidate triggering fact based on a context associated with the triggering fact. For example, identifying the context can include: evaluating a signal strength for a candidate triggering fact, the signal strength estimating the relevance of the triggering fact and / or a context associated with the triggering fact. The signal strength can be based on the triggering fact and / or a context associated with the triggering fact. In some examples, the signal strength can be evaluated based on a confidence value associated with the triggering fact and / or other user-centric facts in the context associated with the triggering fact.
[0044] In some examples, evaluating the signal strength may be based on one or more graph characteristics or other structural properties of the user-centric graph, such as the amount of clustering or connectivity of the context related to the triggering fact. Non-limiting examples of graph characteristics that may be suitable for evaluating signal strength include 1) connectivity, such as the number of connected components and / or k connected components, k being any suitable value such that k vertices need to be removed to disconnect k connected components; 2) path distance; 3) whether there is a ring; 4) graph strength and / or toughness; and / or any other suitable graph property. In some examples, the browser 110 may identify the triggering fact and display auxiliary information only for the candidate triggering fact and the context evaluated for a relatively high signal strength. For example, identifying the triggering fact may include: evaluating a signal strength score for the candidate fact, the signal strength score indicating the strength of the relationship between the candidate fact and the user-centric fact in the user-centric graph, and identifying the triggering fact in response to the signal strength score exceeding a threshold. Alternatively or additionally, identifying the triggering fact may include identifying the candidate triggering fact with the highest signal strength among multiple triggering facts that occur in the user's interaction with the browser as the triggering fact.
[0045] In an example, a context related to a predefined topic may be defined by at least one node in a user-centric graph having outgoing edges connected to one or more other nodes based on one or more other nodes respectively related to the predefined topic. In an example, a context defined by a topic may be identified based on structural properties of one or more user-centric facts in the user-centric graph (e.g., based on graph theoretic properties, such as user-centric facts forming dense clusters, or any other suitable graph theoretic properties as described above for evaluating the signal strength of a triggering fact).
[0046] At 240, method 200 also includes evaluating the context to determine auxiliary information related to the context. Optionally, at 242, the auxiliary information is based on at least one or more user-centric facts derived from the user's interaction with one or more different computer services. Optionally, at 244, the auxiliary information is based at least on a relationship in the user-centric graph between the triggering fact and one or more other user-centric facts in the user-centric graph. In some examples, after identifying the triggering fact, the user-centric graph data structure 300 can be updated to include the triggering fact. In some examples, in response to identifying the triggering fact, the user-centric graph data structure 300 can be updated to include one or more additional user-centric facts based on aggregating, summarizing, and / or otherwise analyzing other user-centric facts in the user-centric graph data structure 300. For example, if the user Figure 1AIf the user frequently interacts with travel-related websites as shown, the user-centric graph data structure 300 can be updated to include an additional user-centric fact that summarizes the user's interactions by stating that "the user often researches travel." The new user-centric fact can be related to other user-centric facts related to travel via paths in the user-centric graph data structure 300. The new user-centric fact can define a context that can be used to return auxiliary information to the user, such as a "travel" topic context with related auxiliary information, which includes the user's previous travel-related interactions and other general information related to travel (e.g., travel websites, travel-related web searches, etc.).
[0047] At 250, method 200 further includes displaying auxiliary information to the user. The auxiliary information may be displayed in any suitable manner. For example, Figure 1A-1B As shown, the auxiliary information can be displayed as a list of facts grouped into different categories, such as "travel itinerary", "upcoming tasks", "related documents", "websites", "web searches", "tasks", "emails", and "people" categories, as shown in FIG. Figure 1A-1B By identifying the user's context and displaying auxiliary information related to the context, the browser can facilitate easier completion of tasks related to the context. Therefore, method 200 can improve the efficiency and / or ease of use of one or more computers.
[0048] Figure 3A-3B Details of an exemplary graph data structure for representing a set of user-centric facts as a user-centric graph are shown. Figure 3A An exemplary graph data structure 300 is shown in more detail in the form of a computer-readable data format 300' for the graph data structure 300. The graph data structure 300 represents a collection of user-centric facts in the data format 300' or any other suitable data format. The user-centric facts can be associated with application-specific data distributed across multiple different computer services. The graph data structure 300 allows for centralized querying and is suitable for implementing a user-centric graph.
[0049] The graph data structure 300 may be distributed over one or more application-specific constituent graph structures (e.g., corresponding to different computer services), such as constituent graph structure 302A indicated in a first region surrounded by dashed lines A, constituent graph structure 302B indicated in a second region surrounded by different dashed lines, and Figure 3A Other constituent diagram structures not shown.
[0050] Each constituent graph structure includes a plurality of user-centric facts 304, such as the user-centric fact F stored in the constituent graph structure A.A.1 、F A.2 Etc., user-centric facts F stored in the graph structure B B.1 、F B.2 Etc. User-centric facts include subject graph nodes 306, object graph nodes 310, and edges 308 connecting the subject graph nodes to the object graph nodes.
[0051] Subject graph nodes and object graph nodes may be generically referred to as nodes. A node may represent any noun, where "noun" is used to refer to any entity, event, or concept, or any suitable application-specific information (e.g., details of a previous action performed by a user using a computer service). Similarly, "subject noun" and "object noun" are used herein to refer to a noun represented by a subject graph node or an object graph node, respectively. Representing a collection of user-centric facts as a graph data structure may facilitate manipulation and traversal of the graph data structure (e.g., in response to a query).
[0052] As shown in the graph data structure 300, it is instructive to visualize the collection of user-centric facts in a graph format. Nodes 352 are depicted as solid circles, while edges 354 are depicted as arrows. Solid circles with outgoing edges (arrows pointing away from the solid circle) describe subject graph nodes, while solid circles with incoming edges (arrows pointing toward the solid circle) describe object graph nodes. Multiple constituent graphs can be treated as a single graph by treating the edges between constituent graphs as edges in a larger graph. Therefore, Figure 3B A single combined graph data structure 300 including multiple constituent graph structures is shown. To simplify the illustration, the example graph data structure 300 includes only two constituent graphs, with a total of twelve nodes. In actual implementations, a user-centric graph will include many more (e.g., hundreds, thousands, millions, or more) nodes distributed among more constituent graphs.
[0053] Exemplary user-centric facts constituting the graph structure F A.1 Including the main graph node S A.1 , edge EA.1 and object graph node O A.1 For example, the subject graph node S A.1 can represent the user's employer, and the object graph node O A.1 May represent tasks assigned to a user by the user's employer. A.1 Can describe the main graph node S A.1 and object graph node O A.1 Any suitable relationship between them. In the example above, edge E A.1 It can represent the “new task assigned” relationship. User-centric facts F A.1The subject-edge-object triples of collectively represent the fact that the user's employer has assigned her a new task.
[0054] The subject graph node of a first fact may represent the same noun as the object graph node of a second, different fact. Thus, the subject graph node and the object graph node may be represented as a single node, e.g. Figure 3B Node 322 is shown with incoming and outgoing arrows. By recognizing that certain object graph nodes and subject graph nodes represent the same noun, the graph data structure may be able to represent user-centric facts as complex relationships between multiple different nouns, which can be visualized as paths on the graph data structure 300'. For example, when a particular node is an object graph node for a first fact and a subject graph node for a second, different fact, similar to a logical syllogism, it may be possible to draw an inference from the combination of the two facts.
[0055] A subject graph node of a first user-centric fact may represent the same noun as a subject graph node of a second different user-centric fact. When two different subject graph nodes represent the same noun, the graph data structure may identify the two subject graph nodes as a single node, e.g., Figure 3B 324 with two outgoing arrows in the subject graph. Although the two subject graph nodes can be represented as a single node, each outgoing edge of the node is different from each other. These outgoing edges can point to the same object node or different object nodes. In either case, the subject node can involve two different user-centric facts.
[0056] Similarly, an object graph node of a first user-centric fact may represent the same noun as an object graph node of a second, different user-centric fact. In other words, the same noun may be the object of multiple different user-centric facts, with different subject graph nodes and possibly edges representing different relationship types, e.g. Figure 3B Thus, the graph data structure may identify the two object graph nodes as a single node, which is shown at the same location in the graph data structure 300.
[0057] A particular pairing of a subject noun and an object noun may involve two or more different user-centric facts. For example, a subject noun and an object noun may be represented by a pair of nodes indicated at 328, which are connected by two different edges, e.g., representing different relationships between the subject graph node and the object graph node. In the example, the subject graph node corresponds to a user account (e.g., identified by an email address). In the same example, the object graph node may correspond to a second different user account. In this example, the first edge may represent a "send email to" relationship, while the second edge represents a different "scheduled meeting with..." relationship. Thus, the graph data structure includes two or more user-centric facts having the same subject noun and object noun.
[0058] In other examples, two nouns can be related to two different user-centric facts, but the roles between subject and object are exchanged. In other words, the first noun is the subject noun of the first fact and the second noun is the object noun of the first fact, while the first noun is the object noun of the second fact and the second noun is the subject noun of the second fact. For example, a pair of nouns representing "Alice" and "Bob" can be related to the first fact, i.e., "Alice arranged a meeting with Bob", and also to the second fact, i.e., "Bob arranged a meeting with Alice". In addition to exchanging the roles of subject and object, the two facts using these two nouns can have different types of edges, for example, "Alice" and "Bob" may additionally be related to the third fact, i.e., "Bob sent an email to Alice". More generally, nouns can be connected via edges so as to represent a relationship in any suitable manner, for example, with any number of edges between two nodes, which travel between the first node and the second node in any combination of forward and / or backward directions.
[0059] In still another example, a graph data structure may identify a noun from one constituent graph and a noun from a different constituent graph as a single node. This identification that two or more different nodes correspond to the same noun may be referred to herein as a "node cross-reference" between the two nodes. A subject graph node or an object graph node representing a particular noun may store node cross-references to other nodes of the same constituent graph structure or any other constituent graph structure.
[0060] Similarly, user-centric facts in a first constituent graph structure may define subject graph nodes in the first constituent graph structure, along with edges pointing to object graph nodes in a second, different constituent graph structure (referred to herein as "cross-reference edges"). The user-centric graph may represent connections between constituent graphs in any suitable manner, for example, by storing constituent graph identifiers (which indicate connections to object graph nodes in constituent graph structure B) and object graph node identifiers (which indicate specific object graph nodes in constituent graph structure B) in constituent graph structure A.
[0061] Node cross-references and cross-reference edges connect multiple constituent graph structures. For example, node cross-references and cross-reference edges can be traversed in the same manner as edges, thereby allowing the traversal of the graph data structure to traverse paths across multiple constituent graph structures. In other words, the graph data structure 300 and / or data format 300' promotes an overall artificial intelligence knowledge base that includes facts from different computer services and / or facts across different computer services. Node cross-references and cross-reference edges may be collectively referred to as cross-references in this article. Similarly, when a node involves cross-references within multiple constituent graph structures, the node may be referred to as a cross-reference that spans a constituent graph structure.
[0062] In addition, in addition to connecting two different constituent graph structures via cross-references, the constituent graph structure may also include one or more user-centric facts that relate to a subject graph node in the constituent graph structure and an object graph node in an external knowledge base (e.g., an Internet-based, social network, or networked enterprise computer service). As with cross-reference edges, outgoing edges connected to subject graph nodes may indicate a connection to an external object graph node in any suitable manner, such as by storing a pair of identifiers indicating the external graph and the object graph node within the external graph. In some cases, such as when the external graph is a global knowledge base derived from knowledge published on the Internet, the external graph may not store any user-centric data.
[0063] By including cross-references between constituent graph structures, facts about a particular noun (e.g., an event or entity) can be distributed across multiple constituent graph structures while still supporting centralized reasoning about relationships between user-centric facts in different constituent graph structures and external databases (e.g., by traversing multiple constituent graph structures via cross-references and cross-reference edges).
[0064] Each user-centric fact can be stored in a predictable shared data format that stores user-centric facts, including application-specific facts associated with a computer service, without changing the format of the application-specific data of the computer service. This predictable shared data format is referred to herein as an "application-independent data format". The application-independent data format can store the information required to query a user-centric graph while avoiding redundant storage of application-specific data. The graph data structure can be implemented using a complementary application programming interface (API) that allows read and write access to the graph data structure. The API can restrict access to the graph data structure to ensure that all data written to the graph data structure has an application-independent data format. At the same time, the API can provide a mechanism that any computer service can use to add new user-centric facts to the graph data structure in an application-independent data format, thereby ensuring that other computer services using the API can predictably use all data stored in the graph data structure. In addition to providing read / write access to user-centric facts stored in the graph data structure, the API can also provide data processing operations including both read and write, such as query operations and caches of query results.
[0065] Without departing from the spirit of the present disclosure, a graph data structure including user-centric facts about a plurality of different computer services, such as graph data structure 300, may be implemented in various ways. For example, the graph data structure may be stored as a collection of node records, each node record optionally indicating one or more outgoing edges to other nodes. Alternatively or additionally, the graph data structure may be stored as a pointer-based data structure, a list data structure, a matrix data structure, or any other suitable representation of graph data.
[0066] In one example, the user-centric facts stored in the graph data structure are application-specific facts. In addition to the application-specific facts, the user-centric facts may include one or more extensions, where an extension is additional data that includes application-independent facts associated with the application-specific facts. For example, an extension may include browser-specific extensions based on details of a user's interaction with a browser program (e.g., browser 110). The browser-specific extensions may include information describing a single interaction between a user and a browser, an entire session of user / browser interactions, and / or any suitable collection of historical user / browser interactions. Figure 1AIn the example shown, the browser-specific expansion can include a computer-readable description of the user's access to the travel reservation page. Alternatively or additionally, the browser-specific expansion can include an analysis of the further interaction between the user and the browser. For example, the expansion can include a description of the user's frequent visits to the travel reservation page in the last week, together with a description of the frequency and specific time of the user's visits to the travel reservation page. Triggering facts, user-centric facts that define contexts, and other user-centric facts in the user-centric figure can include any suitable browser-specific or other application-specific expansions. Therefore, the auxiliary information presented by the browser can be based on such expansions.
[0067] In addition to the application-independent representation of connections in the graph data structure, the application-independent data format for user-centric facts also allows for efficient storage of application-specific facts. For example, a subject graph node can represent the subject of an application-specific fact, such as the fact that a new meeting is scheduled. Accordingly, the graph data structure can store facet pointers for application-specific facts, which indicate auxiliary application-specific data associated with the application-specific fact. In one example, the auxiliary application-specific data is indicated in the form of facet pointers. A facet pointer is an identifier (e.g., a digital identifier) indicating the storage location of auxiliary application-specific data associated with a user-centric graph, such as the storage location of a calendar item representing meeting details. A facet pointer can be associated with a specific type of data based on its inclusion in an application-specific graph structure, for example, a constituent graph structure A can be associated with a calendar planning software and can accordingly include calendar data, rather than other types of application-specific data. In other examples, a facet pointer can include additional identification information specifying the type of auxiliary application-specific data, so that the facet pointer can be used to represent different types of auxiliary application-specific data (e.g., a variety of file types that can be used by a word processing application). The graph data structure may be used to look up auxiliary application-specific data via facet pointers while avoiding redundant storage of the auxiliary application-specific data within the graph data structure.
[0068] In addition to the facet pointers stored in the subject graph node, application-specific facts are also represented by edges connecting the subject graph node to one or more object graph nodes. Although a single subject graph node can be included in more than one user-centric fact, the graph data structure efficiently stores only a single node record for the subject graph node. The node record includes a list of all outgoing edges of the node, which can reduce storage space requirements compared to storing a copy of the subject graph node for each user-centric fact that appears therein. For some nodes, such as nouns that are only object graph nodes, the list of outgoing edges can be empty. The list of outgoing edges of the subject graph node includes one or more edge records that define one or more edges. A graph node can have any number of outgoing edges to appropriately represent the relationship with other nodes of the graph data structure, for example, zero, one, or three or more edges.
[0069] In addition to representing application-specific facts through facet pointers and lists of outgoing edges, the main graph node can also represent one or more extensions of the application-specific facts. In one example, one or more extensions include a node confidence value, which can indicate the relevance of the graph node to the user. For example, the confidence value can be determined by a machine learning model that is trained to identify the relevance to the user by learning to distinguish labeled samples of relevant data from labeled samples of irrelevant data. For example, training the machine learning model can include supervised training using user-labeled samples (e.g., derived from direct user feedback on the application during application), and / or unsupervised training. One or more extensions can also include edge confidence values associated with each edge. Like node confidence values, edge confidence values can indicate the relevance of a particular edge to the user. Different edges between a pair of nodes can have different confidence values. For example, if a scheduled meeting is considered more relevant to the user than a commitment, an edge indicating a "scheduled meeting" relationship can have a lower edge confidence than a different edge indicating "making a commitment."
[0070] In addition to node confidence values and edge confidence values, one or more extensions of application-specific facts may include other application-independent data and / or application-specific data. For example, a node may include access metadata indicating information associated with accessing the node in a user-centric graph, as well as associated application-specific data (e.g., data indicated by a facet pointer). The access metadata may include a timestamp indicating the time and date of the most recent access, a delta value indicating a change caused by the most recent access, or any other suitable metadata.
[0071] For user-centric facts, the graph data structure may additionally store one or more tags that define auxiliary data associated with the user-centric fact, including any suitable auxiliary data associated with the node / edge. For example, when a graph node represents a person (e.g., associated with a contact book entry), the tags may include a nickname for the person and an alternate email address for the person.
[0072] User-centric facts and nodes / edges of user-centric facts can be expanded by additional semantics stored in tags. For example, tags can be used to store one or more expansions of user-centric facts. Tags stored in node records can be associated with user-centric facts, where the node record represents a subject graph node, or where the node record represents an object graph node. Alternatively or additionally, tags stored in node records can be associated with the node record itself (e.g., associated with a subject graph node represented by the node record and / or associated with an object graph node represented by the node record), or associated with one or more edges connected to the node record. In other examples, tags can be used to store metadata of user-centric facts (e.g., instead of or in addition to access metadata of user-centric facts). For example, when a user-centric fact is associated with a timestamp, the timestamp can be optionally stored in the tag of the user-centric fact.
[0073] In some examples, one or more tags are searchable tags, and the graph data structure is configured to enable searching for user-centric facts by searching among the searchable tags (e.g., searching for tags storing a search string, or searching for tags storing a specific type of data).
[0074] The graph data structure may be represented as a plurality of application-specific constituent graph structures, where nodes of the constituent graph structures are cross-referenced across the constituent graph structures (eg, via cross-reference edges and node cross-references, as described above).
[0075] The user-centric graph is application-independent because it can be used to track facts from two or more potentially unrelated computer services that may have different native data formats. The user-centric graph can store application-specific facts for any computer service by storing facet pointers, so that user-centric facts can include application-specific facts even if the application-specific facts can be stored in an application-specific format. In addition, by storing cross-references in the form of domain identifiers (e.g., object domain identifiers in the outgoing edge list of each subject graph node, or reference domain identifiers in a node cross-reference representation) and node identifiers (e.g., object graph node identifiers and reference node identifiers), the user-centric graph enables the representation of user-centric facts defined in the context of two or more computer services. In addition, the user-centric graph is user-centric because a different user-centric graph can be defined for each user. The user-centric graph can be suitable for storing user-centric facts in context when each of multiple different users interacts with a shared computer service (e.g., a web browser). Because the user-centric graph stores application-specific facts via facet pointers and represents relationships to facts in other data structures via cross-references, it may be possible to store user-centric facts about a user in a user-specific user-centric graph data structure without requiring write access to application-specific data of a shared computer service.
[0076] When new user-centric facts are added based on application-specific facts, the graph data structure may optionally store application-independent extensions, such as application-independent facts associated with the application-specific facts. The user-centric facts may be defined in part by extensions received from an application-specific data provider. Alternatively or additionally, the user-centric facts provided by the application-specific data provider may be pre-processed to include one or more extensions via an extension pipeline including one or more extension adapters. When the graph data structure stores labels associated with the user-centric facts (e.g., labels stored in node records), the one or more extensions may be included in the labels.
[0077] In an example, the augmentation adapter includes a machine learning model configured to receive application-specific facts, identify the relevance of application-specific facts to a user, and output a confidence value indicating the relevance with a numerical value. The machine learning model can be any suitable model, such as a statistical model or a neural network. The machine learning model can be trained, for example, based on user feedback. For example, when the machine learning model is a neural network, the output of the neural network can be evaluated via an objective function, which indicates the error level of the predicted relevance output by the neural network compared to the actual relevance indicated in the user feedback. The gradient of the objective function can be calculated based on the derivative of each function in each layer of the neural network using back propagation. Therefore, the weights of the neural network can be adjusted based on the gradient (e.g., via gradient descent) to minimize the error level indicated by the objective function. In some examples, the machine learning model can be trained for a specific user based on direct feedback provided by the user when interacting with the computer service, for example, indicating the relevance of search results in a search application. Therefore, the trained machine learning model may be able to estimate the relevance to the user. In some examples, the machine learning model can be trained based on indirect feedback from the user, for example, by estimating the similarity of the relevant content to other content that the user has indicated as relevant in the past.
[0078] In another example, the augmentation adapter includes a natural language program for identifying natural language features of application-specific facts. For example, the natural language program can determine a subject graph node and / or an object graph node for an application-specific fact by identifying the natural language feature as being associated with an existing subject graph node and / or an object graph node. In some examples, the natural language program can determine the relationship type of an edge of the application-specific fact. In some examples, the natural language program can determine one or more labels of a subject graph node and / or an object graph node for an application-specific fact. The natural language program can be configured to recognize the following features, including: 1) named entities (e.g., people, organizations, and / or objects), 2) intent (e.g., emotions or goals associated with the natural language feature), 3) events and tasks (e.g., tasks that a user intends to perform later), 4) topics (e.g., topics contained or represented by user-centric facts), 5) locations (e.g., geographic locations to which user-centric facts refer, or locations where user-centric facts are generated), and / or 6) dates and times (e.g., timestamps indicating past events or future scheduled events associated with user-centric facts).
[0079] The augmentations associated with the user-centric facts may provide augmented semantics of the user-centric facts (e.g., additional meaningful information beyond the information provided by the connection structure formed by the edges between the subject graph nodes and the object graph nodes of the user-centric facts). The graph data structure may identify and include additional user-centric facts that may be derived from the augmented semantics (e.g., based on one or more augmentations added in the augmentation pipeline). Thus, adding user-centric facts including one or more augmentations may also include: identifying additional user-centric facts based on the one or more augmentations, and adding the additional user-centric facts to the graph data structure having an application-independent data format.
[0080] Identifying additional user-centric facts based on one or more extensions may include: identifying that an extension of the one or more extensions corresponds to another user-centric fact already included in the graph data structure (e.g., because the extension is associated with a subject noun or an object noun of another user-centric fact). Alternatively or additionally, identifying additional user-centric facts based on one or more extensions may include: identifying a first extension of the one or more extensions that is associated with a subject noun that is not yet related to any user-centric fact; identifying that a second extension of the one or more extensions is associated with an object noun; identifying a relationship between the subject noun and the object noun; and adding a new user-centric fact involving the object noun and the subject noun to the graph data structure. In some examples, identifying additional user-centric facts based on one or more extensions includes: identifying any suitable relationship between the one or more extensions, and adding a user-centric fact representing the identified relationship.
[0081] In an example, a first node and a second node both include an extension that specifies an identified named entity, wherein the two extensions specify the same named entity. Thus, the graph data structure may store an edge connecting the first node to the second node, and the relationship type of the edge may indicate that the two nodes are inferred to be associated with the same entity. Alternatively or additionally, the graph data structure may store a node cross-reference in each node indicating that another node is associated with the same named entity. Optionally, the graph data structure may modify the first node to include data for the second node and delete the second node, thereby avoiding redundant storage of data for the second node by collapsing the representation to include a single node instead of two nodes.
[0082] In another example, a first node includes an expansion of a specified named entity, and an edge can be added to connect the first node to a second node representing the same named entity. In another example, an edge can be added between a first node and a second node having the same associated topic. In another example, an edge can be added between a first node and a second node having the same associated time and / or location. For example, an edge can be added between a first node that mentions a specific calendar date (e.g., in a tag) and a second node that was created on the specific calendar date (e.g., as indicated by access metadata). In another example, an edge can be added between two nodes that were created at the same location or that mention the same location.
[0083] There is no limit to the number of different computer services that may contribute to the knowledge base. In addition, there is no requirement that different computer services be related to each other or to the user-centric graph in any particular way (e.g., different computer services and user-centric graphs may be unrelated to each other and provided by different computer service providers). Therefore, a user-centric knowledge AI repository can include user-centric facts derived from multiple different computer services, thereby including user-centric facts from more different contexts. In addition, cross-references between application-specific constituent graph structures enable user-centric facts to express relationships between aspects of different computer services, which can further improve practicality compared to maintaining multiple different, independent knowledge bases without cross-references.
[0084] The graph data structure represents structured relationships between user-centric facts (e.g., two facts with the same subject noun may be represented by a single node record within the constituent graph and cross-referenced between the constituent graphs). Therefore, traversing the graph data structure to find user-centric facts that satisfy a query is more efficient than exhaustively searching the set of user-centric facts. For example, if a user has frequently interacted with a particular other person, the frequent interactions may indicate that the other person may be related to the user. Therefore, there may be more user-centric facts with the person as the subject or object, and when traversing the edges of the graph data structure, it is more likely to encounter a node representing the other person because there are many edges leading to and from the node representing the other person.
[0085] Traversing the graph data structure may include a "random walk" along the edges of the graph data structure. The random walk may begin from a current user context, which is used herein to refer to any suitable starting point for answering a query. In an example, the current user context may be defined by a query (e.g., by including a contextual keyword indicating a subject graph node to be used as a starting point). In other examples, the current user context may be an application-specific context (e.g., "answering an email") suitable for determining a subject graph node to be used as a starting point.
[0086] When a node is encountered during a random walk (e.g., at the starting point), the node can be checked to determine whether it satisfies the constraints of the query. If it does, it can be output in a subset of user-centric facts in response to the query. Then, after encountering the node, the random walk can continue, thereby encountering more nodes. In order to find more nodes, the random walk can continue along the outgoing edges of the encountered nodes. Determining whether to continue along the outgoing edge can be a weighted random determination, including evaluating a weight representing the possibility of following the edge, and sampling whether to follow the edge based on the weight and random data, for example, a "roulette wheel selection" algorithm implemented using a random number generator. The weight of the edge connecting the subject graph node to the object graph node can be determined based on the confidence value of the subject graph node, edge and / or object graph node. In the example, the confidence value can be interpreted as an indication of relevance to the user, so that it is more likely to follow the edge that is more relevant or connects more relevant nodes. The weight of the edge may be further determined based on other data of the subject graph nodes, object graph nodes, and edges, for example, by evaluating the relevance of the edge to the query based on a natural language comparison of the edge type with one or more natural language features of the query.
[0087] By specifying constraints (eg, characteristics of subject graph nodes, object graph nodes, and edges that define user-centric facts), a user may be able to formulate a variety of queries to be answered using the user-centric graph.
[0088] In addition to selecting a subset of user-centric facts that satisfy the constraints specified in the query, the user-centric graph is also able to respond to other specialized queries, such as slicing queries and ranking queries.
[0089] In the example, the query is a slice query indicating a starting node and a distance parameter. The reply to the slice query is a subset of user-centric facts, including user-centric facts reached by starting at the starting node and traversing the edges of the graph data structure to form a path of length at most equal to the distance parameter from the starting node. For example, if the distance parameter is set to 1, the reply to the query will include the starting node and all once-deleted nodes directly connected to the starting node; if the distance parameter is set to 2, the reply to the query will include the starting node, all once-deleted nodes, and all twice-deleted nodes directly connected to at least one of these once-deleted nodes. The slice query can represent a set of user-centric facts that are potentially related to user-centric facts of particular interest (e.g., user-centric facts involving the starting node) in a manner that is connected to the starting node by a path of at most the distance parameter. By setting a smaller distance parameter, the reply to the query can represent a relatively small set of facts that are closely related to the starting node; similarly, by setting a larger distance parameter, the reply can represent a larger set of facts that are indirectly related to the starting node. As an alternative to specifying a starting node, a query can also use the current user context as a starting node, thereby representing a collection of user-centric facts related to the user's current context.
[0090] In another example, the query is a ranking query that ranks multiple user-centric facts based at least on a confidence value associated with each user-centric fact, and a subset of the user-centric facts is ranked according to the confidence value of each user-centric fact. The ranking query can be interpreted as collecting user-centric facts that may be relevant to the user without imposing additional specific constraints on the query. In addition to ranking multiple user-centric facts based on confidence values, multiple user-centric facts can also be ranked based on other features. For example, if a user-centric fact is newer, it can be weighted as more relevant (e.g., based on a timestamp associated with each fact). In another example, the ranking query can include a keyword, and if a user-centric fact includes at least one node with the keyword in its label, it can be weighted as more relevant.
[0091] Although two examples of specialized queries ("slice" and "rank") are described above, the user-centric graph enables other kinds of specialized queries based on "slice" queries, "rank" queries, and other traversals of the user-centric graph according to any suitable constraints. For example, a subset of user-centric facts in response to a slice query can be additionally sorted according to confidence values as in a ranking query, thereby combining the functionality of these two queries. In another example, the query is a pivot query that indicates a starting node. The reply to the pivot query is a subset of user-centric facts, including user-centric facts reached by starting at the starting node and traversing the edges of the graph data structure to form a path of unlimited (or arbitrarily, large) length. The pivot query can be interpreted as a slice query that does not restrict the length of the path reached by the starting node, for example, the distance parameter is infinite. In some examples, a response to a query may include a visualization of a subset of the responses to the user-centric facts as a graph diagram, which may be annotated or animated to include any appropriate information about the user-centric facts (e.g., auxiliary application-specific data indicated by facet pointers to a node included in one of the user-centric facts).
[0092] A query may define a time constraint such that a subset of user-centric facts output in response to the query is limited to user-centric facts associated with timestamps indicating times within a range defined by the query. In some examples, time is an inherent property of nodes and edges in a graph data structure (e.g., each of a plurality of user-centric facts included in the graph data structure is associated with one or more timestamps). One or more timestamps associated with a node or edge may indicate a time when the user-centric fact was created, accessed, and / or modified (e.g., access metadata of a node defining the user-centric fact). Alternatively or additionally, one or more timestamps may indicate a time mentioned in the user-centric fact (e.g., a time when a meeting was scheduled, or any other timestamp added by an augmentation adapter in an augmentation pipeline). One or more timestamps may optionally be stored as searchable tags.
[0093] In some examples, one or more constraints defined by the query include a reply type constraint, and accordingly, the subset of user-centric facts selected in response to the query may include only user-centric facts that satisfy the reply type constraint. For example, the reply type constraint may constrain the characteristics of the subject graph node, the object graph node, and / or the edge of the user-centric fact. For example, the reply type constraint may indicate a specific type of subject and / or object graph node, such as: 1) the subject or object is a person; 2) both the subject and the object are colleagues; 3) the subject is a place; 4) the object is a subject; or 5) the subject is a person and the object is a scheduled event. Alternatively or additionally, the reply type constraint may indicate one or more specific subjects and / or objects, such as 1) the subject is a user; 2) the subject is the user's boss Alice; or 3) the object is any one of Alice, Bob, or Charlie. Alternatively or additionally, the reply type constraint may indicate one or more specific types of edges, such as by indicating a relationship type such as a "send email" relationship, a "have lunch together" relationship, or a "research topic" relationship.
[0094] In some examples, one or more constraints defined by the query may include a graph context constraint, and accordingly, a subset of user-centric facts selected in response to the query may include only user-centric facts related to contextualized user-centric facts in a user-centric graph that satisfies the graph context constraint. Based on any appropriate feature of a graph data structure that can indicate a possible relationship, two different user-centric facts may be described herein as being related. For example, when a graph context constraint indicates Alice, the boss of the user, the contextualized user-centric fact may be any fact having a node representing Alice as a subject graph node or an object graph node. Therefore, a subset of user-centric facts selected in response to the query may include other user-centric facts having subject graph nodes and / or object graph nodes that are directly connected to a node representing Alice via an edge. Alternatively or additionally, a subset of user-centric facts may include user-centric facts indirectly related to Alice, for example, user-centric facts having a subject graph node and / or an object graph node that is indirectly connected to a node representing Alice via a path of two or more edges. In some cases, the query including the graph context constraint can be a slicing query, and the subset of user-centric facts can include only user-centric facts that are related to the contextualized user-centric fact and reachable within a maximum specific distance of a node of the contextualized user-centric fact. In other examples, the query including the graph context constraint can be a ranking query, and the subset of user-centric facts can include a selection of user-centric facts that are most likely related to the contextualized user-centric fact, for example, user-centric facts that are connected to the contextualized user-centric fact via a large number of different paths or via paths that include edges with high confidence values.
[0095] Answering a query may include traversing a graph data structure based on one or more timestamps associated with each user-centric fact, which may be referred to herein as traversing the time dimension of the graph data structure. For example, answering a query may include starting at a node associated with a time defined by the query and traversing the graph by following any edges with timestamps indicating subsequent times, such that the timestamps increase in the same order as the traversal. In other examples, answering a query may include traversing the graph data structure by following any edges with timestamps prior to a date defined by the query. In addition, the inherent time properties of each node and edge in the graph data structure may enable a timeline view of the graph data structure. In an example, a reply to a query may include a timeline view of the graph data structure, e.g., user-centric facts arranged in chronological order according to the timestamps associated with each user-centric fact.
[0096] As another example of ad hoc queries, the graph data structure may be configured to allow searching for user-centric facts based on searchable tags for the user-centric facts stored by the graph data structure. Searching for user-centric facts based on the searchable tags may include traversing the graph in any suitable manner (e.g., as described above for slice queries or pivot queries), and while traversing the graph, outputting any user-centric facts encountered during the traversal for which the graph structure has stored searchable tags.
[0097] As another example of a specialized query, the graph data structure can be configured to serve a user context query by searching for user-centric facts related to the user's current context. Thus, the user context query can include one or more constraints related to the user's current context. For example, the one or more constraints can include a time constraint based on the current time of serving the user context query. Alternatively or in addition, the one or more constraints can include a graph context constraint related to the user's current context, for example, a graph context constraint that specifies tasks in which the user can participate.
[0098] In some examples, one or more constraints of the user context query may be based on state data of a computer service that published the user context query. In some examples, the state data of the computer service includes natural language features (e.g., intent, entity, or subject), and the one or more constraints include indications of the natural language features. For example, when the computer service is an email program, the one or more constraints of the user context query may include: 1) a time limit based on the time when the user begins composing the email; 2) a graph context constraint indicating the subject of the email body; 3) a graph context constraint indicating the recipients of the email. Thus, the subset of user-centric facts selected in response to the user context query may include user-centric facts that are current (based on timestamps) and may be relevant to the user's task of composing the email (based on subject and recipients).
[0099] Exemplary Computing System
[0100] The methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, these methods and processes may be implemented as an executable computer application, a network accessible computing service, an application programming interface (API), a library, or a combination of the above and / or other computing resources.
[0101] Figure 4Schematically illustrated is a simplified representation of a computing system 400 that is configured to provide any to all of the computing functionality described herein. The computing system 400 may take the form of one or more personal computers, network-accessible server computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), virtual / augmented / mixed reality computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, and / or other computing devices. For example, the computing system 400 may include any combination of logic subsystems, storage subsystems, and / or user-centric graph servers 103, client computers 102, user-centric data sources 105A-105N, networks 104, and / or other computing devices not in the network. Figure 1A Other subsystems of one or more of the other computers shown.
[0102] The computing system 400 includes a logic subsystem 402 and a storage subsystem 404. The computing system 400 may optionally include an input / output subsystem 406 (e.g., including one or more input devices or sensors, and one or more output devices, such as a graphic display and / or audio speakers), a communication subsystem 408, and / or other components not described herein. Figure 4 Other subsystems shown in .
[0103] The logic subsystem 402 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions that are part of one or more applications, services, or other logical structures. The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processor of the logic subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic subsystem may optionally be distributed in two or more separate devices, which may be located at a remote location and / or configured for collaborative processing. Some aspects of the logic subsystem may be virtualized and executed by a remotely-accessible networked computing device configured in a cloud computing configuration.
[0104] The storage subsystem 404 includes one or more physical devices configured to temporarily and / or permanently hold computer information, such as data and instructions executable by the logic subsystem. When the storage subsystem includes two or more devices, the devices can be co-located and / or remotely located. The storage subsystem 404 can include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices. The storage subsystem 404 can include removable and / or built-in devices. As the logic subsystem executes instructions, the state of the storage subsystem 404 can change—for example, to hold different data.
[0105] Aspects of logic subsystem 402 and storage subsystem 404 may be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, program-specific and application-specific integrated circuits (PASIC / ASIC), program-specific and application-specific standard products (PSSP / ASSP), systems on chips (SOC), and complex programmable logic devices (CPLD).
[0106] The logic subsystem and the storage subsystem can collaborate to instantiate one or more logic machines. As used herein, the term "machine" is used to collectively refer to hardware and any software, instructions, and / or other components that collaborate with the hardware to provide computer functions. In other words, a "machine" is never an abstract idea, but always has an entity form. A machine can be instantiated by a single computing device, or a machine can include two or more subcomponents instantiated by two or more different computing devices. In some embodiments, the machine includes a local component (e.g., a computer service) that collaborates with a remote component (e.g., a cloud computing service). The software and / or other instructions that give a particular machine its functions can be optionally stored on a suitable storage device as an unexecuted module. Non-limiting examples of machines that can be instantiated by a computing system 400 according to the present disclosure include a browser 110 and a user-centric graph data structure 300, as well as any computer service that can be configured to interact with a browser 110 and a user-centric graph data structure 300, such as an email client, a scheduler, and the like.
[0107] Machines according to the present disclosure may be implemented using any suitable combination of state-of-the-art and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) techniques. Non-limiting examples of techniques that may be incorporated into implementations of one or more machines include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and merge features in one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing), and / or other suitable convolutional neural networks. machine and / or neural random access memory), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graph models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, region delimiting and / or dependency parsing, and / or intent identification, segmentation models, and / or hyper-segmentation models (e.g., hidden dynamic models)).
[0108] In some examples, one or more differentiable functions may be used to implement the methods and processes described herein, wherein the gradient of the differentiable function may be calculated and / or estimated with respect to the input and / or output of the differentiable function (e.g., with respect to the training data and / or with respect to the target function). Such methods and processes may be determined, at least in part, by a set of trainable parameters. Thus, the trainable parameters for a particular method or process may be adjusted by any suitable training process in order to continually improve the functionality of the method or process.
[0109] Non-limiting examples of training processes for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on categories derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning), and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual bandit methods, maximum likelihood methods, and / or expectation maximization. In some examples, multiple methods, processes, and / or components of the system described herein can be trained simultaneously for an objective function that measures the performance of the collective function of multiple components (e.g., for enhanced feedback and / or for labeled training data). Training multiple methods, processes, and / or components simultaneously can improve such collective function. In some examples, one or more methods, processes, and / or components can be trained independently of other components (e.g., offline training for historical data).
[0110] The methods and processes disclosed herein can be configured to give users and / or any other person control over any private and / or potentially sensitive data. Whenever data is stored, accessed and / or processed, it can be processed according to privacy and / or security standards. When collecting user data, users or other stakeholders can specify how the data is used and / or stored. Whenever user data is collected for any purpose, the user who owns the data should be notified, and the user data should only be collected when the user provides affirmative consent. If data is to be collected, it can and should be collected under the premise of respecting the privacy of the user to the greatest extent. If the data is to be released for access by anyone other than the user or for any decision-making process, the user's consent can be collected before using and / or releasing the data. Users can choose to join and / or opt out of data collection at any time. After collecting data, users can issue commands to delete data and / or restrict access to data. All potentially sensitive data can optionally be encrypted and / or anonymized where feasible to further protect user privacy. Users can specify partial data, metadata, or statistical information / results of processing data to be published to other parties, such as for further processing. Private and / or confidential data may be kept completely private, e.g., only temporarily decrypted for processing, or only decrypted for processing on a user device and otherwise stored in encrypted form. The user may hold and control the encryption keys used to encrypt the data. Alternatively or additionally, the user may designate a trusted third party to hold and control the encryption keys used to encrypt the data, e.g., to provide the user with access to the data in accordance with an appropriate authentication protocol.
[0111] When the methods and processes described herein incorporate ML and / or AI components, the ML and / or AI components may make decisions based at least in part on training the components for training data. Thus, the ML and / or AI components may and should be trained on a variety of representative data sets that include sufficient relevant data for a variety of users and / or user groups. Specifically, the training data sets should be inclusive for different human individuals and groups so that when training the ML and / or AI components, performance is improved for the user experience of the users and / or user groups.
[0112] For example, a dialog system according to the present disclosure may be trained to interact with different groups of users using language models that are trained based on characteristics of the groups' languages, dialects, accents, and / or any other speaking styles in order to be appropriate for those groups.
[0113] ML and / or AI components may additionally be trained to make decisions to minimize potential biases against human individuals and / or groups. For example, when AI systems are used to assess any qualitative and / or quantitative information about human individuals or groups, these AI systems may be trained to be invariant to differences between individuals or groups that are not intended to be measured by the qualitative and / or quantitative assessments, for example, so that any decisions are not influenced in an unintended manner by differences between individuals and groups.
[0114] ML and / or AI components can and should be designed to provide context about how they operate as much as possible so that implementers of ML and / or AI systems can be held accountable for decisions / evaluations made by the system. For example, ML and / or AI systems should have replicable behavior, e.g., when they make pseudo-random decisions, random seeds should be used and logged so that the decisions can be replicated later. As another example, data used to train and / or test ML and / or AI systems should be collated and maintained to facilitate future investigations of the behavior of ML and / or AI systems with respect to the data. In addition, ML and / or AI systems can and should be continuously monitored to identify potential biases, errors, and / or unexpected results.
[0115] When included, input / output subsystem 406 can be used to present a visual representation of the data held by storage subsystem 404. The visual representation can take the form of a graphical user interface (GUI). Input / output subsystem 406 can include one or more display devices using almost any type of technology. In some implementations, the display subsystem can include one or more virtual reality displays, augmented reality displays, or mixed reality displays.
[0116] When included, the input / output subsystem may also include or interface with one or more input devices. An input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) component portions. Such component portions may be integrated or peripheral, and the conversion and / or processing of input actions may be on-board or off-board processing. Example NUI component portions may include: a microphone for voice and / or sound recognition; an infrared, color, stereo, and / or depth camera for machine vision and / or gesture recognition; a head tracker, an eye tracker, an accelerometer, and / or a gyroscope for motion detection and / or intent recognition.
[0117] When included, the communication subsystem 408 can be configured to communicatively couple the computing system 400 to one or more other computing devices. The communication subsystem 408 can include wired and / or wireless communication devices compatible with one or more different communication protocols. The communication subsystem can be configured for communication via a personal area network, a local area network, and / or a wide area network.
[0118] The language model can use vocabulary features to guide sampling / searching words to recognize speech. For example, the language model can be defined at least in part by the statistical distribution of words or other vocabulary features. For example, the language model can be defined by the statistical distribution of n-grams, which define the transition probabilities between candidate words based on vocabulary statistics. The language model can also be based on any other appropriate statistical features and / or the results of processing statistical features using one or more machine learning and / or statistical algorithms (e.g., confidence values generated by such processing). In some examples, the statistical model can limit the words that can be recognized for the audio signal, for example, based on the assumption that the words in the audio signal are from a specific vocabulary.
[0119] Alternatively or additionally, the language model can be based on one or more neural networks that have been previously trained to represent audio input and words in a shared latent space, such as a vector space learned by one or more audio and / or word models (e.g., wav21etter and / or word2vec). Thus, finding candidate words can include searching the shared latent space for the audio input based on vectors encoded by the audio model to find candidate word vectors to decode with the word model. The shared latent space can be used to evaluate, for one or more candidate words, the confidence that the candidate word plays an important role in the speech audio.
[0120] The language model can be used in conjunction with an acoustic model that is configured to evaluate the confidence of the candidate word in the speech audio included in the audio signal based on the acoustic features of the word, for the candidate word and the audio signal (e.g., Mel-frequency cepstral coefficients, resonance peaks, etc.). Optionally, in some examples, the language model can be incorporated into the acoustic model (e.g., the evaluation and / or training of the language model can be based on the acoustic model). The acoustic model, for example, defines a mapping between an acoustic signal and basic sound units such as phonemes based on labeled speech audio. The acoustic model can be based on any appropriate combination of state-of-the-art technology or future machine learning (ML) and / or artificial intelligence (AI) models, such as: deep neural networks (e.g., long short-term memory, temporal convolutional neural networks, restricted Boltzmann machines, deep belief networks), hidden Markov models (HMMs), conditional random fields (CRTs) and / or Markov random fields, Gaussian mixture models, and / or other graph models (e.g., deep Bayesian networks). The audio signal to be processed by the acoustic model can be preprocessed in any suitable manner, such as encoding at any suitable sampling rate, Fourier transform, bandpass filter, etc. The acoustic model can be trained to identify the mapping between the acoustic signal and the sound unit based on training with labeled audio data. For example, the acoustic model can be trained based on labeled audio data including speech audio and corrected text to learn the mapping between the speech audio signal and the sound unit represented by the corrected text. Thus, the acoustic model can be continuously improved to improve its utility in correctly identifying speech audio.
[0121] In some examples, in addition to statistical models, neural networks, and / or acoustic models, the language model can also incorporate any suitable graphical model, such as a hidden Markov model (HMM) or a conditional random field (CRF). Given the speech audio and / or other words recognized so far, the graphical model can use statistical features (e.g., transition probabilities) and / or confidence values to determine the probability of recognizing a word. Therefore, the graphical model can use statistical features, previously trained machine learning models, and / or acoustic models to define transition probabilities between states represented in the graphical model.
[0122] In an example, a method for providing contextual information to a user during a browsing session includes: maintaining a user-centric graph, the graph including a plurality of user-centric facts associated with the user and derived from the user's interactions with a plurality of different computer services; identifying a context in which the user interacts with a browser application, the context being defined by at least one or more contextualized user-centric facts in the user-centric graph; evaluating the context to identify auxiliary information related to the context; and displaying the auxiliary information to the user. In this example or any other example, the auxiliary information is based at least on one or more user-centric facts derived from the user's interactions with a plurality of different computer services. In this example or any other example, the method further includes: identifying a trigger fact to trigger identification of the context; and adding the trigger fact to the user-centric graph, wherein identifying the auxiliary information is based at least on a relationship between the trigger fact and one or more other user-centric facts in the user-centric graph in the user-centric graph. In this example or any other example, the trigger fact includes one or more previous interactions of the user with the browser. In this example or any other example, the triggering fact includes a previous interaction of the user with a computer service other than the browser. In this example or any other example, the relationship between the triggering fact and the one or more other user-centric facts in the user-centric graph is a temporal relationship. In this example or any other example, identifying the triggering fact includes: evaluating a signal strength score for a candidate fact, the signal strength score indicating the strength of the relationship between the candidate fact and the user-centric facts in the user-centric graph, and identifying the triggering fact in response to the signal strength score exceeding a threshold.
[0123] In an example, a method for providing contextual information to a user during a browsing session includes: maintaining a user-centric graph, the graph including a plurality of user-centric facts associated with the user and derived from the user's interactions with a plurality of different computer services; identifying a context of the user's interactions with a browser application; identifying auxiliary information related to the context, the auxiliary information being based at least on one or more user-centric facts in the user-centric graph; and displaying the auxiliary information to the user. In this example or any other example, the context is defined by at least one or more contextualized, user-centric facts derived from the user's interactions with a plurality of different computer services. In this example or any other example, the method further includes: identifying a triggering fact to trigger identification of the context, wherein identifying the auxiliary information is based at least on a relationship between the triggering fact and one or more other user-centric facts in the user-centric graph in the user-centric graph. In this example or any other example, the triggering fact includes one or more previous interactions of the user with the browser. In this example or any other example, the triggering fact includes previous interactions of the user with computer services other than the browser. In this example or any other example, the relationship between the triggering fact and the one or more other user-centric facts in the user-centric graph is a temporal relationship. In this example or any other example, the triggering fact includes: evaluating a signal strength score for a candidate fact, the signal strength score indicating the strength of the relationship between the candidate fact and the user-centric facts in the user-centric graph, and identifying the triggering fact in response to the signal strength score exceeding a threshold.
[0124] In an example, a system for providing contextual information to a user during a browsing session includes: a logic device; and a storage device configured to hold: a user-centric graph, the graph including a plurality of user-centric facts associated with the user and derived from the user's interactions with a plurality of different computer services; and instructions executable by the logic device to: identify a triggering fact to identify a context of the user's interaction with a browser application; and identify auxiliary information related to the context, the auxiliary information being based on the context and at least based on one or more user-centric facts in the user-centric graph. In this example or any other example, the instructions may also be executable to receive from the browser a computer-readable description of one or more previous interactions of the user with the browser, and wherein the triggering fact includes the computer-readable description of the one or more previous interactions. In this example or any other example, the triggering fact includes a description of the user's previous interactions with a computer service other than the browser. In this example or any other example, the relationship between the triggering fact in the user-centric graph and the one or more other user-centric facts in the user-centric graph is a temporal relationship. In this example or any other example, identifying the triggering fact includes: evaluating a signal strength score for a candidate fact, the signal strength score indicating the strength of a relationship between the candidate fact and a user-centric fact in the user-centric graph, and identifying the triggering fact in response to the signal strength score exceeding a threshold. In this example or any other example, the auxiliary information includes information based on one or more of the following: 1) emails sent to or from the user, 2) contacts of the user, 3) calendar data of the user, 4) documents, 5) web pages, 6) location data associated with the user, and / or 7) application usage data associated with the user.
[0125] It will be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered to have a restrictive meaning, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the individual actions illustrated and / or described may be performed or omitted in the order illustrated and / or described, in other orders, in parallel. Likewise, the order of the above process may be changed.
[0126] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein and any and all equivalents thereof.
Claims
1. A method for providing contextual information to a user during a browsing session, comprising: maintaining a user information graph, the graph comprising a plurality of facts associated with the user and derived from the user's interactions with a plurality of different computer services, wherein the user information graph comprises application-specific component graph structures associated with the plurality of different computer services, each application-specific component graph structure comprising edges connecting it to one or more other application-specific component graph structures in the application-specific component graph structures, each application-specific component graph structure comprising a plurality of nodes; identifying a context of the user's interaction with the browser application based on a relationship between the application-specific graph structures; identifying auxiliary information related to the context, the auxiliary information comprising a natural language description; and The auxiliary information is displayed to the user.
2. The method according to claim 1, wherein: The context is defined by at least one or more contextualized facts derived from the user's interactions with the plurality of different computer services.
3. The method according to claim 1, further comprising: A triggering fact is identified to trigger identification of the context, wherein identifying the auxiliary information is based at least on a relationship in the user information graph between the triggering fact and one or more other facts in the user information graph.
4. The method according to claim 3, wherein: The triggering facts include one or more previous interactions of the user with the browser application.
5. The method according to claim 3, wherein: The triggering fact includes a previous interaction of the user with a computer service other than the browser application.
6. The method according to claim 3, wherein: The relationship between the triggering fact and the one or more other facts in the user information graph in the user information graph is a temporal relationship.
7. The method according to claim 3, wherein: Identifying the triggering fact includes evaluating a signal strength score for a candidate fact, the signal strength score indicating a strength of a relationship between the candidate fact and facts in the user information graph, and identifying the triggering fact in response to the signal strength score exceeding a threshold.
8. A system for providing contextual information to a user during a browsing session, comprising: Logical devices; as well as A storage device configured to hold: A user information graph, the graph comprising a plurality of facts, the facts being associated with the user, and is derived from said user's interaction with a plurality of different computer services, wherein, The user information graph includes application-specific component graph structures associated with the plurality of different computer services, each application-specific component graph structure including edges connecting it to one or more other application-specific component graph structures in the application-specific component graph structure, Each application-specific graph structure includes a plurality of nodes; and Instructions executable by the logic device to: identifying triggering facts based on relationships between the application-specific composition graph structures so as to identify a context of the user's interaction with the browser application; as well as Auxiliary information related to the context is identified, the auxiliary information being based on the context and at least on one or more facts in the user information graph, the auxiliary information comprising a natural language description.
9. The system according to claim 8, wherein: The instructions are also executable to receive, from the browser application, a computer-readable description of one or more previous interactions of the user with the browser application, and wherein the triggering fact includes the computer-readable description of the one or more previous interactions.
10. The system according to claim 8, wherein: The triggering facts include a description of the user's previous interactions with a computer service external to the browser application.
11. The system according to claim 8, wherein: The relationship between the triggering fact and the one or more other facts in the user information graph in the user information graph is a temporal relationship.
12. The system according to claim 8, wherein: Identifying the triggering fact includes evaluating a signal strength score for a candidate fact, the signal strength score indicating a strength of a relationship between the candidate fact and facts in the user information graph, and identifying the triggering fact in response to the signal strength score exceeding a threshold.
13. The system according to claim 8, wherein: The auxiliary information includes information based on one or more of the following: 1) emails sent to or from the user, 2) contacts of the user, 3) calendar data of the user, 4) documents, 5) web pages, 6) location data associated with the user, and / or 7) application usage data associated with the user.
Citation Information
Patent Citations
Content recommendations based on browsing information
CN102144243A
Contextual user assistance for cloud services
CN105074659A
Session context modeling for conversational understanding systems
CN106575503A
User-aware notification delivery
CN107210948A
Facilitating discovery of information items using dynamic knowledge graph
CN107408116A
