Semantic representation using structural ontology for assistant systems

Abstract

The invention relates to semantic representation using structural ontology for an assistant system. In one embodiment, a method includes receiving, by an assistant system, a user input including a natural language utterance from a client system associated with a user, determining a semantic representation of the user input based at least in part on a structural ontology defining a relationship between a plurality of actions, objects, and attributes, in one embodiment, a method includes receiving a semantic representation from a client system, sending a request based at least in part on the semantic representation to an agent to execute a task corresponding to a user input, receiving a result of the executed task from the agent, the result associated with the structural ontology, and sending an instruction from an assistant system to a client system to present a response based on the result of the executed task.

Description

[0001] This application is a divisional application of the application filed on October 16, 2020, with application number 202080073029.6 and invention title "Semantic Representation of Usage Structure Ontology for Assistant Systems". priority

[0002] This application claims priority and benefit to U.S. non-provisional patent application No. 16 / 998,423, filed August 20, 2020, and U.S. provisional patent application No. 62 / 923,342, filed October 18, 2019. The disclosures of the aforementioned patent applications are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to database and file management in a network environment, and more particularly to the hardware and software of intelligent assistant systems. background

[0004] Assistant systems can provide information or services to users based on a combination of user input, location awareness, and the ability to access information from various online sources, such as weather conditions, traffic congestion, news, stock prices, user schedules, retail prices, etc. User input can include text (e.g., online chat) (especially in instant messaging apps or other applications), voice, images, motion, or a combination thereof. Assistant systems can perform concierge-type services (e.g., booking dinner, purchasing event tickets, arranging trips) or provide information based on user input. Assistant systems can also perform administrative or data processing tasks based on online information and events without user initiation or interaction. Examples of tasks that can be performed by an assistant system include schedule management (e.g., sending an alert to a user about being late for a dinner appointment due to traffic, updating both parties' schedules, and changing restaurant reservation times). Assistant systems can be implemented through a combination of computing devices, application programming interfaces (APIs), and application proliferation on user devices.

[0005] Social networking systems, including social networking websites, enable their users (such as individuals or organizations) to interact with them and with each other. Social networking systems can utilize user input to create and store user profiles associated with each user. User profiles can include a user's demographic information, communication channels, and information about their personal interests. Social networking systems can also use user input to create and store records of a user's relationships with other users on the social network, and provide services (such as profile / news feed posts, photo sharing, event organization, messaging, games, or advertising) to facilitate social interaction between or among users.

[0006] Social networking systems can send content or messages related to their services to users' mobile devices or other computing devices through one or more networks. Users can also install software applications on their mobile devices or other computing devices to access their user profiles and other data within the social networking system. Social networking systems can generate a set of personalized content objects to display to users, such as a newsfeed that aggregates stories of other users connected to that user. Overview of specific embodiments

[0007] This invention discloses methods, systems, and computer-readable non-transitory storage media according to the appended claims. In certain embodiments, an assistant system can assist a user in obtaining information or services. The assistant system enables the user to interact with it in a stateful and multi-turn conversation using multimodal user input (e.g., voice, text, images, video, motion). By way of example and not limitation, the assistant system may support audio (verbal) input and non-verbal input, such as visual, location, gesture, motion, or mixed / multimodal input. The assistant system can create and store a user profile that includes personal and contextual information associated with the user. In certain embodiments, the assistant system can use natural language understanding to analyze user input. The analysis can be based on the user's user profile to obtain a more personalized and context-aware understanding. The assistant system can resolve entities associated with the user input based on the analysis. In certain embodiments, the assistant system can interact with different agents to obtain information or services associated with the resolved entities. The assistant system can generate responses for the user regarding information or services using natural language generation. Through interaction with users, the assistant system can use conversation management techniques to manage and advance the flow of conversations with users. In certain embodiments, the assistant system can also help users effectively and efficiently digest the information they receive by summarizing it. The assistant system can also help users better participate in online social networks by providing tools to assist them in interacting with online social networks (e.g., creating posts, comments, and messages). The assistant system can also help users manage different tasks, such as continuously tracking events. In certain embodiments, the assistant system can proactively perform tasks related to the user's interests and preferences based on the user profile at user-relevant times without user input. In certain embodiments, the assistant system can check privacy settings to ensure that access to the user's profile or other user information and the execution of different tasks are permitted according to the user's privacy settings.

[0008] In a particular embodiment, the assistant system can assist the user through a hybrid architecture built on client-side and server-side processes. The client-side and server-side processes can be two parallel workflows for processing user input and providing assistance to the user. In a particular embodiment, the client-side process can execute locally on the client system associated with the user. In contrast, the server-side process can execute remotely on one or more computing systems. In a particular embodiment, an arbitrator on the client system can coordinate the reception of user input (e.g., audio signals), determine whether to use the client-side process, the server-side process, or both to respond to the user input, and analyze the processing results from each process. Based on the above analysis, the arbitrator can instruct the client-side or server-side agent to perform the task associated with the user input. The execution result can then be further rendered as output to the client system. By utilizing client-side and server-side processes, the assistant system can effectively help users optimize the use of computing resources while protecting user privacy and enhancing security.

[0009] In certain embodiments, the assistant system may use a structured semantic representation framework, or “next-generation ontology” (NGO), to construct instructions for various modules of the assistant system, including instructions for interfacing with first-party and third-party agents. An NGO may include an ontology with several different types of semantic units, such as actions, objects, and attributes. An NGO may support a tagging system derived from the ontology, allowing users to request subgraphs represented as ontologies. An NGO may allow the assistant system to unify the semantic representations used by the natural language understanding process (i.e., input processing) and the natural language generation process (i.e., output processing). All modules in the assistant system stack can use the same type of semantic representation based on the NGO, rather than their own semantics, thereby reducing mappings between them and increasing consistency. Objects may be hosted throughout the entire stack of the assistant system, where the assistant system can use a dialogue manager to change actions between input and output based on mapping layers. Furthermore, because all parts of the ontology are defined, the assistant system can determine whether any semantic parsing requested by the user is valid before attempting to perform the corresponding task. Although this disclosure describes a particular ontology in a specific manner, this disclosure contemplates any suitable ontology in any suitable manner.

[0010] In a particular embodiment, the assistant system may receive user input, including natural language utterances, from a client system associated with the user via an assistant xbot. The assistant system may then determine a semantic representation of the user input based on a structural ontology. In a particular embodiment, the structural ontology may define a labeling syntax for parsing the natural language utterance into semantic units comprising multiple actions, objects, and attributes. The semantic representation may embed at least one object into at least one action and declare at least one attribute of the embedded object for which the action will be performed. In a particular embodiment, the assistant system may send a request to an agent to perform a task corresponding to the user input. This request may be based on the semantic representation. In a particular embodiment, the assistant system may receive the result of the performed task from the agent, wherein the result is mapped to a structure determined by the structural ontology. The assistant system may also send instructions from the assistant xbot to the client system for presenting a response to the user input, wherein the response is based on the result of the performed task.

[0011] Several technical challenges exist in improving the consistency of semantic representation. One such challenge might include providing third-party users with the flexibility to design their own semantic units while maintaining the integrity of the structural ontology. A solution to this challenge proposed in the embodiments disclosed herein could be defining core and general subgraphs, which are functionally separate, and whose basic structure is maintained by the core subgraph, which is only visible to third-party users. Another technical challenge might include representing complex instances with attribute constraints. A solution to this challenge proposed in the embodiments disclosed herein could be type conversion, which allows objects to be nested within attributes and declares which attribute value of the object is the embedded attribute, thereby representing complex instances with a clearer structure that improves interpretability. Yet another technical challenge could include enabling developers to easily use structural ontology without requiring extensive knowledge of the ontology. A solution to this challenge proposed in the embodiments disclosed herein could be an NGO tool that allows developers to customize the use of structural ontology with different functionalities using a high-level user interface, in which the system can also automatically populate content to facilitate the use of such a tool.

[0012] Certain embodiments disclosed herein may provide one or more technical advantages. These advantages may include reduced duplication of effort, as the NGO allows for the automatic generation of semantic spaces through reusability, eliminates domain distinctions, and supports type hierarchies, which can create reusable tagging elements and reduce duplication of effort. Another advantage of the embodiments may include consistent ontology and type checking, as each object, action, or attribute has a type, enabling the NGO to have formal and consistent agreements on semantic parsing throughout the assistant stack. Another advantage of these embodiments may include hierarchy-aware back-off, as the NGO can encode automatic and systematic back-off for ambiguous or incorrectly tagged utterances and avoids hard-coded rules for back-off that can increase with the domain. Another advantage of the embodiments may include a shared semantic space, as the NGO can directly link concepts and their lexicalizations to annotations without requiring a mapping between the two. Certain embodiments disclosed herein may not provide, provide some or all of the above-described technical advantages. Given the accompanying drawings, description, and claims of this disclosure, one or more other technical advantages may be apparent to those skilled in the art.

[0013] In one aspect, the present invention relates to a method comprising one or more computing systems: The assistant xbot receives user input, including natural language utterances, from the client system associated with the user; The semantic representation of user input is determined based on the structural ontology, wherein the structural ontology defines a markup syntax for parsing natural language discourse into semantic units including multiple actions, objects and attributes, and wherein the semantic representation embeds at least one object into at least one action and declares at least one attribute of the embedded object that will be subject to the action. Send a request to the agent to perform a task corresponding to user input, wherein the request is based on a semantic representation; Receive the results of the executed tasks from the agent, where the results are mapped to a structure determined by the structure ontology; and The assistant xbot sends instructions to the client system to present a response to user input, where the response is based on the result of the task being performed.

[0014] In embodiments of the method according to the present invention, the structural ontology can be defined. Each of the multiple actions operates on one or more of the multiple objects. Each of the multiple objects includes one or more of the multiple properties, or Each of the multiple properties declares a restriction on the action or object.

[0015] In embodiments of the method according to the invention, semantic representation can be performed by each module associated with the assistant xbot configured to process user input.

[0016] In embodiments of the method according to the invention, a structural ontology can define multiple objects hierarchically organized into multiple supertypes and subtypes, each subtype inheriting one or more properties of its associated supertype.

[0017] In embodiments of the method according to the invention, the value associated with each of at least one action, at least one object, and at least one attribute can be determined by each module performing the semantic representation.

[0018] In an embodiment of the method according to the present invention, the method may further include: The response is generated by the natural language generation module associated with the assistant xbot, based on the structure ontology and the results of the task being performed.

[0019] In one embodiment of the method according to the invention, the structural ontology may define semantic units that also include a plurality of methods and enumerations, wherein each of the plurality of methods may include an action whose lifetime is bound to an object, and wherein each of the plurality of enumerations includes an object representing one of the plurality of objects.

[0020] In an embodiment of the method according to the invention, the structural ontology may further define a graph structure including one or more core subgraphs and one or more general subgraphs, wherein one or more core subgraphs are inaccessible by a third-party proxy, and one or more general subgraphs are accessible by a third-party proxy.

[0021] In an embodiment of the method according to the present invention, the method may further include: The reasoning module associated with the assistant xbot resolves one or more entities associated with at least one object based on at least one attribute of the semantic representation.

[0022] In an embodiment of the method according to the invention, the agent may be a first-party agent associated with the assistant xbot.

[0023] In embodiments of the method according to the invention, the agent may be a third-party agent associated with an external computing system, and the structure of the execution result may include one or more customized semantic units.

[0024] In embodiments of the method according to the invention, the agent may be a third-party agent associated with an external computing system, and the structure of the execution result may include one or more custom semantic units, and sending requests to the agent and receiving execution results from the agent may be done via an application programming interface (API), wherein the API defines a mapping from one or more custom semantic units to a structure determined by a structure ontology.

[0025] In an embodiment of the method according to the invention, the agent may be a third-party agent associated with an external computing system, and the structure of the execution result may include one or more customized semantic units, and the request sent to the agent may include a semantic representation modified by a dialogue manager associated with the assistant xbot, wherein the execution result includes a semantic representation modified by the agent, and wherein the request and the execution result use the markup syntax of a structural ontology.

[0026] In an embodiment of the method according to the invention, the agent may be a third-party agent associated with an external computing system, and the structure of the execution result may include one or more custom semantic units, each of which may be associated with a default namespace defined by the computing system, and one or more custom semantic units may be associated with a personalized namespace defined by the third-party agent.

[0027] In an embodiment of the method according to the invention, at least one action may have one or more arguments, wherein each argument may specify a constraint that the object must satisfy, such that at least one action can be applied to the object, and wherein at least one argument specifies at least one constraint that the object must satisfy.

[0028] In one aspect, the present invention relates to a computer-readable non-transitory storage medium comprising software, which, when executed, is operable to perform the above-described methods or: The assistant xbot receives user input, including natural language utterances, from the client system associated with the user; The semantic representation of user input is determined based on the structural ontology, wherein the structural ontology defines a markup syntax for parsing natural language discourse into semantic units including multiple actions, objects and attributes, and wherein the semantic representation embeds at least one object into at least one action and declares at least one attribute of the embedded object that will be subject to the action. Send a request to the agent to perform a task corresponding to user input, wherein the request is based on a semantic representation; Receive the results of the executed tasks from the agent, where the results are mapped to a structure determined by the structure ontology; and The assistant xbot sends instructions to the client system to present a response to user input, where the response is based on the result of the task being performed.

[0029] In one aspect, the present invention relates to a system comprising: one or more processors; and a non-transitory memory coupled to the processors, comprising instructions executable by the processors, which, when executing the instructions, are operable to perform the methods described above or: The assistant xbot receives user input, including natural language utterances, from the client system associated with the user; The semantic representation of user input is determined based on the structural ontology, wherein the structural ontology defines a markup syntax for parsing natural language discourse into semantic units including multiple actions, objects and attributes, and wherein the semantic representation embeds at least one object into at least one action and declares at least one attribute of the embedded object that will be subject to the action. Send a request to the agent to perform a task corresponding to user input, wherein the request is based on a semantic representation; Receive the results of the executed tasks from the agent, where the results are mapped to a structure determined by the structure ontology; and The assistant xbot sends instructions to the client system to present a response to user input, where the response is based on the result of the task being performed.

[0030] The embodiments disclosed herein are merely examples, and the scope of this disclosure is not limited thereto. Specific embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments of the invention are specifically disclosed in the appended claims and relate to methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim class (e.g., methods) may also be claimed in another claim class (e.g., systems). Dependencies or backreferences in the appended claims are chosen only for formal reasons. However, any subject matter arising from intentional backreferences to any prior claim (particularly multiple dependencies) may also be claimed, thereby disclosing and claiming any combination of claims and their features, regardless of the dependencies chosen in the appended claims. Claimable subject matter includes not only combinations of features set forth in the appended claims but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any feature of the appended claims. Brief description of the attached diagram

[0031] Figure 1 An example network environment associated with the assistant system is shown.

[0032] Figure 2 An example architecture of the assistant system is shown.

[0033] Figure 3 An example flowchart of the server-side process of the assistant system is shown.

[0034] Figure 4 An exemplary flowchart is shown, illustrating how an assistant system processes user input.

[0035] Figure 5 An example comparison between IN / SL representation and NGO representation of discourse is shown.

[0036] Figure 6 An example comparison is shown between the media domain represented by the IN / SL paradigm and the media domain represented by the NGO structure.

[0037] Figure 7 An example structure of an NGO subgraph is shown.

[0038] Figure 8A An example subdomain selection is shown.

[0039] Figure 8B An exemplary continuation of subdomain selection is shown.

[0040] Figure 9 This illustrates example inheritance in NGOs.

[0041] Figure 10 An example semantic parsing of the composition is shown.

[0042] Figure 11A An example tree-like NGO form of an example dialogue is shown.

[0043] Figure 11B Another example of a tree-like NGO form is shown in another example dialogue.

[0044] Figure 12 An example NGO parsing based on a plan tree is shown.

[0045] Figure 13A An example execution graph of the discourse is shown.

[0046] Figure 13B It shows the corresponding Figure 13A Example execution graph of two subtasks of the discourse.

[0047] Figure 14 An example response generation based on an NGO is shown.

[0048] Figure 15 The example performance of the model on various percentages of training data is shown while keeping the test set unchanged.

[0049] Figure 16 An example of an app for enabling voice support in NGOs is shown.

[0050] Figure 17A This shows a sample UI for an NGO tool with a "create" trigger action.

[0051] Figure 17B This shows a sample UI for an NGO tool with an "update" trigger action.

[0052] Figure 18 An example method for improving semantic representation consistency is shown.

[0053] Figure 19 An example social graph is shown.

[0054] Figure 20 An example view of the embedded space is shown.

[0055] Figure 21An example artificial neural network is shown.

[0056] Figure 22 An example computer system is shown. Description of Example Implementations

[0057] System Overview Figure 1 An example network environment 100 associated with an assistant system is shown. Network environment 100 includes client systems 130, assistant systems 140, social networking systems 160, and third-party systems 170 connected to each other via network 110. Although Figure 1 A specific arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110 is shown, but this disclosure contemplates any suitable arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110. By way of example and not limitation, two or more of client system 130, social networking system 160, assistant system 140, and third-party system 170 may bypass network 110 and be directly connected to each other. As another example, two or more of client system 130, assistant system 140, social networking system 160, and third-party system 170 may be physically or logically located in the same place as each other, wholly or partially. Furthermore, although... Figure 1 A specific number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110 are shown, but this disclosure contemplates any suitable number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110. As an example and not as a limitation, network environment 100 may include multiple client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110.

[0058] This disclosure contemplates any suitable network 110. By way of example and not limitation, one or more portions of network 110 may include an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 110 may include one or more networks 110.

[0059] Link 150 can connect client system 130, assistant system 140, social networking system 160, and third-party system 170 to communication network 110 or to each other. This disclosure contemplates any suitable link 150. In a particular embodiment, one or more links 150 include one or more wired (such as, for example, Digital Subscriber Line (DSL) or Cable-Based Data Service Interface Specification (DOCSIS)) links, wireless (such as, for example, Wi-Fi or Global Interoperability Microwave Access (WiMAX)) links, or optical (such as, for example, Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In a particular embodiment, one or more links 150 each include an ad hoc network, intranet, extranet, VPN, LAN, WLAN, WAN, WWAN, MAN, a portion of the Internet, a portion of the PSTN, a cellular-based network, a satellite-based network, another link 150, or a combination of two or more such links 150. Links 150 do not need to be identical throughout the network environment 100. One or more first links 150 may differ from one or more second links 150 in one or more respects.

[0060] In a particular embodiment, client system 130 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more such components, and is capable of performing appropriate functions implemented or supported by client system 130. By way of example and not limitation, client system 130 may include computer systems such as desktop computers, laptop or notebook computers, netbooks, tablet computers, e-book readers, GPS devices, cameras, personal digital assistants (PDAs), handheld electronic devices, cellular phones, smartphones, smart speakers, virtual reality (VR) headsets, augmented reality (AR) smart glasses, other suitable electronic devices, or any suitable combination thereof. In a particular embodiment, client system 130 may be a smart assistant device. More information about intelligent assistant devices can be found in U.S. Patent Application No. 15 / 949011, filed April 9, 2018; U.S. Patent Application No. 16 / 153574, filed October 5, 2018; U.S. Design Patent Application No. 29 / 631910, filed January 3, 2018; U.S. Design Patent Application No. 29 / 631747, filed January 2, 2018; U.S. Design Patent Application No. 29 / 631913, filed January 3, 2018; and U.S. Design Patent Application No. 29 / 631914, filed January 3, 2018, each of which is incorporated herein by reference. This disclosure contemplates any suitable client system 130. Client system 130 enables network users at client system 130 to access network 110. Client system 130 enables its users to communicate with other users at other client systems 130.

[0061] In a particular embodiment, client system 130 may include web browser 132 and may have one or more add-ons, plugins, or other extensions. A user at client system 130 may enter a Uniform Resource Locator (URL) or another address that directs web browser 132 to a specific server (e.g., server 162 or a server associated with third-party system 170), and web browser 132 may generate a Hypertext Transfer Protocol (HTTP) request and pass the HTTP request to the server. The server may accept the HTTP request and, in response, pass one or more Hypertext Markup Language (HTML) files to client system 130. Client system 130 may display a web interface (e.g., a webpage) based on the HTML files from the server for presentation to the user. This disclosure contemplates any suitable source file. By way of example and not limitation, a web interface may be displayed based on HTML files, Extensible Hypertext Markup Language (XHTML) files, or Extensible Markup Language (XML) files, depending on specific needs. Such an interface may also execute scripts, markup languages, and combinations thereof. In this article, references to a web interface include one or more corresponding source files (which the browser can use to display the web interface), and vice versa, where appropriate.

[0062] In a particular embodiment, client system 130 may include a social networking application 134 installed on client system 130. Users at client system 130 can use social networking application 134 to access online social networks. Users at client system 130 can use social networking application 134 to communicate with their social connections (e.g., friends, followers, followed accounts, contacts, etc.). Users at client system 130 can also use social networking application 134 to interact with multiple content objects (e.g., posts, news articles, temporary content, etc.) on the online social network. By way of example and not limitation, users can use social networking application 134 to browse trending topics and breaking news.

[0063] In a particular embodiment, client system 130 may include assistant application 136. A user of client system 130 may use assistant application 136 to interact with assistant system 140. In a particular embodiment, assistant application 136 may include a standalone application. In a particular embodiment, assistant application 136 may be integrated into social networking application 134 or another suitable application (e.g., a messaging application). In a particular embodiment, assistant application 136 may also be integrated into client system 130, assistant hardware device, or any other suitable hardware device. In a particular embodiment, assistant application 136 may be accessed via web browser 132. In a particular embodiment, a user may provide input via different modalities. By way of example and not limitation, modalities may include audio, text, images, video, motion, orientation, etc. Assistant application 136 may transmit user input to assistant system 140. Based on the user input, assistant system 140 may generate a response. Assistant system 140 may send the generated response to assistant application 136. Assistant application 136 may then present the response to the user of client system 130. The presented response may be based on different modalities, such as audio, text, images, and video. As an example, and not a limitation, a user can verbally inquire about traffic information from the assistant application 136 by speaking into the microphone of the client system 130 (i.e., via audio modality). The assistant application 136 can then transmit the request to the assistant system 140. The assistant system 140 can generate a response accordingly and send it back to the assistant application 136. The assistant application 136 can also present the response to the user in text and / or images on the display of the client system 130.

[0064] In a particular embodiment, assistant system 140 can assist a user in retrieving information from different sources. Assistant system 140 can also assist a user in requesting services from different service providers. In a particular embodiment, assistant system 140 can receive a user's request for information or services via assistant application 136 in client system 130. Assistant system 140 can use natural language understanding to analyze the user's request based on the user profile and other relevant information. The results of the analysis may include different entities associated with online social networks. Assistant system 140 can then retrieve information or request services associated with these entities. In a particular embodiment, when retrieving information or requesting services for a user, assistant system 140 can interact with social network system 160 and / or third-party system 170. In a particular embodiment, assistant system 140 can use natural language generation technology to generate personalized communication content for the user. Personalized communication content may include, for example, the status of retrieved information or requested services. In a particular embodiment, assistant system 140 enables the user to interact with it regarding information or services in stateful and multi-turn conversations using dialogue management technology. (See below) Figure 2The discussion describes the functions of the assistant system 140 in more detail.

[0065] In a particular embodiment, the social networking system 160 may be a network-addressable computing system capable of hosting online social networks. The social networking system 160 may generate, store, receive, and transmit social network data (such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to the online social network). The social networking system 160 may be accessed directly or via network 110 by other components of the network environment 100. By way of example and not limitation, the client system 130 may access the social networking system 160 directly or via network 110 using a web browser 132 or a native application associated with the social networking system 160 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In a particular embodiment, the social networking system 160 may include one or more servers 162. Each server 162 may be a unitary server or a distributed server spanning multiple computers or multiple data centers. Server 162 can be of various types, including, and without limitation, web servers, news servers, mail servers, messaging servers, advertising servers, file servers, application servers, exchange servers, database servers, proxy servers, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 162 may include hardware, software, or embedded logic components, or a combination of two or more such components for performing appropriate functions implemented or supported by server 162. In a particular embodiment, social networking system 160 may include one or more data stores 164. Data stores 164 can be used to store various types of information. In a particular embodiment, the information stored in data stores 164 may be organized according to a particular data structure. In a particular embodiment, each data store 164 may be a relational database, a columnar database, a relevance database, or other suitable database. Although this disclosure describes or illustrates specific types of databases, this disclosure contemplates any suitable type of database. Specific embodiments may provide an interface that enables client system 130, social network system 160, assistant system 140, or third-party system 170 to manage, retrieve, modify, add, or delete information stored in data storage 164.

[0066] In a particular embodiment, the social network system 160 may store one or more social graphs in one or more data stores 164. In a particular embodiment, the social graph may include multiple nodes—which may include multiple user nodes (each corresponding to a specific user) or multiple concept nodes (each corresponding to a specific concept)—and multiple edges connecting the nodes. The social network system 160 may provide users of the online social network with the ability to communicate and interact with other users. In a particular embodiment, a user may join an online social network via the social network system 160 and then add connections (e.g., relationships) with multiple other users in the social network system 160 that they wish to associate with. In this document, the term "friend" may refer to any other user of the social network system 160 with whom the user has formed an association or relationship via the social network system 160.

[0067] In a particular embodiment, social networking system 160 may provide users with the ability to take action on various types of items or objects supported by social networking system 160. By way of example and not limitation, items and objects may include groups or social networks to which a user of social networking system 160 can belong, events or calendar entries that a user may be interested in, computer-based applications that a user can use, transactions that allow a user to buy or sell goods via a service, interactions with advertisements that a user can perform, or other suitable items or objects. Users may interact with anything that can be represented in social networking system 160 or by an external system, third-party system 170, which is decoupled from social networking system 160 and coupled to social networking system 160 via network 110.

[0068] In a particular embodiment, the social networking system 160 is capable of linking various entities. By way of example and not limitation, the social networking system 160 may enable users to interact with each other and receive content from third-party systems 170 or other entities, or allow users to interact with these entities through application programming interfaces (APIs) or other communication channels.

[0069] In certain embodiments, third-party system 170 may include one or more types of servers, one or more data storage devices, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable components (e.g., servers may communicate with these components). Third-party system 170 may be operated by an entity different from the entity operating social networking system 160. However, in certain embodiments, social networking system 160 and third-party system 170 may operate in combination to provide social networking services to users of either social networking system 160 or third-party system 170. In this sense, social networking system 160 may provide a platform or backbone that other systems (such as third-party system 170) can use to provide social networking services and functionality to users across the Internet.

[0070] In a particular embodiment, third-party system 170 may include a third-party content object provider. The third-party content object provider may include one or more sources of content objects that can be delivered to client system 130. As an example, and not a limitation, the content object may include information about things or activities that a user is interested in, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example, and not a limitation, the content object may include incentivized content objects (e.g., coupons, discount vouchers, gift certificates, or other suitable incentives). In a particular embodiment, the third-party content provider may use one or more third-party proxies to deliver content objects and / or services. The third-party proxies may be implementations hosted and executed on third-party system 170.

[0071] In a particular embodiment, the social networking system 160 also includes user-generated content objects that can enhance user interaction with the social networking system 160. User-generated content can include any content that a user can add, upload, send, or "post" to the social networking system 160. As an example, and not as a limitation, a user transmits a post from client system 130 to social networking system 160. A post can include data such as status updates or other text data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to the social networking system 160 by third parties via "communication channels" such as news feeds or streams.

[0072] In certain embodiments, the social networking system 160 may include various servers, subsystems, programs, modules, logs, and data stores. In certain embodiments, the social networking system 160 may include one or more of the following: a web server, an action logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertising-targeting module, a user interface module, a user profile store, a relationship store, a third-party content store, or a location store. The social networking system 160 may also include suitable components such as network interfaces, security mechanisms, load balancers, failover servers, management and network operations consoles, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 160 may include one or more user profile stores for storing user profiles. User profiles may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, affinity, or location). Interest information may include interests associated with one or more categories. Categories can be general or specific. As an example, and not a limitation, if a user “likes” an article about a brand of shoes, the category could be the brand, or the general category of “shoes” or “clothing.” A relationship store can be used to store relationship information about a user. Relationship information can indicate users who have similar or shared work experience, group memberships, hobbies, educational history, or are related or share common attributes in any way. Relationship information can also include user-defined relationships between different users and content (internal and external). A web server can be used to link the social networking system 160 to one or more client systems 130 or one or more third-party systems 170 via network 110. The web server can include a mail server or other messaging functionality for receiving and routing messages between the social networking system 160 and one or more client systems 130. An API request server can allow, for example, an assistant system 140 or a third-party system 170 to access information from the social networking system 160 by calling one or more APIs. An action recorder can be used to receive communications from the web server regarding user actions on or outside the social networking system 160. By combining action logs, a log of third-party content objects exposed to users can be maintained. The notification controller can provide information about the content objects to the client system 130. The information can be pushed to the client system 130 as a notification, or it can be retrieved from the client system 130 in response to a request received from the client system 130.An authorization server can be used to implement one or more privacy settings for users of the social networking system 160. A user's privacy settings determine how specific information associated with that user can be shared. The authorization server can, for example, allow users to opt in or out by setting appropriate privacy settings, enabling their actions to be recorded by the social networking system 160 or shared with other systems (e.g., third-party system 170). A third-party content object store can be used to store content objects received from third parties (e.g., third-party system 170). A location store can be used to store location information received from the client system 130 associated with the user. An advertising pricing module can combine social information, current time, location information, or other suitable information to provide relevant advertisements to users in the form of notifications.

[0073] Assistant System Figure 2An example architecture of assistant system 140 is shown. In a particular embodiment, assistant system 140 can assist a user in obtaining information or services. Assistant system 140 enables users to interact with it in stateful and multi-turn conversations using multimodal user input (such as voice, text, images, video, motion) to obtain assistance. By way of example and not limitation, assistant system 140 may support audio input (verbal) and non-verbal input, such as visual, location, gesture, motion, or mixed / multimodal input. Assistant system 140 may create and store user profiles that include personal and contextual information associated with the user. In a particular embodiment, assistant system 140 may use natural language understanding to analyze user input. The analysis may be based on the user's user profile to obtain a more personalized and context-aware understanding. Assistant system 140 may use the analysis to resolve entities associated with the user input. In a particular embodiment, assistant system 140 may interact with different agents to obtain information or services associated with the resolved entities. Assistant system 140 may generate responses for the user regarding information or services by using natural language generation. Through interaction with the user, the assistant system 140 can use conversation management techniques to manage and forward the conversation flow with the user. In a particular embodiment, the assistant system 140 can also help the user effectively and efficiently process the information received by aggregating information. The assistant system 140 can also help the user participate more in online social networks by providing tools to help the user interact with online social networks (e.g., create posts, comments, messages). The assistant system 140 can also help the user manage different tasks, such as continuously tracking events. In a particular embodiment, the assistant system 140 can proactively perform pre-authorized tasks related to the user's interests and preferences based on the user's profile at user-related times, without user input. In a particular embodiment, the assistant system 140 can check privacy settings to ensure that access to the user's profile or other user information and the performance of different tasks are permitted according to the user's privacy settings. More information on assisting users subject to privacy settings can be found in U.S. Patent Application No. 16 / 182542, filed November 6, 2018, which is incorporated herein by reference.

[0074] In a particular embodiment, the assistant system 140 can assist the user through a hybrid architecture built on client-side and server-side processes. The client-side and server-side processes can be two parallel workflows for processing user input and providing assistance to the user. In a particular embodiment, the client-side process can execute locally on the client system 130 associated with the user. In contrast, the server-side process can execute remotely on one or more computing systems. In a particular embodiment, an assistant orchestrator on the client system 130 can coordinate the reception of user input (e.g., audio signals) and determine whether to respond to the user input using the client-side process, the server-side process, or both. A dialogue arbitrator can analyze the processing results from each process. Based on the above analysis, the dialogue arbitrator can instruct the client-side or server-side agent to perform the task associated with the user input. The execution result can be further rendered as output to the client system 130. By utilizing client-side and server-side processes, the assistant system 140 can effectively help the user optimize the use of computing resources while protecting user privacy and enhancing security.

[0075] In a particular embodiment, assistant system 140 may receive user input from client system 130 associated with the user. In a particular embodiment, user input may be user-generated input sent to assistant system 140 in a single round. User input may be verbal, nonverbal, or a combination thereof. By way of example, and not limitation, nonverbal user input may be based on the user's voice, vision, location, activity, gestures, actions, or a combination thereof. If user input is based on the user's voice (e.g., the user may speak to client system 130), such user input may first be processed by system audio API 202 (application programming interface). System audio API 202 may perform echo cancellation, noise removal, beamforming and self-activated user voice, speaker recognition, voice activity detection (VAD), and any other acoustic techniques to generate audio data that can be easily processed by assistant system 140. In a particular embodiment, system audio API 202 may perform wake word detection 204 based on user input. By way of example, and not limitation, a wake word may be "Hey, assistant." If such a wake word is detected, assistant system 140 may be activated accordingly. In an alternative embodiment, the user can activate the assistant system 140 via a visual signal without a wake word. The visual signal can be received at a low-power sensor (e.g., a camera) capable of detecting various visual signals. As an example, and not a limitation, the visual signal could be a barcode, QR code, or Universal Product Code (UPC) detected by the client system 130. As another example, and not a limitation, the visual signal could be the user's gaze at an object. As yet another example, and not a limitation, the visual signal could be a user gesture, such as the user pointing at an object.

[0076] In a particular embodiment, audio data from the system audio API 202 can be sent to the assistant orchestrator 206. The assistant orchestrator 206 can execute on the client system 130. In a particular embodiment, the assistant orchestrator 206 can determine whether to respond to user input using a client process, a server process, or both. Figure 2 As shown, the client process is shown below dashed line 207, while the server process is shown above dashed line 207. The assistant orchestrator 206 can also determine whether to respond to user input by using both the client and server processes simultaneously. Although Figure 2 Assistant orchestrator 206 is shown as a client process, but assistant orchestrator 206 can be a server process, or it can be a hybrid process that is separate between client and server processes.

[0077] In a particular embodiment, after audio data is generated from the system audio API 202, the server-side process may proceed as follows. The assistant orchestrator 206 may send the audio data to a remote computing system that hosts different modules of the assistant system 140 in response to user input. In a particular embodiment, the audio data may be received at a remote automatic speech recognition (ASR) module 208. The ASR module 208 may allow a user to dictate and have speech transcribed into written text, synthesize a document into an audio stream, or issue a command that is recognized by the system as such. The ASR module 208 may use a statistical model to determine the most probable sequence of words corresponding to a given portion of speech received by the assistant system 140 as audio input. The model may include one or more of a hidden Markov model, a neural network, a deep learning model, or any combination thereof. The received audio input may be encoded into digital data at a specific sampling rate (e.g., 16, 44.1, or 96 kHz) and have a specific number of bits representing each sample (e.g., 8 or 16 bits out of 24 bits).

[0078] In a particular embodiment, ASR module 208 may include different components. ASR module 208 may include one or more of a grapheme-to-phoneme (G2P) model, a pronunciation learning model, a personalized acoustic model, a personalized language model (PLM), or an endpoint model. In a particular embodiment, a G2P model can be used to determine a user's grapheme-to-phoneme style, for example, what it sounds like when a particular user says a particular word. A personalized acoustic model can be a model of the relationship between an audio signal and the sounds of speech units in a language. Thus, such a personalized acoustic model can identify how a user's voice sounds. A personalized acoustic model can be generated using training data, such as training speech received as audio input and corresponding speech units. A personalized acoustic model can be trained or refined using a particular user's voice to recognize that user's speech. In a particular embodiment, a personalized language model can then determine the most likely phrase corresponding to the identified speech units for a particular audio input. A personalized language model can be a probabilistic model of various word sequences that may occur in a language. Personalized language models can be used to match the sounds of speech units in audio input with word sequences, assigning greater weight to word sequences that are more likely to be phrases in that language. The word sequence with the highest weight can then be selected as the text corresponding to the audio input. In a particular embodiment, the personalized language model can also be used to predict what words a user is most likely to say given a context. In a particular embodiment, an endpoint model can detect when the endpoint of a utterance is reached.

[0079] In a particular embodiment, the output of ASR module 208 can be sent to remote natural language understanding (NLU) module 210. NLU module 210 can perform named entity resolution (NER). When analyzing user input, NLU module 210 can additionally consider contextual information. In a particular embodiment, intent and / or slots can be the output of NLU module 210. An intent can be an element in a predefined category of semantic intents that can indicate the purpose of the user's interaction with assistant system 140. NLU module 210 can classify user input into a member of a predefined category; for example, for the input "play Beethoven's Fifth Symphony," NLU module 210 can classify the input as having the intent [IN: play music]. In a particular embodiment, a domain can represent the social context of the interaction, such as education, or a namespace of a set of intents, such as music. A slot can be a named substring corresponding to a string in the user input, representing a basic semantic entity. For example, the slot for "pizza" could be [SL: dish]. In a particular embodiment, the set of valid or expected named slots can be based on the classified intents. As an example, and not a limitation, for the intent [IN: play music], a valid slot could be [SL: song title]. In a particular embodiment, NLU module 210 may additionally extract information from one or more social graphs, knowledge graphs, or concept graphs, and retrieve user profiles from one or more remote data stores 212. NLU module 210 may also process information from these diverse sources by determining what information to aggregate, annotating the user input n-grams, ranking the n-grams with confidence scores based on the aggregated information, and formulating the ranked n-grams into features that can be used by NLU module 210 to understand the user input.

[0080] In a particular embodiment, NLU module 210 can identify one or more domains, intents, or slots from user input in a personalized and context-aware manner. As an example, and not a limitation, user input may include “show me how to get to the coffee shop”. NLU module 210 can identify the specific coffee shop the user wants to go to based on the user's personal information and associated contextual information. In a particular embodiment, NLU module 210 may include a language-specific lexicon, a parser, and grammar rules to segment sentences into internal representations. NLU module 210 may also include one or more programs that use pragmatics to perform naive or stochastic semantic parsing to understand user input. In a particular embodiment, the parser may be based on a deep learning architecture including multiple Long Short-Term Memory (LSTM) networks. As an example, and not a limitation, the parser may be based on a Recurrent Neural Network Grammar (RNNG) model, a type of recurrent and recursive LSTM algorithm. Further information on natural language understanding can be found in U.S. Patent Application No. 16 / 011062, filed June 18, 2018; U.S. Patent Application No. 16 / 025317, filed July 2, 2018; and U.S. Patent Application No. 16 / 038120, filed July 17, 2018, each of which is incorporated herein by reference.

[0081] In a particular embodiment, the output of NLU module 210 can be sent to remote inference module 214. Inference module 214 may include a dialogue manager and an entity resolution component. In a particular embodiment, the dialogue manager may have complex dialogue logic and product-related business logic. The dialogue manager can manage the dialogue state and session flow between the user and assistant system 140. The dialogue manager may additionally store previous sessions between the user and assistant system 140. In a particular embodiment, the dialogue manager can communicate with the entity resolution component to resolve entities associated with one or more slots, which supports the dialogue manager in advancing the session flow between the user and assistant system 140. In a particular embodiment, when resolving entities, the entity resolution component can access one or more of a social graph, knowledge graph, or concept graph. Entities may include, for example, unique users or concepts, each of which may have a unique identifier (ID). By way of example and not limitation, a knowledge graph may include multiple entities. Each entity may include a single record associated with one or more attribute values. A specific record may be associated with a unique entity identifier. Each record may have a different value for one attribute of an entity. Each attribute value may be associated with a confidence probability. The confidence probability of an attribute value represents the probability that the value is accurate for a given attribute. Each attribute value can also be associated with a semantic weight. The semantic weight of an attribute value can represent the degree to which the value is semantically appropriate for a given attribute, taking into account all available information. For example, a knowledge graph can include an entity for the book "Alice's Adventures," which includes information extracted from multiple content sources (e.g., online social networks, online encyclopedias, book review sources, media databases, and entertainment content sources), which is then deduplicated, parsed, and merged to generate a single unique record for the knowledge graph. This entity can be associated with the attribute value "fantasy," which indicates the type of the book "Alice's Adventures." More information about knowledge graphs can be found in U.S. Patent Application No. 16 / 048049, filed July 27, 2018, and U.S. Patent Application No. 16 / 048101, filed July 27, 2018, each of which is incorporated herein by reference.

[0082] In certain embodiments, the entity parsing component may check privacy constraints to ensure that entity parsing does not violate privacy policies. As an example, and not a limitation, the entity to be parsed could be another user whose privacy settings specify that their identity should not be searchable on the online social network; therefore, the entity parsing component may not return that user's identifier in response to a request. Based on information obtained from social graphs, knowledge graphs, concept graphs, and user profiles, and in accordance with applicable privacy policies, the entity parsing component can therefore parse entities associated with user input in a personalized, context-aware, and privacy-aware manner. In certain embodiments, each parsed entity may be associated with one or more identifiers hosted by the social network system 160. As an example, and not a limitation, identifiers may include a unique user identifier (ID) corresponding to a specific user (e.g., a unique username or user ID number). In certain embodiments, each parsed entity may also be associated with a confidence score. More information about the resolving entity can be found in U.S. Patent Application No. 16 / 048049, filed July 27, 2018, and U.S. Patent Application No. 16 / 048072, filed July 27, 2018, each of which is incorporated herein by reference.

[0083] In certain embodiments, the dialogue manager may perform dialogue optimization and assistant state tracking. Dialogue optimization is the problem of using data to understand what the most likely branch of the dialogue should be. As an example, and not a limitation, using dialogue optimization, the assistant system 140 may not need to confirm who the user wants to call, because the assistant system 140 has a high confidence that the person inferred based on dialogue optimization is likely the person the user wants to call. In certain embodiments, the dialogue manager may use reinforcement learning for dialogue optimization. Assistant state tracking aims to track the state that changes over time as the user interacts with the world and the assistant system 140 interacts with the user. As an example, and not a limitation, assistant state tracking may track what the user is talking about, who the user is with, where the user is, what task the user is currently performing, and where the user is gazing, etc., in accordance with applicable privacy policies. In certain embodiments, the dialogue manager may use a set of operators to track the dialogue state. Operators may include the necessary data and logic to update the dialogue state. Each operator may act as an increment of the dialogue state after processing an incoming request. In certain embodiments, the dialogue manager may further include a dialogue state tracker and an action selector. In alternative embodiments, the dialogue state tracker may replace the entity resolution component and resolve references / mentions and track the state.

[0084] In a particular embodiment, the inference module 214 may further perform error trigger mitigation. The goal of error trigger mitigation is to detect erroneous triggers of assistant requests (e.g., wake words) and avoid generating error logs when the user does not actually intend to invoke the assistant system 140. As an example, and not a limitation, the inference module 214 may implement error trigger mitigation based on a meaninglessness detector. If the meaninglessness detector determines that the wake word is meaningless at this point in the user interaction, the inference module 214 may determine that inferring the user's intention to invoke the assistant system 140 is likely incorrect. In a particular embodiment, the output of the inference module 214 may be sent to a remote dialogue arbitrator 216.

[0085] In a particular embodiment, each of the ASR module 208, NLU module 210, and inference module 214 may access remote data storage 212, which includes user episode memories to determine how to more effectively assist the user. Further information regarding episode memories can be found in U.S. Patent Application No. 16 / 552559, filed August 27, 2019, which is incorporated herein by reference. Data storage 212 may additionally store a user profile. A user profile may include profile data including demographic, social, and contextual information associated with the user. The profile data may also include the user's interests and preferences on multiple topics aggregated through conversations on news feeds, search logs, messaging platforms, etc. Use of user profiles may be subject to privacy restrictions to ensure that a user's information is used only for his / her benefit and not shared with any other person. Further information regarding user profiles can be found in U.S. Patent Application No. 15 / 967239, filed April 30, 2018, which is incorporated herein by reference.

[0086] In a particular embodiment, the client process can run in parallel with the aforementioned server-side processes involving ASR module 208, NLU module 210, and inference module 214. In a particular embodiment, the output of assistant orchestrator 206 can be sent to a local ASR module 216 on client system 130. ASR module 216 may include a personalized language model (PLM), a G2P model, and an endpoint model. Due to the limited computing power of client system 130, assistant system 140 can optimize the personalized language model at runtime during the client process. By way of example and not limitation, assistant system 140 can pre-compute multiple personalized language models for multiple possible topics the user might discuss. When the user requests help, assistant system 140 can then quickly exchange these pre-computed language models, allowing the personalized language model to be locally optimized by assistant system 140 at runtime based on user activity. As a result, assistant system 140 can have the technical advantage of saving computing resources while efficiently determining what the user might be talking about. In a particular embodiment, assistant system 140 can also quickly relearn the user's pronunciation at runtime.

[0087] In a particular embodiment, the output of ASR module 216 may be sent to local NLU module 218. In a particular embodiment, NLU module 218 may be more compact than a server-supported remote NLU module 210. When ASR module 216 and NLU module 218 process user input, they may access local assistant memory 220. For user privacy purposes, local assistant memory 220 may differ from user memory stored on data storage 212. In a particular embodiment, local assistant memory 220 may be synchronized with user memory stored on data storage 212 via network 110. By way of example and not limitation, local assistant memory 220 may synchronize the calendar on user client system 130 with the server-side calendar associated with the user. In a particular embodiment, any secure data in local assistant memory 220 may only be accessed by modules of assistant system 140 that execute locally on client system 130.

[0088] In a particular embodiment, the output of NLU module 218 can be sent to local inference module 222. Inference module 222 may include a dialogue manager and entity parsing components. Due to limited computing power, inference module 222 can perform on-device learning based on learning algorithms specifically tailored for client system 130. By way of example and not limitation, inference module 222 may use federated learning. Federated learning is a specific category of distributed machine learning methods that uses distributed data residing on terminal devices such as mobile phones to train machine learning models. In a particular embodiment, inference module 222 may use a specific federated learning model, namely federated user representation learning, to extend existing neural network personalization techniques to federated learning. Federated user representation learning can personalize the model in federated learning by learning task-specific user representations (i.e., embeddings) or by personalizing model weights. Federated user representation learning is simple, scalable, privacy-preserving, and resource-efficient. Federated user representation learning can categorize model parameters into federated parameters and private parameters. Private parameters, such as private user embeddings, can be trained locally on client system 130 instead of being transmitted to a remote server or averaged on a remote server. In contrast, federated parameters can be trained remotely on a server. In a particular embodiment, inference module 222 may use another specific federated learning model, namely active federated learning, to transfer a global model trained on a remote server to client systems 130 and compute gradients locally on these client systems 130. Active federated learning allows the inference module to minimize the transmission costs associated with downloading the model and uploading gradients. For active federated learning, in each round, client systems are not uniformly randomized but probabilistically selected conditioned on the current model and the data on the client systems to maximize efficiency. In a particular embodiment, inference module 222 may use another specific federated learning model, namely federated Adam. Traditional federated learning models may use stochastic gradient descent (SGD) optimizers. In contrast, the federated Adam model may use a moment-based optimizer. The federated Adam model may use an averaged model to compute approximate gradients instead of directly using an averaged model as in traditional work. These gradients can then be fed into the federated Adam model, which can remove noise from stochastic gradients and use an adaptive learning rate for each parameter. Gradients generated by federated learning may be noisier than those generated by stochastic gradient descent (because the data may not be independent and identically distributed), so the federated Adam model may help to handle noise better. The federated Adam model can use gradients to take smarter steps to minimize the objective function. Experiments show that traditional federated learning on the baseline has a 1.6% decrease in the ROC (Receiver Operating Characteristic) curve, while the federated Adam model only has a 0.4% decrease. Furthermore, the federated Adam model does not add computation to the communication or on-device.In a particular embodiment, the inference module 222 may also perform error trigger mitigation. This error trigger mitigation can help detect erroneous activation requests, such as wake words, on the client system 130 when the user's voice input includes privacy-constrained data. By way of example and not limitation, when a user is in the middle of a voice call, the user's session is private, and error trigger detection based on this session can only occur locally on the user's client system 130.

[0089] In a particular embodiment, assistant system 140 may include a local context engine 224. Context engine 224 may process all other available signals to provide more information prompts to inference module 222. By way of example, and not limitation, context engine 224 may have human-related information, sensing data from sensors (e.g., microphones, cameras) of client system 130 (which are further analyzed using computer vision techniques), geometry, activity data, inertial data (e.g., data collected by a VR headset), location, etc. In a particular embodiment, computer vision techniques may include human skeleton reconstruction, face detection, face recognition, hand tracking, eye tracking, etc. In a particular embodiment, geometry may include constructing objects around the user using data collected by client system 130. By way of example, and not limitation, the user may be wearing AR glasses, and the geometry may be designed to determine where the floor is, where the walls are, where the user's hands are, etc. In a particular embodiment, inertial data may be data associated with linear and angular motion. By way of example, and not limitation, inertial data may be captured by AR glasses that measure how parts of the user's body move.

[0090] In a particular embodiment, the output of the local inference module 222 can be sent to the dialogue arbitrator 216. The dialogue arbitrator 216 can operate differently in three scenarios. In the first scenario, the assistant orchestrator 206 determines to use the server-side process; for this purpose, the dialogue arbitrator 216 can transmit the output of the inference module 214 to the remote action execution module 226. In the second scenario, the assistant orchestrator 206 determines to use both the server-side and client-side processes; for this purpose, the dialogue arbitrator 216 can aggregate and analyze the outputs of the two inference modules (i.e., the remote inference module 214 and the local inference module 222) from the two processes. By way of example and not limitation, the dialogue arbitrator 216 can perform sorting and select the best inference result to respond to user input. In a particular embodiment, the dialogue arbitrator 216 can also determine, based on analysis, whether to use an agent on the server or the client to perform the relevant task. In the third scenario, the assistant orchestrator 206 determines to use the client-side process, and the dialogue arbitrator 216 needs to evaluate the output of the local inference module 222 to determine whether the client-side process is capable of handling the user input task. In an alternative embodiment, if the assistant orchestrator 206 determines that the client process is being used and that the client process is fully capable of handling user input, the output of the inference module 222 may not be sent to the dialogue arbitrator 216.

[0091] In a particular embodiment, for the first and second scenarios described above, the dialogue arbitrator 216 can determine that a server-side agent is necessary to perform a task in response to user input. Therefore, the dialogue arbitrator 216 can send the necessary information about the user input to the action execution module 226. The action execution module 226 can invoke one or more agents to perform the task. In an alternative embodiment, the dialogue manager's action selector can determine the action to be performed and accordingly instruct the action execution module 226. In a particular embodiment, an agent can be an implementation that acts as a broker between multiple content providers in a domain. A content provider can be an entity responsible for performing an action associated with an intent or completing a task associated with an intent. In a particular embodiment, agents can include first-party agents and third-party agents. In a particular embodiment, a first-party agent can include an internal agent accessible and controllable by the assistant system 140 (e.g., an agent associated with a service provided by an online social network, such as a messaging service or photo-sharing service). In a particular embodiment, a third-party agent can include an external agent that the assistant system 140 cannot control (e.g., a third-party online music application agent, a ticket sales agent). First-party agents are associated with first-party providers offering content objects and / or services hosted by social networking system 160. Third-party agents may be associated with third-party providers offering content objects and / or services hosted by third-party system 170. In a particular embodiment, each of the first-party or third-party agents may be designated for a specific domain. By way of example, and not limitation, the domain may include weather, traffic, music, etc. In a particular embodiment, assistant system 140 may use multiple agents collaboratively to respond to user input. By way of example, and not limitation, user input may include “direct me to my next meeting”. Assistant system 140 may use a calendar agent to retrieve the location of the next meeting. Assistant system 140 may then use a navigation agent to guide the user to the next meeting.

[0092] In a specific embodiment, for the second and third scenarios described above, the dialogue arbitrator 216 can determine that the agent on the client side is capable of performing a task in response to user input, but requires additional information (e.g., a response template), or that the task can only be handled by the agent on the server side. If the dialogue arbitrator 216 determines that the task can only be handled by the agent on the server side, then the dialogue arbitrator 216 can send the necessary information about the user input to the action execution module 226. If the dialogue arbitrator 216 determines that the agent on the client side is capable of performing the task, but requires a response template, then the dialogue arbitrator 216 can send the necessary information about the user input to the remote response template generation module 228. The output of the response template generation module 228 can also be sent to the local action execution module 230 executed on the client system 130. In a specific embodiment, if the assistant orchestrator 206 determines that the client process is used, and the client process is fully capable of handling user input, then the output of the inference module 222 can be directly sent to the action execution module 230.

[0093] In a particular embodiment, the action execution module 230 may invoke a local agent to perform a task. The local agent on the client system 130 is capable of performing simpler tasks compared to a server-side agent. As an example, and not a limitation, multiple device-specific implementations (e.g., real-time invocations of the client system 130 or messaging applications on the client system 130) may be handled internally by a single agent. Alternatively, these device-specific implementations may be handled by multiple agents associated with multiple domains. In a particular embodiment, the action execution module 230 may additionally execute a set of generic executable dialogue actions. This set of executable dialogue actions can interact with the agent, user, and assistant system 140 itself. These dialogue actions may include dialogue actions for slot requests, acknowledgments, ambiguity resolution, agent execution, etc. Dialogue actions can be independent of the underlying implementation of the action selector or dialogue policy. Both tree-based and model-based policies can generate the same basic dialogue actions, and callback functions hide any implementation details specific to the action selector.

[0094] In a particular embodiment, the output from the remote action execution module 226 on the server side can be sent to the remote response execution module 232. In a particular embodiment, the action execution module 226 can send back more information to the dialogue arbitrator 216. The response execution module 232 can be based on a remote session understanding (CU) writer. In a particular embodiment, the output from the action execution module 226 can be formulated as follows: <k,c,u,d> tuple, where k Indicating knowledge source, c Indicates communication target, u Instructions for user models and dThe CU writer may include a natural language generation (NLG) module and a user interface (UI) payload generator. The NLG module may generate communication content based on the output of the action execution module 226 using different language models and / or language templates. In a particular embodiment, the generation of communication content may be application-specific and personalized for each user. The CU writer may also use the UI payload generator to determine the modality of the generated communication content. In a particular embodiment, the NLG module may include a content determination component, a sentence planner, and a surface realization component. The content determination component may determine the communication content based on knowledge sources, communication goals, and user expectations. As an example, and not a limitation, determination may be based on description logic. Description logic may include, for example, three basic notions: an individual (representing an object in a domain), a concept (describing a set of individuals), and a role (representing a binary relationship between individuals or concepts). Description logic may be characterized by a set of constructors that allow the NLG module to construct complex concepts / roles from atomic notions / roles. In a particular embodiment, the content determination component may perform the following tasks to determine the communication content. The first task may include a translation task, where input to the natural language generator can be translated into concepts. The second task may include a selection task, where relevant concepts can be selected from the concepts generated by the translation task based on a user model. The third task may include a verification task, where the consistency of the selected concepts can be verified. The fourth task may include an instantiation task, where the verified concepts can be instantiated into an executable file that can be processed by the natural language generator. A sentence planner can determine the organization of the communication content to make it human-readable. A surface implementation component can determine the specific words to use, the sentence order, and the style of the communication content. A UI payload generator can determine the preferred modality of the communication content to be presented to the user. In a particular embodiment, the CU writer can check privacy constraints associated with the user to ensure that the generation of the communication content complies with a privacy policy. Further information on natural language generation can be found in U.S. Patent Application No. 15 / 967279, filed April 30, 2018, and U.S. Patent Application No. 15 / 966455, filed April 30, 2018, each of which is incorporated herein by reference.

[0095] In a particular embodiment, the output from the local action execution module 230 on the client system 130 can be sent to the local response execution module 234. The response execution module 234 can be based on a local conversation understanding (CU) writer. The CU writer may include a natural language generation (NLG) module. Since the computing power of the client system 130 may be limited, the NLG module can be simple for computational efficiency. Because the NLG module can be simple, the output of the response execution module 234 can be sent to the local response extension module 236. The response extension module 236 can further extend the results of the response execution module 234 to make the response more natural and contain richer semantic information.

[0096] In a particular embodiment, if the user input is based on an audio signal, the output of the server-side response execution module 232 can be sent to the remote text-to-speech (TTS) module 238. Similarly, the output of the client-side response extension module 236 can be sent to the local TTS module 240. Both TTS modules can convert the response into an audio signal. In a particular embodiment, the output from the response execution module 232, the response extension module 236, or the TTS modules at both ends can ultimately be sent to the local rendering output module 242. The rendering output module 242 can generate a response suitable for the client system 130. By way of example and not limitation, the output of the response execution module 232 or the response extension module 236 may include one or more of a natural language string, speech, parameterized actions, or rendered images or videos that can be displayed in a VR headset or AR smart glasses. As a result, the rendering output module 242 can determine what task to perform based on the output of the CU writer to appropriately render the response for display on a VR headset or AR smart glasses. For example, the response may be a visual modality (e.g., an image or video clip) that can be displayed on a VR headset or AR smart glasses. As another example, the response could be an audio signal that the user can play through a VR headset or AR smart glasses. As yet another example, the response could be augmented reality data, which can be rendered by a VR headset or AR smart glasses to enhance the user experience.

[0097] In certain embodiments, the assistant system 140 may possess various capabilities, including audio recognition, visual recognition, signal intelligence, reasoning, and memory. In certain embodiments, the audio recognition capability enables the assistant system 140 to understand user input associated with various domains of different languages, understand and summarize conversations, perform audio recognition on complex command execution devices, recognize users through speech, extract topics from conversations and automatically tag parts of conversations, enable audio interaction without wake words, filter and amplify user speech from ambient noise and conversations, and understand which client system 130 the user is talking to (if multiple client systems 130 are nearby).

[0098] In a particular embodiment, the visual recognition capability enables the assistant system 140 to perform face detection and tracking, identify users, identify most people of interest in major metropolitan areas from different angles, identify most objects of interest in the world through a combination of existing machine learning models and one-time learning, identify moments of interest and automatically capture them, achieve semantic understanding of multiple visual frames across different time periods, provide platform support for additional capabilities for people, places, and objects recognition, identify full-set settings and micro-locations including personalized locations, identify complex activities, recognize complex gestures to control the client system 130, process images / videos from egocentric cameras (e.g., with motion, capture angle, resolution, etc.), achieve accuracy and speed regarding similarity levels for images with lower resolution, perform one-time registration and recognition of people, places, and objects, and perform visual recognition on the client system 130.

[0099] In certain embodiments, assistant system 140 may utilize computer vision techniques to achieve visual cognition. In addition to computer vision techniques, assistant system 140 may explore options that complement these techniques to expand object recognition. In certain embodiments, assistant system 140 may use supplementary signals, such as optical character recognition (OCR) for object tags, GPS signals for location recognition, and signals from user client system 130, to identify the user. In certain embodiments, assistant system 140 may perform general scene recognition (home, work, public space, etc.) to set context for the user and narrow the computer vision search space to identify the most likely objects or people. In certain embodiments, assistant system 140 may guide the user to train assistant system 140. For example, crowdsourcing may be used to allow users to tag and help assistant system 140 recognize more objects over time. As another example, when using assistant system 140, users can register their personal objects as part of the initial setup. Assistant system 140 may also allow users to provide positive / negative signals for the objects they interact with in order to train and improve personalized models used for them.

[0100] In a particular embodiment, the signal intelligence capability enables the assistant system 140 to determine the user's location, understand the date / time, determine the home location, understand the user's calendar and expected future location, integrate richer voice understanding to identify settings / context by voice alone, and build a signal intelligence model at runtime that can be personalized for the user's personal routine.

[0101] In a particular embodiment, reasoning capabilities enable the assistant system 140 to pick up any previous conversation thread at any point in the future, synthesize all signals to understand micro and personalized context, learn interaction patterns and preferences from the user’s historical behavior and accurately suggest interactions they may value, generate highly predictive proactive suggestions based on micro context understanding, understand what content the user may want to see at what time of day, and understand changes in the scene and how this may affect the content the user expects.

[0102] In a particular embodiment, the memory capability enables the assistant system 140 to remember which social connections the user previously invoked or interacted with, write to and query memory at will (i.e., open dictation and automatic tagging), extract richer preferences based on previous interactions and long-term learning, remember the user's life history, extract rich information from egocentric data streams and automatic catalogs, and write to memory in a structured form to form rich short-term, episodic, and long-term memories.

[0103] Figure 3An exemplary flowchart of the server-side process of the assistant system 140 is shown. In a particular embodiment, the server assistant service module 301 may access the request manager 302 upon receiving a user request. In an alternative embodiment, if the user request is based on an audio signal, the user request may first be processed by the remote ASR module 208. In a particular embodiment, the request manager 302 may include a context extractor 303 and a session understanding object generator (CU object generator) 304. The context extractor 303 may extract context information associated with the user request. The context extractor 303 may also update the context information based on the assistant application 136 executing on the client system 130. By way of example, and not limitation, updating the context information may include displaying content items on the client system 130. By way of another example, and not limitation, updating the context information may include whether an alarm clock is set on the client system 130. By way of another example, and not limitation, updating the context information may include whether a song is playing on the client system 130. The CU object generator 304 may generate specific content objects associated with the user request. The content object may include dialogue session data and features associated with the user request, which can be shared with all modules of the assistant system 140. In a particular embodiment, the request manager 302 may store context information and the generated content object in a data storage 212, which is a specific data storage implemented in the assistant system 140.

[0104] In a particular embodiment, the request manager 302 may send the generated content object to a remote NLU module 210. The NLU module 210 may perform multiple steps to process the content object. In step 305, the NLU module 210 may generate a whitelist of content objects. In a particular embodiment, the whitelist may include explanatory data matching the user request. In step 306, the NLU module 210 may perform feature identification based on the whitelist. In step 307, the NLU module 210 may perform domain classification / selection on the user request based on the features generated from the feature identification to classify the user request into a predefined domain. The domain classification / selection results may also be further processed based on two related processes. In step 308a, the NLU module 210 may use an intent classifier to process the domain classification / selection results. The intent classifier may determine the user intent associated with the user request. In a particular embodiment, each domain may have one intent classifier to determine the most probable intent in a given domain. As an example, and not a limitation, the intent classifier may be based on a machine learning model that takes the domain classification / selection result as input and calculates the probability that the input is associated with a specific predefined intent. In step 308b, NLU module 210 can use a meta-intent classifier to process the domain classification / selection results. The meta-intent classifier can determine the category describing the user's intent. In a particular embodiment, intents common to multiple domains can be processed by the meta-intent classifier. By way of example and not limitation, the meta-intent classifier can be based on a machine learning model that takes the domain classification / selection results as input and calculates the probability that the input is associated with a specific predefined meta-intent. In step 309a, NLU module 210 can use a slot tagger to annotate one or more slots associated with the user request. In a particular embodiment, the slot tagger can annotate one or more slots for the n-grams of the user request. In step 309b, NLU module 210 can use a meta-slot tagger to annotate one or more slots for the classification results from the meta-intent classifier. In a particular embodiment, the meta-slot tagger can tag generic slots, such as references to items (e.g., the first item), slot types, slot values, etc. By way of example and not limitation, a user request could include "convert $500 in my account into Japanese yen". An intent classifier takes a user request as input and formulates it as a vector. The intent classifier can then calculate the probability that the user request is associated with different predefined intents based on vector comparisons between the vector representing the user request and vectors representing different predefined intents. Similarly, a slot tagger takes a user request as input and formulates each word as a vector. The intent classifier can then calculate the probability that each word is associated with different predefined slots based on vector comparisons between the vector representing the word and vectors representing different predefined slots.A user's intent can be categorized as "changing money." The slots for user requests can include "500," "dollars," "account," and "Japanese yen." A user's meta intent can be categorized as "financial service." Meta slots can include "finance."

[0105] In a particular embodiment, NLU module 210 may include semantic information aggregator 310. Semantic information aggregator 310 can help NLU module 210 improve domain classification / selection of content objects by providing semantic information. In a particular embodiment, semantic information aggregator 310 may aggregate semantic information in the following manner: Semantic information aggregator 310 may first retrieve information from user context engine 315. In a particular embodiment, user context engine 315 may include an offline aggregator and an online inference service. The offline aggregator may process multiple data associated with the user collected from previous time windows. By way of example and not limitation, the data may include news feed posts / comments collected within a predetermined time range (e.g., from a previous 90-day window), interactions with news feed posts / comments, search history, etc. The processing results may be stored in user context engine 315 as part of the user profile. The online inference service may analyze session data associated with the user received by assistant system 140 at the current time. The analysis results may also be stored in user context engine 315 as part of the user profile. In a particular embodiment, both the offline aggregator and the online inference service may extract personalized features from multiple data. The extracted personalized features can be used by other modules of the assistant system 140 to better understand user input. In a particular embodiment, the semantic information aggregator 310 can then process the information retrieved from the user context engine 315, i.e., the user profile, in the following steps. In step 311, the semantic information aggregator 310 can process the information retrieved from the user context engine 315 based on natural language processing (NLP). In a particular embodiment, the semantic information aggregator 310 can: tokenize the text through text normalization, extract syntactic features from the text, and extract semantic features from the text based on NLP. The semantic information aggregator 310 can also extract features from contextual information accessed from the dialogue history between the user and the assistant system 140. The semantic information aggregator 310 can also perform global word embedding, domain-specific embedding, and / or dynamic embedding based on the contextual information. In step 312, the processing results can be annotated with entities by an entity tokenizer. In step 313, based on the annotations, the semantic information aggregator 310 can generate a dictionary for the retrieved information. In a particular embodiment, the dictionary may include global dictionary features that can be dynamically updated offline. In step 314, the semantic information aggregator 310 may sort the entities tagged by the entity tagger. In a particular embodiment, the semantic information aggregator 310 may communicate with different graphs 320, including one or more social graphs, knowledge graphs, or concept graphs, to extract ontology data related to the retrieved information from the user context engine 315. In a particular embodiment, the semantic information aggregator 310 may aggregate user profiles, sorted entities, and information from graphs 320.The semantic information aggregator 310 can then provide the aggregated information to the NLU module 210 to facilitate domain classification / selection.

[0106] In a particular embodiment, the output of NLU module 210 may be sent to remote inference module 214. Inference module 214 may include coreference component 325, entity resolution component 330, and dialogue manager 335. The output of NLU module 210 may first be received at coreference component 325 to interpret the reference of the content object associated with the user request. In a particular embodiment, coreference component 325 may be used to identify the item referred to in the user request. Coreference component 325 may include reference creation 326 and reference resolution 327. In a particular embodiment, reference creation 326 may create references for entities determined by NLU module 210. Reference resolution 327 may accurately resolve these references. By way of example and not limitation, a user request may include “Find me the nearest grocery store and direct me there.” Coreference component 325 may interpret “there” as “the nearest grocery store.” In a particular embodiment, coreference component 325 may access user context engine 315 and dialogue engine 335 as needed to interpret references with improved accuracy.

[0107] In a particular embodiment, the identified domains, intents, meta-intents, slots and meta-slots, and the resolved referents can be sent to entity resolution component 330 to resolve related entities. Entity resolution component 330 can perform general and domain-specific entity resolution. In a particular embodiment, entity resolution component 330 may include domain entity resolution 331 and general entity resolution 332. Domain entity resolution 331 can resolve entities by classifying slots and meta-slots into different domains. In a particular embodiment, entities can be resolved based on ontology data extracted from graph 320. The ontology data may include structural relationships between different slots / meta-slots and domains. The ontology may also include information on how slots / meta-slots can be grouped, related, and subdivided according to similarity and difference within a higher-level hierarchy including domains. General entity resolution 332 can resolve entities by classifying slots and meta-slots into different general topics. In a particular embodiment, resolution may also be based on ontology data extracted from graph 320. The ontology data may include structural relationships between different slots / meta-slots and general topics. An ontology can also include information about how slots / meta-slots can be grouped, related, and subdivided based on similarity and difference within a higher-level hierarchy that includes topics. As an example, and not a limitation, in response to an input query for the advantages of a specific brand of electric vehicle, generic entity resolution 332 can resolve the brand name of electric vehicle to vehicle, and domain entity resolution 331 can resolve the brand name of electric vehicle to electric vehicle.

[0108] In a particular embodiment, the output of entity resolution component 330 can be sent to dialogue manager 335 to advance the conversation flow with the user. Dialogue manager 335 can be an asynchronous state machine that repeatedly updates its state and selects actions based on the new state. Dialogue manager 335 can include dialogue intent parser 336 and dialogue state tracker 337. In a particular embodiment, dialogue manager 335 can execute the selected action and then call dialogue state tracker 337 again until the selected action requires a user response or there are no more actions to execute. Each selected action may depend on the result of the execution of a previous action. In a particular embodiment, dialogue intent parser 336 can parse user intents associated with the current dialogue session based on the dialogue history between the user and assistant system 140. Dialogue intent parser 336 can map the intents determined by NLU module 210 to different dialogue intents. Dialogue intent parser 336 can also sort dialogue intents based on signals from NLU module 210, entity resolution component 330, and dialogue history between user and assistant system 140. In a particular embodiment, the dialogue state tracker 337 may be a component without side effects, generating n best candidates for dialogue state update operators that suggest dialogue state updates, rather than directly changing the dialogue state. The dialogue state tracker 337 may include an intent resolver containing logic for processing different types of NLU intents based on the dialogue state and generating operators. In a particular embodiment, the logic may be organized by an intent processor, such as an ambiguity resolution intent processor that processes intents when the assistant system 140 requests ambiguity resolution, an acknowledgment intent processor that includes logic for handling acknowledgments, etc. The intent resolver may combine turn intents with the dialogue state to generate a context update for the user session. The slot resolution component may then recursively resolve slots in the update operators using a resolution provider that includes a knowledge graph and domain proxies. In a particular embodiment, the dialogue state tracker 337 may update / sort the dialogue state of the current dialogue session. As an example, and not a limitation, if the dialogue session ends, the dialogue state tracker 337 may update the dialogue state to "complete". As another example, and not a limitation, the dialogue state tracker 337 may sort the dialogue states based on their associated priorities.

[0109] In a particular embodiment, the inference module 214 may communicate with the remote action execution module 226 and the dialogue arbitrator 216, respectively. In a particular embodiment, the dialogue manager 335 of the inference module 214 may communicate with the task completion component 340 of the action execution module 226 regarding dialogue intents and associated content objects. In a particular embodiment, the task completion module 340 may sort different dialogue hypotheses for different dialogue intents. The task completion module 340 may include an action selector 341. In an alternative embodiment, the action selector 341 may be included in the dialogue manager 335. In a particular embodiment, the dialogue manager 335 may also check against a dialogue policy 345 regarding the dialogue state included in the dialogue arbitrator 216. In a particular embodiment, the dialogue policy 345 may include a data structure describing the action execution plan of the agent 350. The dialogue policy 345 may include a general policy 346 and a task policy 347. In a particular embodiment, the general policy 346 may be used for actions not specific to a single task. The general policy 346 may include handling low-confidence intents, internal errors, unacceptable user retry responses, skipping or inserting acknowledgments based on ASR or NLU confidence scores, etc. The general policy 346 may also include logic for sorting dialogue state update candidates from the output of the dialogue state tracker 337 and selecting one to update (e.g., selecting the highest-ranked task intent). In a particular embodiment, the assistant system 140 may have a specific interface for the general policy 346 that allows the incorporation of disparate cross-domain policies / business rules, particularly those found in the dialogue state tracker 337, into the functionality of the action selector 341. The interface of the general policy 346 may also allow the creation of independent sub-policy units that can be bound to specific situations or clients, such as policy functions that can be easily turned on or off based on clients, situations, etc. The interface of the general policy 346 may also allow for a policy hierarchy with fallback, i.e., multiple policy units, where highly specialized policy units handling specific situations are supported by a more general policy 346 applicable in a broader context. In this case, the general policy 346 may alternatively include intent- or task-specific policies. In a particular embodiment, task policy 347 may include logic for an action selector 341 based on the task and the current state. In a particular embodiment, the type of task policy 347 may include one or more of the following types: (1) a handcrafted tree-based dialogue plan; (2) an coded policy that directly implements an interface for generating actions; (3) a slot-filling task specified by a configurator; or (4) a policy based on a machine learning model learned from data. In a particular embodiment, assistant system 140 may guide a new domain with rule-based logic and later refine task policy 347 using a machine learning model. In a particular embodiment, dialogue policy 345 may be a tree-based policy, which is a pre-built dialogue plan.Based on the current dialogue state, dialogue policy 345 can select nodes to execute and generate corresponding actions. As an example and not a limitation, tree-based policies can include topic grouping nodes and dialogue action (leaf) nodes.

[0110] In a particular embodiment, the action selector 341 may employ candidate operators from the dialogue state and consult the dialogue policy 345 to determine which action should be performed. The assistant system 140 may use a hierarchical dialogue policy, where a general policy 346 handles cross-domain business logic, and a task policy 347 handles task / domain specific logic. In a particular embodiment, the general policy 346 may select an operator from the candidate operators to update the dialogue state, and then select a user-facing action via the task policy 347. Once a task is active in the dialogue state, the appropriate task policy 347 can be consulted to select the correct action. In a particular embodiment, both the dialogue state tracker 337 and the action selector 341 may remain unchanged in the dialogue state until the selected action is executed. This allows the assistant system 140 to execute the dialogue state tracker 337 and the action selector 341 to process speculative ASR results and utilize dry runs for n-optimal ranking. In a particular embodiment, the action selector 341 may select a dialogue action by taking the dialogue state update operator as part of its input. The execution of a dialogue action can generate a set of expectations to instruct the dialogue state tracker 337 to process future rounds. In a particular embodiment, these expectations can be used to provide context to the dialogue state tracker 337 when processing user input from the next round. As an example, and not a limitation, a slot request dialogue action may expect to prove the value of the requested slot.

[0111] In a particular embodiment, the dialogue manager 335 may support multi-turn composition parsing of slot references. For composition analysis from NLU 210, the parser may recursively parse nested slots. The dialogue manager 335 may additionally support disambiguation of nested slots. As an example, and not a limitation, a user request could be “Remind me to call Alex.” The parser may need to know which Alex to call before creating an actionable to-do reminder entity. When a particular slot requires further user clarification, the parser may pause parsing and set a parsing state. A general policy 346 may check the parsing state and create a corresponding dialogue action for user clarification. In the dialogue state tracker 337, the dialogue manager may update nested slots based on the user request and the last dialogue action. This capability allows the assistant system 140 to interact with the user, not only collecting missing slot values ​​but also reducing the ambiguity of more complex / vague utterances to complete tasks. In a particular embodiment, the dialogue manager may further support requesting nested intents and missing slots in multi-intent user requests (e.g., “Take this picture and send it to Dad”). In a particular embodiment, the dialogue manager 335 may support machine learning models for a more robust dialogue experience. As an example, and not a limitation, the dialogue state tracker 337 may use a neural network-based model (or any other suitable machine learning model) to model beliefs on task assumptions. As another example, and not a limitation, for the action selector 341, the highest priority policy unit may include whitelist / blacklist overlay, which may have to be designed in; medium priority units may include a machine learning model designed for action selection; and lower priority units may include rule-based backoff when the machine learning model chooses not to handle a situation. In a particular embodiment, a general policy unit based on a machine learning model can help the assistant system 140 reduce redundant disambiguation or confirmation steps, thereby reducing the number of rounds required to execute a user request.

[0112] In a particular embodiment, the action execution module 226 may invoke different agents 350 to perform a task. Agents 350 may be selected from registered content providers to complete the action. The data structure may be constructed by the dialogue manager 335 based on an intent and one or more slots associated with that intent. The dialogue policy 345 may also include multiple targets that are interrelated through logical operators. In a particular embodiment, a target may be the output of a part of the dialogue policy, and it may be constructed by the dialogue manager 335. A target may be represented by an identifier (e.g., a string) with one or more named parameters that parameterize the target. By way of example and not limitation, a target and its associated target parameters may be represented as {confirm artist, parameter: {artist: “Madonna”}}. In a particular embodiment, the dialogue policy may be represented based on a tree structure, where targets are mapped to leaves. In a particular embodiment, the dialogue manager 335 may execute the dialogue policy 345 to determine the next action to be performed. The dialogue policy 345 may include a general policy 346 and a domain-specific policy 347, both of which may guide how to select the next system action based on the dialogue state. In a particular embodiment, the task completion component 340 of the action execution module 226 can communicate with the dialogue policy 345 included in the dialogue arbitrator 216 to obtain guidance for the next system action. In a particular embodiment, the action selection component 341 can therefore select an action based on the dialogue intent, the associated content object, and the guidance from the dialogue policy 345.

[0113] In a particular embodiment, the output of the action execution module 226 can be sent to the remote response execution module 232. Specifically, the output of the task completion component 340 of the action execution module 226 can be sent to the CU writer 355 of the response execution module 226. In an alternative embodiment, the selected action may require the participation of one or more agents 350. Therefore, the task completion module 340 can notify the agents 350 of the selected action. Simultaneously, the dialogue manager 335 can receive instructions to update the dialogue state. By way of example and not limitation, the update may include waiting for a response from the agents 350. In a particular embodiment, the CU writer 355 can generate communication content for the user using the natural language generation (NLG) module 356 based on the output of the task completion module 340. In a particular embodiment, the NLG module 356 can use different language models and / or language templates to generate natural language output. The generation of natural language output can be application-specific. The generation of natural language output can also be personalized for each user. The CU writer 355 can also use the UI payload generator 357 to determine the modality of the generated communication content. Since the generated communication content can be considered a response to a user request, the CU writer 355 can additionally use a response sorter 358 to sort the generated communication content. As an example, and not a limitation, the sorting can indicate the priority of the responses.

[0114] In a particular embodiment, the response execution module 232 may perform different tasks based on the output of the CU writer 355. These tasks may include writing (i.e., storing / updating) the dialogue state 361 retrieved from the data storage 212 and generating a response 362. In a particular embodiment, the output of the CU writer 355 may include one or more of a natural language string, speech, parameterized actions, or a rendered image or video that can be displayed in a VR headset or AR smart glasses. As a result, the response execution module 232 may determine what task to perform based on the output of the CU writer 355. In a particular embodiment, the generated response and communication content may be sent by the response execution module 232 to the local rendering output module 242. In an alternative embodiment, if the modality of the determined communication content is audio, the output of the CU writer 355 may be additionally sent to a remote TTS module 238. The speech generated by the TTS module 238 and the response generated by the response execution module 232 may then be sent to the rendering output module 242.

[0115] Figure 4An exemplary flowchart illustrating the processing of user input by assistant system 140 is shown. As an example and not a limitation, user input may be based on an audio signal. In a particular embodiment, microphone array 402 of client system 130 may receive audio signals (e.g., voice). The audio signal may be transmitted to processing loop 404 in the format of audio frames. In a particular embodiment, processing loop 404 may send audio frames for Voice Activity Detection (VAD) 406 and Voice Wake-up Detection (WoV) 408. The detection results may be returned to processing loop 404. If WoV detection 408 indicates that the user wants to invoke assistant system 140, the audio frame, along with the VAD 406 result, may be sent to encoding unit 410 to generate encoded audio data. After encoding, for privacy and security purposes, the encoded audio data may be sent to encryption unit 412, followed by linking unit 414 and decryption unit 416. After decryption, the audio data may be sent to microphone driver 418, which may further transmit the audio data to audio service module 420. In an alternative embodiment, user input can be received at a wireless device (e.g., a Bluetooth device) paired with client system 130. Accordingly, audio data can be sent from wireless device driver 422 (e.g., a Bluetooth driver) to audio service module 420. In a particular embodiment, audio service module 420 can determine that the user input can be implemented by an application executing on client system 130. Therefore, audio service module 420 can send the user input to real-time communication (RTC) module 424. RTC module 424 can transmit audio packets to a video or audio communication system (e.g., VoIP or video call). RTC module 424 can invoke a related application (App) 426 to perform tasks related to the user input.

[0116] In a particular embodiment, the audio service module 420 can determine that the user is requesting assistance that requires a response from the assistant system 140. Therefore, the audio service module 420 can notify the client assistant service module 426. In a particular embodiment, the client assistant service module 426 can communicate with the assistant orchestrator 206. The assistant orchestrator 206 can determine whether to use a client process or a server process to respond to the user input. In a particular embodiment, the assistant orchestrator 206 can determine to use a client process and notify the client assistant service module 426 of this decision. As a result, the client assistant service module 426 can invoke the relevant module to respond to the user input.

[0117] In a particular embodiment, the client assistant service module 426 may use a local ASR module 216 to analyze user input. The ASR module 216 may include a glyph-to-phoneme (G2P) model, a pronunciation learning model, a personalized language model (PLM), an endpoint model, and a personalized acoustic model. In a particular embodiment, the client assistant service module 426 may further use a local NLU module 218 to understand user input. The NLU module 218 may include a named entity resolution (NER) component and a context-based session NLU component. In a particular embodiment, the client assistant service module 426 may use an intent mediator 428 to analyze the user's intent. To accurately understand the user's intent, the intent mediator 428 may access an entity store 430 that includes entities associated with the user and the world. In an alternative embodiment, user input may be submitted via an application 432 executing on the client system 130. In this case, the input manager 434 may receive the user input and analyze it by the application environment (AppEnv) module 436. The analysis results can be sent to application 432, which can further send the analysis results to ASR module 216 and NLU module 218. In an alternative embodiment, user input can be submitted directly to client assistant service module 426 via assistant application 438 running on client system 130. Client assistant service module 426 can then perform a similar process based on the modules described above, namely ASR module 216, NLU module 218, and intent mediator 428.

[0118] In a particular embodiment, the assistant orchestrator 206 may determine the server-side process for the user. Therefore, the assistant orchestrator 206 may send user input to one or more computing systems of different modules of the managed assistant system 140. In a particular embodiment, the server assistant service module 301 may receive user input from the assistant orchestrator 206. The server assistant service module 301 may instruct a remote ASR module 208 to analyze the audio data of the user input. The ASR module 208 may include a glyph-to-phoneme (G2P) model, a pronunciation learning model, a personalized language model (PLM), an endpoint model, and a personalized acoustic model. In a particular embodiment, the server assistant service module 301 may further instruct a remote NLU module 210 to understand the user input. In a particular embodiment, the server assistant service module 301 may invoke a remote inference model 214 to process the output from the ASR module 208 and the NLU module 210. In a particular embodiment, the inference model 214 may perform entity resolution and dialogue optimization. In a particular embodiment, the output of the inference model 314 may be sent to an agent 350 to perform one or more related tasks.

[0119] In a particular embodiment, agent 350 can access ontology module 440 to accurately understand the results from entity parsing and dialogue optimization, thereby enabling it to accurately perform relevant tasks. Ontology module 440 can provide ontology data associated with multiple predefined domains, intents, and slots. The ontology data may also include structural relationships between different slots and domains. The ontology data may also include information on how slots can be grouped, related, and subdivided based on similarity and difference within a higher-level hierarchy including domains. The ontology data may also include information on how slots can be grouped, related, and subdivided based on similarity and difference within a higher-level hierarchy including topics. Once a task is executed, agent 350 can return the execution result along with a task completion indication to inference module 214.

[0120] The embodiments disclosed herein may include or be implemented in conjunction with artificial reality systems. Artificial reality is a form of reality that has been adapted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., real-world photographs). Artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of these may be presented in a single channel or in multiple channels (e.g., generating stereoscopic video with a three-dimensional effect for the viewer). Additionally, in some embodiments, artificial reality may be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or be used in artificial reality (e.g., performing activities in artificial reality). Artificial reality systems that deliver artificial reality content can be implemented on a variety of platforms, including head-mounted displays (HMDs) connected to a host computer system, stand-alone HMDs, mobile devices or computing systems, or any other hardware platform capable of delivering artificial reality content to one or more viewers.

[0121] Semantic representation using structural ontology In a particular embodiment, the assistant system 140 may use a structured semantic representation framework, or “next-generation ontology” (NGO), to construct instructions for various modules of the assistant system 140, including instructions for interfacing with first-party and third-party agents. An NGO may include an ontology with several different types of semantic units, such as actions, objects, and attributes. An NGO may support a tagging system derived from the ontology, allowing the user to request subgraphs represented as ontologies. An NGO may allow the assistant system 140 to unify the semantic representations used by the natural language understanding process (i.e., input processing) and the natural language generation process (i.e., output processing). All modules in the stack of the assistant system 140 may use the same type of semantic representation based on the NGO, rather than their own semantics, thereby reducing mappings between them and increasing consistency. Objects may be hosted throughout the stack of the assistant system 140, where the assistant system 140 may use a dialogue manager to change actions between input and output based on mapping layers. Furthermore, because all parts of the ontology are defined, the assistant system 140 may determine whether any semantic parsing requested by the user is valid before attempting to perform the corresponding task. Although this disclosure describes a particular ontology in a specific manner, this disclosure contemplates any suitable ontology in any suitable manner.

[0122] In a particular embodiment, assistant system 140 may receive user input, including natural language utterances, from client system 130 associated with the user via assistant xbot. Assistant system 140 may then determine a semantic representation of the user input based on a structural ontology. In a particular embodiment, the structural ontology may define a marked syntax for parsing natural language utterances into semantic units comprising multiple actions, objects, and attributes. The semantic representation may embed at least one object into at least one action and declare at least one attribute of the embedded object for which the action will be performed. In a particular embodiment, assistant system 140 may send a request to an agent to perform a task corresponding to the user input. This request may be based on the semantic representation. In a particular embodiment, assistant system 140 may receive the results of the performed task from the agent, wherein these results are mapped to a structure determined by the structural ontology. Assistant system 140 may also send instructions from assistant xbot to client system 130 for presenting a response to the user input, wherein the response is based on the results of the performed task.

[0123] In a particular embodiment, the assistant system 140 may require a semantic representation to encode the task the user wants to complete. The semantic representation can be a formal language describing the semantics of natural language. Most traditional semantic representations used in task-oriented semantic parsing likely revolve around a flat intent / slot paradigm. The flat intent / slot (IN / SL) paradigm extracts a natural language utterance into a single intent with a set of slots. Related intents can be collected into a domain. The term "flat" can refer to a parsing where slot values ​​are necessarily terminal values. The representational power of flat intent / slot representations may be limited.

[0124] Tags (specific intentions and slots) in the IN / SL paradigm may be unfounded, meaning they may be arbitrary and lack formal relations or hierarchy because there is no ontology providing structure for the tag and the semantic concept it represents. Such unfounded tag sets may lack the formalization to position them within a coherent worldview expressed by an ontology. Furthermore, intentions / slots may not provide a rich system of types. Therefore, maintaining a tag set with internal consistency and semantic expression can be difficult.

[0125] To address the aforementioned limitations, the embodiments disclosed herein utilize NGOs to generate semantic representations. NGOs can include graph ontologies with several different types of semantic units, such as strongly typed actions and objects. Constraints and definitions can be defined on the semantic units. These constraints may aid in the prediction and validation of NLU outputs, fall back when the agent does not support specific features or the utterance is ambiguous with multiple possible interpretations (e.g., "Halo" could be a game or music title), and have the same entity types for knowledge graph and query modeling, which facilitates seamless entity recognition. Semantic units can be nodes in a graph, and their connections can be edges in the graph. NGOs can combine the semantic space of natural language understanding and natural language generation labeling systems with concepts from knowledge graphs, concept graphs, and multimodal understanding into a single, shared semantic representation. From this graph ontology, the assistant system 140 can derive domain- and modality-agnostic labeled representations. The unified graph ontology of NGOs can promote cross-domain reusability. Furthermore, the ontology can be hierarchically organized, which inherently supports fallback. Node-edge relationships can be leveraged to capture complex queries, and the shared semantic space allows for seamless integration of multimodal inputs. Furthermore, the reusability of actions, objects, and attributes may slow down the trajectory of tag set addition. In specific embodiments, NGOs can approach semantic representation by decomposing semantic representation into orthogonal elements, semantic units, tokenized representations (e.g., NLU outputs), and a type system (semantic constraints / definitions). An NGO may have three core semantic units: actions, objects, and attributes, and three derived semantic units: methods, enumerations, and lists. Each semantic unit can have a type. NGOs can support tokenized systems derived from ontology, allowing users to request subgraphs represented as ontologies. The markup language syntax can combine actions, objects, attributes, methods, and enumerations in a predictable manner.

[0126] Figure 5 This section shows an example comparison between the IN / SL and NGO representations of a utterance. The utterance could be something like, "remind meto call john." Figure 5As shown, the planar IN / SL representation 505 can be [IN:CREATE_REMINDER Remind [SL:PERSON REMINDED me] to [SL:TODO call John]] 510. The combined IN / SL representation 515 can be [IN:CREATE_REMINDER Remind [SL:PERSON REMINDED me] to [SL:TODO[IN:CREATE_CALL] [SL:CONTACT John]]]] 520. In contrast, the NGO representation 525 can be (create[reminers hasPersonReminded=[User] hasToDo=[todo hasToDoAction=(create [callhasParticipant=[person name=[John]]])]]) 530.

[0127] Figure 6 This diagram illustrates an example comparison between media domains represented by the IN / SL paradigm and those represented by the NGO structure. The grouping on the left (IN / SL 600) shows a simplified view of how various media domains are represented in a traditional asynchronously stood-up intent / slot-based assistant. As shown, intents IN:PLAY_MEDIA 602 and IN:GET_MEDIA_TITLE 604 may both require slot SL:MEDIA_TITLE 606; IN:PLAY_GAME 608 and IN:GET_GAME_TITLE 610 may require SL:GAME_TITLE 612; IN:PLAY_VIDEO 614 and IN:GET_VIDEO_TITLE 616 may require SL:VIDEO_TITLE 618; IN:PLAY_MUSIC 620 and IN:GET_SONG_TITLE 622 may require SL:SONG_TITLE 624. The grouping on the right (NGO 630) shows how NGOs can represent the same domain in the proposed structure. As shown, game 632, music 634, and video 636 can all be covered by media 638, which can be referred to by title 640. The corresponding NGO representation for game 632, music 634, and video 636 can be the same, which can be "action play (obj: Media)" 642.

[0128] In a particular embodiment, an action may represent an action that the assistant system 140 can take. A structural ontology may define each of a plurality of actions that operates on one or more of a plurality of objects. By way of example, and not limitation, an action may include one or more of “create,” “get,” “update,” “play,” or “delete.” Actions may be strongly typed with static lifetimes and may operate on strongly typed objects. All actions in an NGO may begin and end with parentheses. By way of example, and not limitation, for the parameterless action “play,” the NGO resolution may be (play). In a particular embodiment, at least one action may have one or more parameters. Each parameter may specify a constraint that an object must satisfy, such that at least one action can act on that object. At least one parameter may specify a constraint that at least one object satisfies. By way of example, and not limitation, an action may have positional parameters, such as (action [obj1]), for example, “create” (action) “call” (object). An action may also have named parameters, such as attributes, for example, (action [obj1, attribute =obj2]), for example, “create” with “call” for participant “John.” Actions can also have a type signature with a required return value. A Void return value for an action may not be allowed. Actions may exclude information about a specific entity type. By way of example, and not limitation, (play_music) might be an incorrect action. Instead, an object can be used to describe the target entity type. In other words, (play[music]) might be a correct form. This may allow actions, like all entities, to be reused across domains and scenarios. By way of example, and not limitation, both "give me a song" and "give me their availability" could correspond to the "get" action. Although this disclosure describes a particular action in a specific way, this disclosure contemplates any suitable action in any suitable way.

[0129] In certain implementations, objects can record types containing strongly typed properties and methods. A structural ontology can define each of multiple objects as including one or more properties. Objects can be manipulated by actions or linked by properties. All objects can begin and end with square brackets, with the object name directly following the left square bracket. Objects can have subtyped or supertyped objects. They can contain other typed objects (properties) and methods. Objects may not be used to enumerate specific characteristics; for example, an action (get [track_title]) might be incorrect. Conversely, properties can be used for specific characteristics; for example, (get [track].title) might be the correct form of an action. This can facilitate inheritance between objects, provide structure, and increase reusability. Consider the media and subdomains music, games, and video as examples. Each domain might introduce repetitive intents and slots using the IN / SL paradigm, prone to inconsistencies and errors. In contrast, NGOs can have a technical advantage in reducing repetitive work because they can allow the automatic generation of semantic spaces through reusability, eliminate domain differences, and support type hierarchies, which can create reusable markup elements and reduce repetitive work. Although this disclosure describes a particular object in a particular manner, this disclosure considers any suitable object in any suitable manner.

[0130] In certain embodiments, attributes can be used to enumerate specific characteristics of actions and objects, and specify the nature of the relationship between two entities. A structural ontology can define a set of attributes, each declaring a constraint on an action or object. Typed objects can be bound to other objects via attributes, which can be a strict and unique relationship. As an example, and not a constraint, a name can be an attribute of a "person" object of type string. As another example, but not a constraint, "hasParticipant" can be an attribute of a "call" object of type participant. For example, a user request to "play a Beyoncé song" can be represented as (play[trackhasPerformer =[person name =[Beyoncé]]]), where "play" is the action, "track" and "person" are objects, and "hasperformer" and "name" are attributes. In certain embodiments, attributes can be used to express directed triples linking one entity to another through unidirectional semantic roles, i.e., (Entity_1, Attribute, Entity_2) such as (Bill, fatherOf, Brian). Syntactically, attributes may appear in two positions during parsing. Attributes can appear as named parameters to specify characteristics of certain references, such as "play the song hello goodbye" = (play [track title=[hello goodbye]]). Attributes can also be displayed as dotted attributes to specify the properties of the object to which the action is targeted, such as "what's the title of this song" = (get [track].title). Attributes may be syntactically and functionally similar to class member variables. Although this disclosure describes specific attributes in a particular manner, this disclosure considers any suitable attribute in any appropriate manner.

[0131] In a particular embodiment, the structural ontology NGO can define semantic units that also include multiple methods and enumerations. Each of the multiple methods can include actions whose lifetimes are associated with an object. In a particular embodiment, a method can own an object. A method may also have the same characteristics as an action, i.e., a namespaced method name, positional parameters, and named parameters. Methods can be introduced by the "." operator and can be suffixed with parentheses. They may act on attribute values ​​and appear in the following forms: (fb / <action_name> [fb / <object_name> ]. <attributename> . <methodname>([fb / <object_name> ]))) (fb / <action_name> [fb / <object_name> ]. <attributename> . <methodname>([ ]))) Both object types and data types can be acted upon by methods. As an example, and not a limitation, there can be three methods. The "add" method updates a property by adding it to the property. The "remove" method updates a property by removing it from the property. The "set" method sets the property value. Methods may be an additional feature of an NGO that trades additional label type complexity for reduced parsing complexity, for example, by reducing the nesting level by one level. As an example, and not a limitation, one method could be to add a receiver to the call as (update [call].hasRecipient.add([person name=[John]]). As another example, and not a limitation, one method could be to set the app as default (update [music_app name=[spotify]].setDefault()). Methods may be inheritable. Although this disclosure describes specific methods in a particular manner, this disclosure contemplates any suitable method in any suitable manner.

[0132] In certain embodiments, each of the plurality of enumerations may include an object representing one of a plurality of objects. An enumeration may be a union type of string tokens. An enumeration may be a specific tag type that utilizes NLU modules 210 / 218 to reduce the need for downstream interpretation of the original string. An enumeration may be a parameter of an attribute expected anywhere in a finite set of inputs. In certain embodiments, enumerations may be useful when there is variability in the form of substrings with equivalent semantics. If desired, the source string may be preserved to provide the ability to check sub-spans in a dialogue. In certain embodiments, semantic units may additionally include lists, which may be objects representing lists of the same type. In certain embodiments, each semantic representation may have a namespace. By way of example and not limitation, a namespace may be "fb / ". Although this disclosure describes particular enumerations and lists in a particular manner, this disclosure contemplates any suitable enumerations and lists in any suitable manner.

[0133] In a particular embodiment, the assistant system 140 may construct the NGO as follows. Construction may begin with a subgraph. In a particular embodiment, the structural ontology may further define a graph structure including one or more core subgraphs and one or more general subgraphs. One or more core subgraphs are inaccessible by third-party proxies, and one or more general subgraphs are accessible by third-party proxies. Actions and objects may be modeled as nodes on the graph, and attributes may be represented as edges between nodes. In a particular embodiment, the basic structure may be: Node (object) --> hasAttribute (attribute) --> Node (object). As an example, and not a limitation, the basic structure may be Track --> hasGenre --> MusicGenre, rendered as [track hasGenre [genre = [some Music Genre]]]. As another example, but not a limitation, the basic structure may be Movie --> hasGenre --> Narrative Genre, rendered as [movie hasGenre [genre = [some Narrative Genre]]]. In a particular embodiment, all semantic units in the NGO may be reusable. The construction of NGOs can primarily involve creating new semantic units (e.g., new actions, new objects, new attributes) and establishing new connections between existing nodes to express new meanings and behaviors. Defining core subgraphs and general subgraphs may be an effective solution to address the technical challenge of providing third-party users with the flexibility to design their own semantic units while maintaining the integrity of the structural ontology, because the core subgraph and general subgraph are functionally separate, and the basic structure of the ontology is maintained by the core subgraph, which is only visible to third-party users. Although this disclosure describes a specific ontology constructed by a specific system in a specific manner, this disclosure envisions any suitable ontology constructed by any suitable system in any suitable manner.

[0134] Figure 7 An example structure of NGO subgraph 700 is shown. In a particular embodiment, the structure ontology can define multiple objects hierarchically organized into multiple supertypes and subtypes. Each subtype can inherit one or more properties from its associated supertype. Figure 7 As shown, nodes can be placed within a hierarchy, allowing for implicit fallback. This is for illustrative purposes only and not a limitation. Figure 7 It is shown that "movie" 705 and "track" 710 can fall back to "media item" 715. The same edge (e.g., hasGenre 720) can be used to link "track" 710 to "music genre" 725 and "movie" 705 to "narrative genre" 730. "music genre" 725 and "narrative genre" 730 can fall back to "media genre" 735. In certain embodiments, valid attributes can be embedded in the object, thus preventing invalid links despite reusability. NGOs may have the technical advantage of hierarchically aware fallback because they can encode automatic and systematic fallback for ambiguously or incorrectly labeled discourse and avoid hard-coded rules for fallback that may increase with domain size. Although this disclosure describes a particular structure of a particular ontology in a particular manner, this disclosure contemplates any suitable structure of any suitable ontology in any suitable manner.

[0135] Figure 8A An example subdomain selection is shown. This is an example, not a limitation, of how subdomain selection can be used to support performers playing music. Figure 8A As shown, a node named "root" 800 can exist, which can be connected to "action" 810 and "object" 820. A subgraph can be selected that includes the action "play" 812, the object "music" 822 and its subtypes, and the object "person" 824. "person" 824 can be associated with "name" 825. The action "get" 814 can be excluded from the subgraph. Both "play" 812 and "get" 814 can be associated with the "takesArgument" 830 property, which can further link to "music" 822. Subtypes of "music" 822 can include "album" 826 and "track" 827. Furthermore, there is no way to link "music" 812 to "person" 822, and all necessary subtypes of "music" 812, such as "playlist," may be missing. Figure 8A As shown.

[0136] Figure 8B This illustrates an exemplary continuation of subdomain selection. "playlist" 828 can be added as a subtype of "music" 822, and then "music" 822 can be connected to "person" 824 via the "hasPerformer" 832 property. The "hasPerformer" 832 property can be inherited by all subtypes of "music" 822. Therefore, Figure 8B The subgraphs constructed within can support performers playing music, for example, "play Beyonce", "play a song by Beyonce", "play my Beyonce playlist", and "play a Beyonce album". Although this disclosure describes specific subdomain selection in a particular manner, this disclosure contemplates any suitable subdomain selection in any suitable manner.

[0137] Figure 9 This section describes an example of inheritance in an NGO. In certain embodiments, NGO objects may allow single inheritance and trait-based implementations. Object inheritance can be used as the primary medium through which tag reuse occurs. In certain embodiments, NGOs may use the same object inheritance semantics as most traditional object-oriented programming languages. That is, parent properties and methods may be passed to child objects, and redefining existing properties and methods may not be allowed. Furthermore, NGOs may enforce strict single inheritance to further improve scalability. In certain embodiments, objects may optionally be initiated with named parameters. As an example, and not a limitation, an NGO-initiated person object named John might look like `[person name=[John]]`. Figure 9 As shown, there may be several objects, including "participant" 902, "group" 904, "organization" 906, "person" 908, "company" 910, and "call" 912. The "call" 912 object can have a single named parameter "has participant" 914, which has a "type" attribute, for example (fb / call hasParticipant=[fb / person name=[Morgan]]). The "person" 908 object type can inherit the "name" 916 attribute from "participant" 902, and the type contract required by the participant can be achieved through an inheritance chain from "person" 908 to "participant" 902. "group" 904 and "organization" 906 can similarly inherit the "name" 916 attribute from "participant" 902. "group" 904 and "person" 908 can have a typed attribute "availability" 918. "person" 908 can also have a typed attribute "employer" 920, which can be associated with "company" 910. In this example ontology, the assistant system 140 can reject the following resolution: (fb / callhasParticipant=[fb / organization]). Except that the embodiments disclosed herein may only allow single inheritance, inheritance here can be the same as in almost all object-oriented programming languages. All properties in an object can be initialized (e.g., "name" 916). In turn, all properties can be looked up from an object. As an example and not a limitation, "getLiz's availability" can be defined as (fb / get [fb / person name=[Liz]].Availability). Although this disclosure describes a particular inheritance in a specific manner, this disclosure contemplates any suitable inheritance in any suitable manner.

[0138] In certain embodiments, single inheritance can limit the formation of union types. To alleviate this pressure, NGOs can leverage features. Features can be named abstract types that define only method signatures. Objects can implement any number of features without conflict. As an example, and not a limitation, an NGO can have a "creatable" feature that can be implemented by both reminders and timers. Thus, they can both be used as parameters in a "create" action whose type signature uses the "creatable" feature. NGOs can use features as a mechanism to formally express to a type checker which entity types are acceptable as parameters to other entities. Where multiple types are legal, features can allow for static checking of semantic resolution while avoiding the pitfalls of multiple inheritance. Features can be viewed as a way of grouping entities that do not belong to the same inheritance chain for type checking. Although this disclosure describes specific features in a particular manner, this disclosure considers any suitable feature in any suitable manner.

[0139] In certain embodiments, user requests may involve taking an action on an object, which can be resolved by embedding the object within the action. These take the following general form, always beginning with parentheses, e.g., (fb / <action_name> [fb / <object_name> Actions can only be embedded in objects that satisfy the parameter constraints. As an example, and not a restriction, the first parameter of the "create" action can only operate on "creatable" objects (e.g., calls, video calls, alerts, etc.). Examples could include (fb / create [fb / reminder]), (fb / play [fb / music]), and (fb / end [fb / call]).

[0140] In a particular embodiment, a user request may involve taking an action on an object's attribute. These requests can be resolved by embedding an object within an action and declaring the attributes of the embedded object to be acted on. To retrieve a specific attribute from an object, the NGO might use standard dot notation from a programming language. As an example, and not a limitation, if you want to see if John is available, in the NGO, this might map to: (person name=[John]].availability). In a particular embodiment, the semantic representation could then take the form: (fb / <action_name> [fb / <object_name> ]. <attributename>Examples could include (fb / create [fb / reminder].startTime), (fb / get [fb / music].hasPerformer), and (fb / add [fb / call].hasRecipient). In certain embodiments, the tagging system may allow attribute linking, i.e., actions can be taken on the properties of an object. This could be represented as (fb / <action_name> [fb / <object_name> ]. <attributename> . <attributename>Note that the return type of an action can also be defined statically. It's also possible to define generics statically on a function, as shown in the following example: > Signature ~ fb / create <t>(T t) -> T; > Invocation ~ (fb / create [fb / call hasParticipant=[fb / person name=[Morgan]]]) Invocation Type ~ fb / call In a particular implementation, all attributes can be typed. There are three possible type families: 1) an attribute can point from one object to another; 2) an attribute can point from an object to a primitive data type, such as a string or date / time; and 3) an attribute can have a finite set of enumerated possible values. Which family a given attribute belongs to may further affect its form in the context of the object specification (also known as entity constraints) described below. If an attribute points to another node, the type of that node may be explicit. The attribute name may be followed by an equals sign, followed by a type declaration, i.e., (fb / <action_name> [fb / <object_name> <attributename>=[fb / <object>Please note that actions can be nested like this: (fb / <action_name> [fb / <object_name> <attributename>=(fb / <action>If the attribute points to a core data attribute / type (i.e., a string), the type may not be explicit. An equals sign can follow the attribute name, but not a type declaration, i.e., (fb / <action_name> [fb / <object_name> <attributename>=[ ]]). If a property is assigned by an enumeration, then a specific value in the enumeration can be declared in the type declaration: (fb / <action_name> [fb / <object_name> <attributename>=[::enum_value]]). This may not mean that the type does not exist or is not dynamic, but is purely syntactic sugar in terms of representation. Although this disclosure describes a particular property type in a specific way, this disclosure considers any suitable property type in any suitable way.

[0141] In certain implementations, list objects can be represented using angle brackets <>. Lists can be useful when representing semantics on collections. As an example, and not a limitation, the utterance "call John and Erika" can be mapped to (create [call hasParticipant =< [person name = [John]],[person name = [Erika]] >]). Attributes can have more than one value. This can be captured using list notation. It can be assumed that all attributes that do not point to an enumeration value are lists, typically with only one member. As an example, and not a limitation, the utterance "start a video call with granddaughter and uncles" can be mapped to (fb / create [fb / video_call hasParticipant=<[fb / persongranddaughterOf=[fb / user]], [fb / person uncleOf=[fb / user]]>]). Note that semantics across lists can still be validated, so placing a "company" object in this list may result in semantically incorrect parsing. Lists can be declared using the following syntax: (fb / <action_name> [fb / <object_name> <attributename>=<[fb / <object>...] , [fb / <object>... ]>]) (fb / <action_name> [fb / <object_name> <attributename>=<[ ... ] , [ ...]>]) Similar to non-list properties, the property name can be followed by an equal sign "=". The list can then be introduced by an open bracket "<" and closed by a closing bracket ">". Within the list, each member can be typed as an object or a primitive data type, and members are separated by commas ",". Although this disclosure describes a particular list in a specific manner, this disclosure contemplates any suitable list in any suitable manner.

[0142] In certain implementations, methods can resemble actions but can be bound to specific objects. As an example, and not a limitation, in NGOs, all properties can have setters, which are methods with the same name as the property. For the utterance "make me unavailable", the NGO resolution could be `[user].availability([::unavailable(unavailable)]`. As another example, and not a limitation, a canonical example could be list-type add / remove methods. For example, the utterance "can you take my grandmother Lisa off the call?" could be mapped to `[fb / call].hasParticipant.remove([fb / person grandmotherOf=[fb / user]name=[Lisa]])`. Here, the "remove" method can be called with one argument, which is the person to be removed from the call, acting on the `hasParticipant` object.

[0143] In a particular embodiment, the NGO can allow any level of nesting, provided that the type signatures of all individual components are satisfied. Figure 10 An example semantic parsing of the combination is shown. The combinatorial decomposition of the syntactic representation of the NGO might be for query purposes: please call Paisley and Myles (corresponding to) Figure 10 (the left part) and Make me offline (corresponding to) Figure 10 (The right side). For example... Figure 10 As shown, for the query Please call Paisley and Myles The syntax can be (create [call hasParticipant =<[person name=[Paisley]], [person name=[Myles]]>]). "create" can be action 1002, and the "[]" after "create" can be the created "call" object 1004. The "call" object 1004 can include the "hasParticipant" property 1006. The "hasParticipant" property 1006 can be defined by property 1008 of "<[person name=[Paisley]], [person name=[Myles]]>". Property 1008 can include object 1010 of "[person name=[Paisley]]" and object 1012 of "[person name=[Myles]]". For queries... Make me offline The syntax can be (set [user].availability ( [::unavailable(offline)])). "set" can be an action 1014 acting on the "user" object 1016. Action 1014 can further include the method ".availability" 1018, whose embeddings enumerate multiple unavailable times 1020. Although this disclosure describes a specific semantic resolution in a particular manner, this disclosure contemplates any suitable semantic resolution in any suitable manner.

[0144] In a particular embodiment, the enumeration may represent one of n variants. As an example, and not a limitation, only a fixed number of phone types (cellular / mobile / home, etc.) may be supported. The utterance "call my mom on the cell" can be mapped to (fb / create [fb / call hasParticipant=[fb / person motherOf=[fb / user]] numberCalled=[fb / phone_number phoneType=[::cell_phone("cell")]]]). Here, "phoneType" can be tagged as the "cell_phone" type invoked by the "cell" symbol in the utterance. In a particular embodiment, the availability attribute can be typed as an availability enumeration with two variants: available and unavailable. Whenever the assistant system 140 instantiates an enumeration object in the NGO, the assistant system 140 can also tag the original string used to parse the variants. The purpose of maintaining the original string may be to allow further parsing or backtracking of variants in downstream modules of the assistant system 140.

[0145] In certain embodiments, an NGO may have certain attributes, including object specifications through attribute and type conversion (e.g., in a composition scenario). Besides return values, attributes can also be used to restrict entities. In certain embodiments, an NGO may restrict the kinds of objects on which actions can be performed or the kinds of actions that can be performed on an object. Restricting attributes may appear within parentheses or curly braces used to introduce the entity to be restricted. Attributes can also restrict objects. In this case, they may appear within square brackets of the object, for example, (fb / <action_name> [fb / <object_name> <attributename> = <attributeinitialization>In this case, actions can operate on objects, and these objects may be constrained by certain properties. Valid properties can be declared in the ontology. In the above case, properties can be properties of embedded objects. Properties may also constrain actions. In this case, they may be nested within parentheses that introduce the action. Named properties may always follow positional properties, for example (fb / <action_name> [fb / <object_name> ] <attributename> = <attributeinitialization>Similarly, valid attributes can be declared within the ontology. In the above cases, the attribute can be an attribute of an embedded action. Finally, attributes may be restricted in type. In these cases, attribute restrictions can be introduced as follows: (fb / <action_name> [fb / <object_name> ]. <attributename>([fb / attribute_type <attributename>Similarly, legal attributes can be declared in the ontology. In the above cases, the restrictive attribute can be an attribute of the attribute type. Note that the above examples may be incomplete. They may only indicate where attributes might be placed. To fully implement the form of attribute tags, typed attributes may also be required. Although this disclosure describes restricting a particular entity by a particular attribute in a particular way, this disclosure contemplates restricting any suitable entity by any suitable attribute in any suitable way.

[0146] In certain embodiments, type conversion can represent complex instances of attribute constraints. In some cases, the value of an attribute can be determined by recognizing the attribute value of another object. Consider the utterance "get directions to the game." This can be understood as a request to return directions to a location that is very similar to the location of a sporting event venue. This behavior can be called type conversion. Type conversion can be achieved by nesting an object within an attribute and declaring which attribute values ​​of that object are embedded in the attribute, for example, (fb / action_name [fb / object_name)). <attributename>=[fb / object_name [fb / object_name]. <attributename>As an example, and not a limitation, a potential annotation for "get directions to the game" could be (fb / get [fb / directions destination=[fb / location location_name=(fb / get[fb / event name="the game"]).venueLocation]). In a particular embodiment, the type of the embedded attribute and the type of the embedded attribute can be the same to make it valid. Type conversion can also be combined with attribute linking to produce a tag of the form: (fb / action_name [fb / object_name]). <attributename>=[fb / object_name[fb / object_name]. <attributename> . <attributename>]]]). Type conversion can be an effective solution to the technical challenge of representing complex instances of property restrictions, as type conversion can nest objects within properties and declare which property values of that object are embedded properties, thereby representing complex instances in a cleaner structure with improved explainability. Although the present disclosure describes particular type conversions in particular ways, the present disclosure contemplates any suitable type conversion in any suitable way.

[0147] In particular embodiments, NGO-based semantic representations can be generic across the stack of assistant system 140. The semantic representations can be executed by each module associated with an assistant xbot configured to process user input. In particular embodiments, the values associated with each of the at least one action, the at least one object, and the at least one property can be determined by each module executing the semantic representation. Because each object, action, or property has a type, NGOs can have a formal and consistent agreement for semantic resolution across the entire assistant stack, which can result in technical advantages of consistent ontology and type checking. Errors that previously required deep integration and unit testing in the IN / SL paradigm can be replaced by continuous complexity static analysis. Resolution is automatically validated during annotation and live traffic, and can not require writing validation code for each particular use case. In particular embodiments, NGOs can prevent annotators from creating semantically incorrect resolutions and provide consistent semantics to models. No domain distinction can allow for zero-cost cross-domain scenarios. Execution can become consistent because types and type relationships are well-defined.

[0148] In particular embodiments, based on the NGO, the NLU module 210 / 218 can generate its output and pass it to the entity resolution module 330. As an example and not by way of limitation, the NLU output for "call my mom" can be (create [call hasParticipant=[person motherOf=[user]]]). As another example and not by way of limitation, the NLU output for "call Jerry's office" can be (create [call hasParticipant=[person name=[Jerry]] numberCalled=[phone_number contactType=[::work (office)]]]). As another example and not by way of limitation, the NLU output for "will it rain tomorrow?" can be (get [fb / forecast duration=[date_time tomorrow]].precipType). As another example and not by way of limitation, the NLU output for "ten day forecast in San Diego" can be (get [fb / forecast duration=[date_time ten day ] forecastLocation=[location San Diego]]]). As another example and not by way of limitation, the NLU output for "open my photos" can be (get [photo hasCreator=[user my]]). As yet another example and not by way of limitation, the NLU output for "show me photos of Andrew and me" can be (get [photo hasPersonSubject=<[person name=[ Andrew ] ], [user me] >]). Although this disclosure describes particular NLU outputs in a particular manner, this disclosure contemplates any suitable NLU outputs in any suitable manner.

[0149] In particular embodiments, the assistant system 140 can resolve, through the reasoning module 214 / 222 associated with the assistant xbot, one or more entities associated with the at least one object based on at least one property of the semantic representation. In particular embodiments, the entity resolution module 330 of the reasoning module 214 / 222 can be a particular component for resolving entities. The entity resolution module 330 can not need to have any type of mapping layer to map the output from the NLU module 210 / 218 to the knowledge graph, as it uses the exact same type of semantic representation. As an example and not by way of limitation, the output from the NLU module 210 / 218 can be (play [track hasPerformer=[person name=[Beyonce]]]), where hasPerformer points to a “person” object. This output can be the same as the performer / artist knowledge graph schema. The “person” object can be mapped directly to the knowledge graph. As another example and not by way of limitation, the output from the NLU module 210 / 218 can be (fb / create [fb / call hasRecipient=[fb / person name=[Andy]]]). The entity resolution module 335 can take fb / person name=[Andy], with the type and return PersonEntity (name=Andy, ID = 123, gender =male). As yet another example and not by way of limitation, the output from the NLU module 210 / 218 can be (fb / create [fb / call hasRecipient=[fb / person motherOf=[fb / user]]]). The entity resolution module 335 can take [fb / person motherOf=[fb / user]] and resolve the leaf object “fb / user”. The entity resolution module 335 can also output the homonym matching text “my”. “motherOf” can be resolved to the concept entity, getting the Spanish relationship name, which is madre. “motherOf + fb / user + fb / person” can need to be mapped to an agent with the get contact information function. In particular embodiments, the entity resolution module 330 can modify the NLU output, if needed. Because the semantic roles and type information are carried in the annotations, the work of the entity resolution module 335 can start in the NLU module 210 / 218. The types and properties can be consistent across modalities, for which the multi-modal input can be represented in the same ontology. As an example and not by way of limitation, the X-ray concept of “person” and the NLU concept of “person” can be the same.Regardless of whether the type information is collected through speech or vision, the NGO can provide uniform behavior and support multi-modal co-reference. Thus, the NGO can have a technical advantage of sharing a semantic space because the NGO can directly link concepts and their lexicalizations to annotations without a mapping between the two. Although this disclosure describes a particular entity resolution in a particular manner, this disclosure contemplates any suitable entity resolution in any suitable manner.

[0150] In particular embodiments, the output from the entity resolution module 330 generated based on the NGO can then be passed to a dialog manager 335. The dialog manager 335 can consult a dialog policy 345 and form a task. The task can be triggered by a combination of NGO actions, objects, and optionally attributes and methods. In particular embodiments, the NGO can have many human-readable surface forms. Textual forms can be used primarily for annotations, while graphical forms can represent execution graphs during resolution. The two forms can be semantically equivalent, which is guaranteed by the NGO resolver. For implementation purposes, the tree form equivalent to the graphical form can be provided to the dialog manager 335 as input. The tree form can push the resolved actionable parts, e.g., "create action+object" or "action+method," to the top of the graph (the trunk), and always have the parameters to create the action or method as trailing branches. Although this disclosure describes a particular NGO form in a particular manner, this disclosure contemplates any suitable NGO form in any suitable manner.

[0151] Figure 11A-11B An example tree NGO form for an example dialog is shown. Figure 11A An example tree form of NGO representation for an example conversation is shown. The example conversation can be "remind me to buy bread when I get home." For this conversation, the text form of NGO representation can be (fb / create [fb / reminder hasTodo=[fb / todo todoText=[buy bread]] triggerLocation=[fb / user].homeLocation]). The NGO representation can begin with "action: create" 1102, which has a parameter 1104 that is "positional (0)" 1106. The "object: reminder" 1108 that is created can have a parameter 1110. The parameter 1110 can be linked by "hasToDo" 1114 to "object: todo" 1112. The "object: todo" 1112 can have a parameter 1116 that is "todoNext" 1118, which can be linked to "object: string" 1120. The "object: string" 1120 can have a parameter 1122 that is "positional (0)" 1106, which can be linked to "buy bread" 1124. The parameter 1110 can also be linked by "triggerLocation" 1128 to "get attribute: homeLocation" 1126. The "triggerLocation" 1128 can have an attribute "called from" 1130, which can be linked to "object: user" 1132.

[0152] Figure 11B Another example tree form of NGO representation is shown for another example dialog. The example dialog can be "add Paul to the call." For this dialog, the text form of NGO representation can be (fb / update[fb / call].hasParticipant.add ([fb / person name=[Paul]])). The NGO representation can begin with "action: update" 1134 with a parameter 1136 that is "positional (0)" 1106. The "method: add" 1138 of the update can have a parameter 1140. The parameter 1140 can be linked by "called from" 1130 to "get attribute: hasParticipant" 1142. The "get attribute: hasParticipant" 1142 can be linked by "called from" 1130 to "object: call" 1144. The parameter 1140 can embed an additional parameter 1146 of "positional (0)" 1106, which can be linked to "object: person" 1148. The "object: person" 1148 can have a parameter 1150 of "name" 1152, which can be linked to "object: string" 1120. The "object: string" 1120 can have a parameter 1154 of "positional (0)" 1106, which can be linked to "Paul" 1156.

[0153] In particular embodiments, the dialog manager 335 can track a task state, where the task state is a subset of the information tracked in the dialog state. The task state can be distinct from the NLU data structure, but can be derived from the NLU module 210 / 218. Dialog state tracking can operate using an NGO-like graph structure as part of the dialog state. This structure can be adorned with additional information, such as entities, confidence scores, and flags indicating the state of actions, objects, parameter values, and entities. In particular embodiments, the state tracking can track independent marginals for each action, object, and parameter. These can then be combined using NGO to rule out invalid combinations. The specifics of how the internal dialog state is chosen can be a tradeoff between characteristics, engineering cost, ease of standing for domain developers, agent and task requirements, and the like. The dialog manager 335 can be at the center of handling the complexity of parsing and managing the task state, the latter of which is closely tied to the agent API specification. For NGO, the assistant system 140 can treat each NGO parse as an unsegmented assumption, and rank N best NLU assumptions given the device context, user preferences, and the like. The provision of "placeholder" variables can provide a mechanism for expanding the range of alternative assumptions that can be considered in this setting. Placeholders can also allow the NLU output to express implicit missing values, e.g., "add to the call" can result in (fb / update [fb / call].hasParticipant.add()), which captures enough signal that the user wants to "add" to the call even if they don't say who. Although this disclosure describes tracking a particular task state in a particular way, this disclosure contemplates tracking any suitable task state in any suitable way.

[0154] In particular embodiments, the dialog manager 335 can encode the post-conversation turns in a way that results in syntax that expresses updates to previously created objects. Assuming NGO-based structures are stored in the dialog state, these NGO parses can be directly translated into operators. As an example and not by way of limitation, "add Samay" can be translated into (fb / update [fb / prev_obj].personAttribute.add ([fb / person name=[Samay]])). As another example and not by way of limitation, "buy a movie ticket" can be translated into (fb / update [fb / prev_obj].todo.add ([buy a movie ticket])). As another example and not by way of limitation, "here" can be translated into (fb / update [fb / prev_obj].locationAttribute.add ([fb / location here])). As an example and not by way of limitation, "remove Trenton and add Kamryn" can be translated into (fb / update [fb / prev_obj].personAttribute.remove ([fb / person name=[Trenton]].add ([fb / person name=[Kamryn]])). As yet another example and not by way of limitation, "first one" can be translated into (fb / get [fb / prev_obj ordinal=[first]]). Although this disclosure describes encoding particular post-conversation turns in a particular way, this disclosure contemplates encoding any suitable post-conversation turns in any suitable way.

[0155] In particular embodiments, the dialog policy 345 can be divided into two levels, namely, general policies 346 and task policies 347. For task policies 347, a plan tree for defining each task can be retrieved based on a combination of NGO actions, objects, attributes, and methods. The plan tree can then use predicates to test for missing arguments and check against state flags such as "confirmed" against objects, actions, and arguments, assuming the dialog state includes an NGO-like graph structure. In particular embodiments, the assistant system 140 can pass all top-level arguments associated with a user request to the entity resolution module 330 and have it process the embedded actions. In alternative embodiments, the assistant system 140 can have the action selector 345 or the dialog manager 335 first traverse the NGO parse tree to process the leaf actions until it reaches the top action. Figure 12 An example NGO resolution based on a plan tree is shown. The NGO resolution can begin with "action: create" 1202, which has a parameter 1204 of "positional (0)" 1206. The created "object: reminder" 1208 can have a parameter 1210. The parameter 1210 can be linked to "object: todo" 1212 by "hasToDo" 1214. The "object: todo" 1212 can have a parameter 1216 of "todoAction" 1218, which can be linked to "action: create" 1202. The "action: create" 1202 can have a parameter 1220 of "positional (0)" 1206, which can be linked to "object: call" 1222. The "object: call" 1222 can further include a parameter 1224 of "hasParticipant" 1226, which can be linked to "object: person" 1228. The "object: person" 1228 can have a parameter 1230 of "motherOf" 1232, which can be linked to "object: user" 1234. The parameter 1210 can also be linked to "object: date_time" 1236 by "hasRemindTime" 1238. The "object: date_time" 1236 can have a parameter 1240 of "positional (0)" 1206, which can be linked to "4pm" 1242. Although this disclosure describes a particular NGO resolution in a particular way, this disclosure contemplates any suitable NGO resolution in any suitable way.

[0156] In particular embodiments, the assistant system 140 can use execution graphs as a way to develop more complex dialog plans that automatically stitch smaller tasks into larger ones. NGO resolution can generate execution graphs based on user utterances. A benefit of this approach can be the ability to handle the long tail of multi-action requests by being able to stitch larger tasks from smaller ones. Figure 13A An example execution graph for the utterance is shown. As an example and not by way of limitation, the utterance can be "share the pictures of me and Kevin skiing with my Mom," which can be represented as (fb / send [fb / message hasContent=[fb / photo hasPersonSubject=<[fb / user], [fb / person name=[Kevin]]> hasSubject=[skiing]] hasRecipient=[fb / person motherOf=[fb / user]]]). The execution graph can include "action: send" 1302 with parameter 1304 that is "positional (0)" 1306. The "object: message" 1308 that the action is taken on can have parameter 1310. Parameter 1310 can be linked by "hasContent" 1314 to "object: photo" 1312. "object: photo" 1312 can have parameter 1316. Parameter 1316 can be linked by "hasPersonSubject" 1320 to "list" 1318. "list" 1318 can be linked to "object: user" 1322 via "positional (0)" 1306 and to "object: person" 1324 via "positional (1)" 1326, respectively. "object: person" 1324 can have parameter 1325 that is "name" 1328, which can be linked to "object: string" 1330. "object: string" 1330 can further include parameter 1332 that is "positional (0)" 1306, which can be linked to "Kevin" 1334. Parameter 1316 can also be linked by "hasSubject" 1338 to "object: string" 1336. "object: string" 1336 can have parameter 1340 that is "positional (0)" 1306, which can be linked to "skiing" 1342. Parameter 1310 can also be linked by "hasRecipient" 1346 to "object: person" 1344."object: person" 1344 can have a parameter 1348 that is "motherOf" 1350, which can link to "object: user" 1352.

[0157] Figure 13B An example execution graph for the two subtasks corresponding to the utterance in Figure 13A can be generated based on breaking the utterance into two subtasks, <(fb / get [fb / photo hasPersonSubject=<[fb / user], [fb / person name=[Kevin]]> hasSubject=[skiing]]) and (fb / send [fb / message hasContent=[fb / photo] hasRecipient=[fb / person motherOf=[fb / user]])>. These two subtasks can be illustrated in Figure 13B , where the dialog policy sequentially executes the "get" and "send" tasks to achieve the user's desired result. As shown in Figure 13B , "list" 1354 can link to "action: get" 1356 via "positional (0)" 1306 and to "action: send" 1302 via "positional (1)" 1326, respectively. It can be seen that the execution graph under "action: get" 1356 can be the same as the execution graph under "action: send" 1302 in Figure 13A . The execution graph under "action: send" 1302 can be substantially the same as the execution graph under "action: send" 1302 in Figure 13A , except that there is no subsequent execution under "object: photo" 1312. Although the present disclosure describes particular execution graphs in a particular manner, the present disclosure contemplates any suitable execution graph in any suitable manner.

[0158] In particular embodiments, the dialog manager 335 can further select an agent for performing the task. Both sending requests to the agent and receiving execution results from the agent can be through an application programming interface (API). The API can define a mapping from one or more custom semantic units to structures determined by the structural ontology. In particular embodiments, the request sent to the agent can include a semantic representation modified by the dialog manager 335 associated with the assistant xbot. The execution result can include a semantic representation modified by the agent and the request, and the execution result can use the notation grammar of the structural ontology. The execution result can then be sent back to the dialog manager 335, which can be responsible for mapping it back to the common semantic representation defined by the NGO. In alternative embodiments, the agent can send back its results using the NGO semantic representation (which can be defined by the API that the assistant system 140 uses to interface with the agent), such that the dialog manager 335 can not need to rewrite / map the output of the agent. In particular embodiments, the agent can be a first-party agent associated with the assistant xbot. In alternative embodiments, the agent can be a third-party agent associated with an external computing system, and the structure of the execution result can include one or more custom semantic units. Although this disclosure describes interacting with a particular agent in a particular manner, this disclosure contemplates interacting with any suitable agent in any suitable manner.

[0159] In particular embodiments, NGOs can provide an easy way for third-party agents to be supported by the assistant system 140. In particular embodiments, for each agent’s API, the agent developer and the NGO can agree on the naming and types of input parameters in order to minimize the amount of remapping needed at the dialog task level. The NGO can have a core namespace of its ontology that is only visible to the modules in the assistant system 140. To enable third-party agent developers to develop agents that are compatible with the assistant system 140, the NGO’s subgraphs, e.g., some actions, objects, and properties, can be made visible to them. These subgraphs can be about general things, that is, not those from the core namespace. As an example and not by way of limitation, a third-party developer can have access to subgraphs related to general functionality, e.g., date time or named entities, but subgraphs related to sensitive content, e.g., social network data, cannot be shared with third-party developers. Third-party agent developers can create custom namespaces with custom objects, actions, and properties. The custom namespaces can be seen as extensions of the NGO’s ontology. As an example and not by way of limitation, a user request can be “get crypto coin price.” The core namespace can have the action “get,” but “crypto coin price” can not exist as an object type in the core namespace. From the third-party’s perspective, a new custom namespace can be created that has a new crypto coin object and the functionality of getting the crypto coin price. The custom namespace can have mappings to the core namespace. In this way, the NGO can enable third-party agent developers to create custom semantic units and also be able to reuse existing semantic units from the core namespace. It can also facilitate communication between different agents of the assistant system 140. Although this disclosure describes particular third-party support in a particular manner, this disclosure contemplates any suitable third-party support in any suitable manner.

[0160] In particular embodiments, the dialog manager 335 can create semantic parts for the NLG module 356 based on the output of the agents and pass them to the NLG module 356. The NLG module 356 can also use the same type of NGO semantic representation to generate responses for the user based on these semantic parts. Figure 14 An example response generation based on NGOs is shown. In particular embodiments, the assistant system 140 can generate a response based on the structural ontology and the results of the executed tasks by the natural language generation module 356 associated with the assistant xbot. As Figure 14 As shown, the NLG module 356 can generate a response based on the dialogue action 1405 including an NLG task and parameters. The dialogue action 1405 can be stored as a subgraph in a graph similar to the NGO. The NLG task can be determined by the dialogue plan. The parameters can include attributes in the NGO, such as temperature, visibility, forecast, precipitation, etc. In particular embodiments, complex parameters can include objects in the NGO and their attributes. In Figure 14 In particular embodiments, the generated response 1410 can be "this weekend in Menlo Park, it'll be sunny on Saturday, but expect rain on Sunday." Although this disclosure describes generating particular responses in particular ways, this disclosure contemplates generating any suitable responses in any suitable ways.

[0161] The following are examples of semantic parts of the NLG module 356 and their corresponding responses. When the user asks "what's the weather like in Seattle?", the semantic part can be (fb / inform [fb / location In city=[Seattle]], it's [fb / temperature 72] and [fb / cloudCover sunny]). The NLG module 356 can accordingly generate the response "in Seattle, it's 72 and sunny." As another example, when the user asks "change my availability to busy", the semantic part can be (fb / ack Ok, availability changed to [::available (busy)]). The NLG module 356 can accordingly generate the response "OK, availability changed to busy." As another example, when the user asks "delete the message", the semantic part can be (fb / confirm Are you sure you want to delete the message?). The NLG module 356 can accordingly generate the response "are you sure you want to delete the message?" As another example, when the user asks "remove the angry emoji", the semantic part can be (fb / ack OK, here is your message) and (fb / prompt Ready to send it?). The NLG module 356 can accordingly generate the response "OK, here is your message. Ready to send it?" As another example, when the user asks "how's the weather in Tokyo, Montana?", the semantic part can be (fb / error I couldn't find that [fb / bad_arg location]). The NLG module 356 can accordingly generate the response "I couldn't find that location."As another example, when the user asks "send a message,” the semantic part can be (fb / request Who would you like to message?). The NLG module 356 can generate the response "who would you like to message?” accordingly. To compare the performance of the NGO and IN / SL paradigms, embodiments disclosed herein compare the representations under two scenarios: 1) how the NGO compares to the IN / SL paradigm given all the data, and 2) how the NGO compares to the IN / SL paradigm given various splits of the data sets, representing low-resource situations. Embodiments disclosed herein can directly compare the NGO to the IN / SL representations, controlling for the natural language utterances used by training and testing on aligned data.

[0162] In particular embodiments, the metric for comparison can be Exact Match (EM) accuracy, where it is checked whether the predicted parses exactly match the target parses, consistent with existing work in the semantic parses of the combined parses. Due to the structured nature of the semantic representations, embodiments disclosed herein also compute the parse validity across each model and representation, which is the frequency of predicted sequences that are both syntactically and semantically valid structures with respect to a predefined ontology.

[0163] Embodiments disclosed herein use seq2seq-based models with pointer projection for comparison. These models utilize a pointer projection layer on top of a traditional seq2seq architecture to decide when to generate a symbol and when to copy a symbol from the input. In particular embodiments, pointer projection is applied to a wide range of base architectures, including BiLSTM, CNN, non-autoregressive CNN, and BART. The same number of experiments are conducted for each model and representation. Table 1 shows the results of comparing these models with different representations. Across all domains, across various model architectures, the models trained to produce NGO labels consistently outperform the models in the IN / SL paradigm. Improvements are obtained even though the NGO parses encode more semantics than their equivalent IN / SL parses (i.e., list types, enumerations).

[0164]

[0165] Table 1: Results for recent seq2seq-based semantic parsing models.

[0166]

[0167] Table 2: Results for syntactic and semantic parse invalidity. The numbers in parentheses are syntactic errors (shown as percentages) of the generated parses.

[0168] The embodiments disclosed herein further evaluate how difficult it is to generate syntactically or semantically valid semantic parses. To this end, the embodiments disclosed herein compute the percentage of invalid generated semantic parses on the test set in Table 2. Overall, across various model architectures, NGO generated parses are both syntactically and semantically valid, with the same percentage of time as IN / SL parses, while having a significantly more strict type system.

[0169] The embodiments disclosed herein evaluate how well the model adapts to NGO in low-resource scenarios. Figure 15 Example performance of the model on various percentages of training data, while keeping the test set constant, is shown. The model is BART. NGO 1510 is strictly superior to IN / SL paradigm 1520 on all sizes of training data.

[0170] Table 3 shows the performance of NGO and IN / SL representations on different domains for the BART model. NGO performs better than IN / SL by 1.2% on weather, 0.64% on timer, and 1.5% on music, while regressing by 0.44% on the reminders domain.

[0171]

[0172] Table 4: Precision of exact match for BART model for each domain for IN / SL label and NGO label.

[0173] In particular embodiments, the assistant system 140 can provide NGO tools for developers to customize the use of NGO, e.g., to create new domains, edit domains, create new entity types, edit entity types, etc. In the NGO tools, a tagging system can be immediately exported by identifying elements from the ontology or by adding elements to the ontology. A developer can build the ontology as follows. The developer can create or update their domains in the NGO tools. The developer can also create domain-specific subgraphs of the ontology, i.e., they can identify the action and object classes for a given domain from the ontology and the properties of these classes. The developer can reuse existing classes, which can ensure semantic compatibility. However, if a class or property does not exist, the developer can create a new class and provide the required properties. These classes can be separated from the core ontology by a namespace. Legal tags can be directly exported from the subgraph of any domain. Once the classes and properties are identified, these possibilities can be immediately generated. Although this disclosure describes a particular NGO tool in a particular way, this disclosure contemplates any suitable NGO tool in any suitable way.

[0174] An NGO parser can be used to determine whether a developer's data is valid in terms of NGO syntax and domain-specific semantics. An NGO tool can automatically pre-populate all actions, objects, properties, method combinations allowed by a subgraph. The developer can choose to remove unwanted combinations. The developer can also point to the required properties and templatic responses for NLG dialog actions. The developer can create property type pairs indicating the role and type played by one object class relative to another. These possibilities can be automatically generated by declaring in the ontology and immediately available for new domains once classes and properties are identified as reusable or created. The developer can link properties with NGO types. Because the ontology is shared between the NLU module 210 / 218 and the knowledge graph, the developer's work of matching elements of the ontology to elements of the knowledge graph can be removed. The NGO tool can have an entity page that can be automatically pre-populated with legal role type information. By default, it can be assumed that these roles are legal in any type of the subgraph, but the page can be used to manually remove some options. The developer can provide example utterances for the domain by simply providing complete annotations that follow the NGO guidelines. Although this disclosure describes using a particular NGO tool in a particular way, this disclosure contemplates using any suitable NGO tool in any suitable way.

[0175] Figure 16 An example is shown for enabling a voice-enabled application (App) in an NGO. In particular embodiments, semantic units can each be associated with a default namespace defined by a computing system. One or more custom semantic units can be associated with a personalized namespace defined by a third-party agent. In Figure 16 In this example, the example can be specifically used with the new timer, `timer_app`. A core ontology existing in the core namespace 1610, such as `fb / `, might already have a timer object 1612 defined with two properties: "hasDuration" 1614 and "hasStartTime" 1616. This allows statements such as "set a timer at 5pm" to be resolved as `fb / set [fb / timer hasStartTime=[fb / date_time 5pm]]`. For the app, the developer can create a new namespace 1620 containing a new timer object 1622 (i.e., `app / timer`) that inherits from the core timer object (i.e., `fb / timer`). Given the following statement "set a timer at 5pm with app", Assistant System 140 can annotate it based on NGO using (fb / set [app / timer hasStartTime=[fb / date_time 5pm]]. Note that due to the inheritance of fb / timer, there may be a lot of reuse.

[0176] In certain embodiments, entity resolution may depend on entity types and entity union definitions. Entity union definitions can be independent of NLU modules 210 / 218. Developers can define entity types in the ontology using the NGO tool. As an example, and not a limitation, to resolve the NGO leaf object fb / person name=[Andy], fb / person name should define a field entity type, which can be assigned by the developer in the NGO tool. In certain embodiments, fb / person may already be an entity type. Therefore, NGO can directly output the entity type integrated with the NGO tool.

[0177] In a particular embodiment, the assistant system 140 may have some concept of plans (for each task) that the assistant system 140 executes. These plans can allow the assistant system 140 to predict the desired outcome, know what steps need to be performed, in what order, and what information needs to be captured and parsed. In a particular embodiment, there may be a "task trigger," which can be considered as plan retrieval, i.e., given the current dialogue state and input, what plan should the assistant system 140 choose to execute. In a particular embodiment, to conceptually simplify the plan retrieval mechanism for developers, an interface can be presented to them in the NGO tool that captures the conditions under which their task triggers. The task name in the NGO may include an independently designated task name.

[0178] In certain embodiments, for NGOs, task triggering can be explicitly defined using a combination of actions, objects, and optional attributes and / or methods. In certain embodiments, developers may need to specify a minimum of two and a maximum of four pieces of information about when their tasks should trigger. As an example, and not a limitation, "fb / create" and "fb / call" may be sufficient to describe the trigger for TASK:CREATE_CALL. As another example, and not a limitation, "fb / update" and "fb / call" with the attribute "callType" may be sufficient to describe the task TASK:UPDATE_METHOD_CALL. As another example, and not a limitation, "fb / update", "fb / timer", the attribute "duration", and the method "add" may be sufficient to describe the task TASK:ADD_TIME_TIMER (e.g., "add 5 minutes to my timer"). The expected attributes and methods may take the special value "none," which may restrict triggering to specific situations when objects and actions are mentioned without attributes or methods. The presence of attributes or methods in NGO annotations can indicate to the user's discourse that the attribute or method is relevant. One requirement for NGOs is to allow a sufficiently fine-grained mapping to the dialogue plan; when attributes and methods are implied by the user's discourse, they should be included in the annotations, even if no value is given. To simplify the protocol or user interface (UI) of NGO tools, the values ​​captured by attributes or methods may not be used. Although this disclosure describes specific task triggers in a particular manner, this disclosure contemplates any suitable task triggers in any suitable manner.

[0179] In a particular embodiment, developers may be provided with two abstractions to represent a general dialogue plan. These two abstractions may have different expressiveness and capabilities: (i) a frame-based (also known as "configurator-based") abstraction—for "parameter population," and (ii) a dialogue policy API—for tasks requiring explicit control over the dialogue flow. Each of these abstractions may impose constraints on the complexity of the tasks that can be described. This can simplify the information that developers must provide. To some extent, it may also limit the complexity of the NGO parsing that can be interpreted and executed. For the "no-code" frame-based abstraction, the NGO ontology can guide developers on what types of input parameters can be added to the NGO tool. Although this disclosure describes a particular abstraction for a particular dialogue plan in a particular manner, this disclosure contemplates any suitable abstraction for any suitable dialogue plan in any suitable manner.

[0180] Figures 17A-17B A sample UI for an NGO tool is shown. Figure 17A This example UI shows an NGO tool with a "create" trigger action. When the trigger action is "fb / create", a set of named parameters related to the object to be created can be specified in the "Input Parameters" section. Each input parameter may have a type and parameter name, which can be automatically populated from the NGO ontology. Developers can mark whether each parameter is required and provide relevant hints. Note that a "fb / create" scheduled trigger may mean that no property or method references are part of the trigger specification. Figure 17A As shown, a task can be designated as "TASK: CREATE_CALL" 1702. The UI can also display some descriptions 1704. The trigger section 1706 can include information associated with actions, objects, etc. The UI can also display alias sections 1708, proxy targets 1710, policy comments 1712, and input parameters 1714.

[0181] Figure 17B This example UI shows an NGO tool with an "update" trigger action. If the trigger action is not "fb / create" and a property or method reference is provided as part of the trigger action, then no input parameters may be allowed, or only one input parameter may be allowed. If that single parameter is specified, it may take the name and type associated with the property mentioned in the trigger. Developers can mark whether the parameter is required and provide relevant hints. Figure 17B As shown, a task can be designated as "TASK: UPDATE_METHOD_CALL" 1716. The UI can also display some descriptions 1704. The trigger section 1706 can include information associated with actions, objects, etc. The UI can also display alias sections 1708, proxy targets 1710, strategy comments 1712, and input parameters 1714. NGO tools can be an effective solution to the technical challenge of enabling developers to easily use structural ontologies without extensive knowledge of ontologies, because NGO tools allow developers to customize the use of structural ontologies with different functionalities using an advanced user interface. Within this advanced user interface, the system can also automatically populate content to facilitate the use of such tools.

[0182] In a particular implementation, the Dialogue Planning API allows developers to specify the expected dialogue flow using a combination of a graphical UI and an object-oriented programming language. The flow can then be transformed into an executable plan tree. Plans constructed in this way may allow branching and looping. At each decision point, developers can use pre-existing "predicates" to test common conditions or build their own custom code. An NGO parser can also allow the definition of predicates. If an NGO-like structure is used to represent dialogue states, it is expected that developers might construct predicates. Furthermore, a predefined set of predefined predicates can be provided for common tests. These predefined predicates can also support the generation of dialogue plans for the aforementioned "no-code" framework-based abstractions.

[0183] Figure 18 An example method 1800 for improving semantic representation consistency is illustrated. The method may begin at step 1810, where an assistant system 140 receives user input, including natural language utterances, from a client system 130 associated with a user via an assistant xbot. At step 1820, the assistant system 140 may determine the semantic representation of the user input based on a structural ontology, wherein the structural ontology defines 1) a marked syntax for parsing natural language utterances into semantic units comprising multiple actions, objects, and attributes; 2) each of the multiple actions operates on one or more of the multiple objects; 3) each of the multiple objects includes one or more of the multiple attributes; 4) each of the multiple attributes declares a constraint on an action or object; 5) the multiple objects are hierarchically organized into multiple supertypes and subtypes, each subtype inheriting one or more attributes from its associated supertype; 6) the semantic units also include multiple methods and enumerations, wherein each of the multiple methods includes an action whose lifetime is bound to an object, and wherein each of the multiple enumerations includes an object representing one of the multiple objects; and 7) including... A graph structure with one or more core subgraphs and one or more general subgraphs, wherein one or more core subgraphs are inaccessible by a third-party agent and one or more general subgraphs are accessible by a third-party agent, wherein the semantic representation embeds at least one object into at least one action and declares at least one attribute of the embedded object to be acted upon, wherein the at least one action has one or more parameters, wherein each parameter specifies a constraint to be satisfied by the object such that the at least one action can act on the object, and wherein at least one of the parameters specifies a constraint satisfied by at least one object, wherein the semantic representation can be executed by each module associated with an assistant xbot configured to process user input, and wherein the value associated with each of the at least one action, at least one object, and at least one attribute is determined by each module executing the semantic representation. In step 1830, the assistant system 140 may send a request to an agent to perform a task corresponding to user input, wherein the request is based on a semantic representation, wherein the agent is a third-party agent associated with an external computing system, and wherein the structure of the execution result includes one or more custom semantic units, wherein sending the request to the agent and receiving the execution result from the agent are both via an application programming interface (API), wherein the API defines a mapping from one or more custom semantic units to a structure determined by a structure ontology, wherein the request sent to the agent includes a semantic representation modified by a dialogue manager associated with the assistant xbot, wherein the execution result includes a semantic representation modified by the agent, and wherein the request and the execution result use the markup syntax of the structure ontology, wherein each semantic unit is associated with a default namespace defined by the computing system, and wherein one or more custom semantic units are associated with a personalized namespace defined by the third-party agent.In step 1840, the assistant system 140 can receive the results of the performed task from the agent, wherein these results are mapped to a structure determined by the structural ontology. In step 1850, the assistant system 140 can generate a response based on the structural ontology and the results of the performed task by the natural language generation module 356 associated with the assistant xbot. In step 1850, the assistant system 140 can send instructions from the assistant xbot to the client system 130 for presenting a response to user input, wherein the response is based on the results of the performed task. Certain embodiments may be repeated where appropriate. Figure 18 One or more steps of the method. Although this disclosure will Figure 18 The specific steps of the method are described and shown as occurring in a specific order, but this disclosure contemplates... Figure 18 Any suitable steps of the method occur in any suitable order. Furthermore, although this disclosure describes and illustrates example methods for improving the consistency of semantic representations, including... Figure 18 This disclosure describes specific steps of the method, but contemplates any suitable method for improving the consistency of semantic representations, including any suitable steps, which, where appropriate, may include... Figure 18 The method may include all, some, or none of these steps. Furthermore, although this disclosure describes and illustrates the execution of... Figure 18 The method may refer to specific components, devices, or systems for specific steps, but this disclosure contemplates the execution of... Figure 18 Any suitable combination of any suitable component, device, or system of any suitable step of the method.

[0184] Social media Figure 19 An example social graph 1900 is illustrated. In a particular embodiment, the social network system 160 may store one or more social graphs 1900 in one or more data stores. In a particular embodiment, the social graph 1900 may include multiple nodes—which may include multiple user nodes 1902 or multiple concept nodes 1904—and multiple edges 1906 connecting these nodes. Each node may be associated with a unique entity (i.e., a user or a concept), and each entity may have a unique identifier (ID), such as a unique number or username. For educational purposes, a two-dimensional visual map representation is shown. Figure 19 The example social graph 1900 is shown. In a particular embodiment, a social network system 160, a client system 130, an assistant system 140, or a third-party system 170 can access the social graph 1900 and related social graph information for appropriate applications. The nodes and edges of the social graph 1900 can be stored as data objects in, for example, a data storage device (e.g., a social graph database). Such a data storage device may include one or more searchable or queryable indexes of the nodes or edges of the social graph 1900.

[0185] In a particular embodiment, user node 1902 may correspond to a user of social networking system 160 or assistant system 140. By way of example and not limitation, a user may be an individual (human user), entity (e.g., a business, company, or third-party application), or group (e.g., an individual or entity) that interacts or communicates with or through social networking system 160 or assistant system 140. In a particular embodiment, when a user registers an account with social networking system 160, social networking system 160 may create a user node 1902 corresponding to the user and store user node 1902 in one or more data storage devices. The user and user node 1902 described herein may, where appropriate, refer to a registered user and the user node 1902 associated with the registered user. Alternatively, where appropriate, the user and user node 1902 described herein may refer to a user who has not registered with social networking system 160. In a particular embodiment, user node 1902 may be associated with information provided by the user or information collected by various systems, including social networking system 160. As an example, and not a limitation, a user may provide his or her name, profile picture, contact information, date of birth, gender, marital status, family status, occupation, educational background, preferences, interests, or other demographic information. In a particular embodiment, user node 1902 may be associated with one or more data objects corresponding to information associated with the user. In a particular embodiment, user node 1902 may correspond to one or more web interfaces.

[0186] In a particular embodiment, concept node 1904 may correspond to a concept. By way of example and not limitation, a concept may correspond to a location (such as, for example, a movie theater, restaurant, landmark, or city); a website (such as, for example, a website associated with social networking system 160 or a third-party website associated with a web application server); an entity (such as, for example, an individual, business, group, sports team, or celebrity); a resource (such as, for example, an audio file, video file, digital photograph, text file, structured document, or application), which may reside on a server (e.g., a web application server) within or outside social networking system 160; real estate or intellectual property (such as, for example, sculpture, painting, film, game, song, idea, photograph, or written work); a game; an activity; an idea or theory; another suitable concept; or two or more such concepts. Concept node 1904 may be associated with information about a concept provided by a user or information collected by various systems, including social networking system 160 and assistant system 140. By way of example and not limitation, concept information may include a name or title; one or more images (e.g., an image of a book cover); location (e.g., an address or geographic location); a website (which may be associated with a URL); contact information (e.g., a phone number or email address); other suitable concept information; or any suitable combination of such information. In a particular embodiment, concept node 1904 may be associated with one or more data objects corresponding to information associated with concept node 1904. In a particular embodiment, concept node 1904 may correspond to one or more web interfaces.

[0187] In a particular embodiment, nodes in social graph 1900 may represent or be represented by a web interface (which may be referred to as a "profile interface"). The profile interface may be hosted by or accessible to social network system 160 or assistant system 140. The profile interface may also be hosted on a third-party website associated with third-party system 170. By way of example, and not limitation, a profile interface corresponding to a particular external web interface may be that particular external web interface, and the profile interface may correspond to a particular concept node 1904. The profile interface may be viewable by all other users or a selected subset of other users. By way of example, and not limitation, user node 1902 may have a corresponding user profile interface, where the corresponding user can add content, make statements, or otherwise express himself or her. By way of another example, and not limitation, concept node 1904 may have a corresponding concept profile interface, where one or more users can add content, make statements, or express themselves, particularly regarding the concept corresponding to concept node 1904.

[0188] In a particular embodiment, concept node 1904 may represent a third-party web interface or resource hosted by third-party system 170. Among other elements, the third-party web interface or resource may include content, selectable or other icons, or other interactive objects representing actions or activities. By way of example and not limitation, the third-party web interface may include selectable icons such as "like," "check-in," "eat," "recommend," or other suitable actions or activities. A user viewing the third-party web interface can perform an action by selecting one of the icons (e.g., "check-in"), causing client system 130 to send a message instructing the user's action to social network system 160. In response to this message, social network system 160 may create an edge (e.g., a check-in type edge) between user node 1902 corresponding to the user and concept node 1904 corresponding to the third-party web interface or resource, and store edge 1906 in one or more data stores.

[0189] In a particular embodiment, a pair of nodes in social graph 1900 can be connected to each other via one or more edges 1906. An edge 1906 connecting a pair of nodes can represent a relationship between the two nodes. In a particular embodiment, edge 1906 may include or represent one or more data objects or attributes corresponding to the relationship between the pair of nodes. As an example, and not a limitation, a first user can indicate that a second user is a "friend" of the first user. In response to this indication, social network system 160 can send a "friend request" to the second user. If the second user confirms the "friend request," social network system 160 can create an edge 1906 in social graph 1900 connecting the first user's user node 1902 to the second user's user node 1902, and store edge 1906 as social graph information in one or more data stores 164. Figure 19 In the example, social graph 1900 includes edges 1906 indicating a friendship relationship between user nodes 1902 of user "A" and user "B", and edges indicating a friendship relationship between user nodes 1902 of user "C" and user "B". Although this disclosure describes or illustrates a specific edge 1906 with a specific attribute relating to a particular user node 1902, this disclosure contemplates any suitable edge 1906 relating to user node 1902 with any suitable attribute. By way of example and not limitation, edge 1906 may represent friendship, family relationship, business or employment relationship, liker relationship (including, for example, likes, etc.), follower relationship, visitor relationship (including, for example, visit, view, check-in, share, etc.), subscriber relationship, superior / subordinate relationship, reciprocal relationship, non-reciprocal relationship, another suitable type of relationship, or two or more such relationships. Furthermore, although this disclosure generally describes nodes as being associated, this disclosure also describes users or concepts as being associated. In this document, references to associated users or concepts may, where appropriate, refer to the nodes in the social graph 1900 that correspond to those users or concepts and are connected by one or more edges 1906. The separation degree between two objects represented by two nodes is the count of edges in the shortest path connecting the two nodes in the social graph 1900. As an example and not a limitation, in the social graph 1900, user node 1902 of user "C" is connected to user node 1902 of user "A" via multiple paths, such as a first path that directly passes through user node 1902 of user "B", a second path that passes through concept node 1904 of company "A1me" and user node 1902 of user "D", and a third path that passes through user node 1902 and concept node 1904 representing school "Stateford", user "G", company "A1me", and user "D". User "C" and user "A" have a separation degree of two because the shortest path connecting their corresponding nodes (i.e., the first path) includes two edges 1906.

[0190] In a particular embodiment, edge 1906 between user node 1902 and concept node 1904 may represent a specific action or activity performed by a user associated with user node 1902 on a concept associated with concept node 1904. This is intended as an example, not a limitation. Figure 19 As shown, users can "like," "attend," "play," "listen," "cook," "work at," or "read" concepts, each of which can correspond to an edge type or subtype. The concept profile interface corresponding to concept node 1904 can include, for example, an optional "check-in" icon (such as, for example, a clickable "check-in" icon) or an optional "add to favorites" icon. Similarly, after a user clicks these icons, the social network system 160 can create a "favorites" edge or a "check-in" edge in response to the user action corresponding to the respective action. As another example, and not as a limitation, a user (user "C") can use a specific application (a third-party online music application) to listen to a specific song ("Imagine"). In this case, the social network system 160 can create a "listen" edge 1906 and a "use" edge (such as...) between the user node 1902 corresponding to the user and the concept node 1904 corresponding to the song and application. Figure 19 As shown), to indicate to the user that they have listened to the song and used the application. Furthermore, the social network system 160 can create a "play" edge 1906 (as shown) between the concept nodes 1904 corresponding to the song and the application. Figure 19 As shown), to indicate that a specific song is played by a specific application. In this case, the "play" edge 1906 corresponds to the action performed by an external application (a third-party online music application) on an external audio file (the song "Imagine"). Although this disclosure describes a specific edge 1906 with a specific attribute connecting user node 1902 and concept node 1904, this disclosure contemplates any appropriate edge 1906 with any appropriate attribute connecting user node 1902 and concept node 1904. Furthermore, although this disclosure describes an edge between user node 1902 and concept node 1904 representing a single relationship, this disclosure contemplates edges between user node 1902 and concept node 1904 representing one or more relationships. By way of example and not limitation, edge 1906 could represent that a user likes and uses a specific concept. Alternatively, another edge 1906 could represent the relationship between user node 1902 and concept node 1904 (such as...). Figure 19 As shown, there are various types of relationships (or multiple single relationships) between user node 1902 of user "E" and concept node 1904 of "Online Music Application".

[0191] In a particular embodiment, the social networking system 160 may create an edge 1906 between a user node 1902 and a concept node 1904 in the social graph 1900. As an example, and not a limitation, (such as, for example, using a web browser or dedicated application hosted by the user's client system 130) a user viewing a concept profile interface may indicate that he or she likes the concept represented by concept node 1904 by clicking or selecting a "like" icon. This may cause the user's client system 130 to send a message to the social networking system 160 indicating that the user likes the concept associated with the concept profile interface. In response to this message, the social networking system 160 may create an edge 1906 between the user node 1902 and the concept node 1904 associated with the user, as shown by the "like" edge 1906 between the user node and the concept node 1904. In a particular embodiment, the social networking system 160 may store the edge 1906 in one or more data stores. In a particular embodiment, the edge 1906 may be automatically formed by the social networking system 160 in response to a specific user action. As an example, and not a limitation, if the first user uploads an image, reads a book, watches a movie, or listens to a song, an edge 1906 can be formed between the user node 1902 corresponding to the first user and the concept node 1904 corresponding to those concepts. Although this disclosure describes the formation of a particular edge 1906 in a specific manner, this disclosure contemplates the formation of any suitable edge 1906 in any suitable manner.

[0192] Vector space and embedding Figure 20 An example view of vector space 2000 is shown. In a particular embodiment, it is possible to... d An object or n-gram is represented in a dimensional vector space, where d This represents any suitable dimension. Although vector space 2000 is shown as a three-dimensional space, this is merely for illustrative purposes, as vector space 2000 can have any suitable dimension. In a particular embodiment, an n-gram can be represented as a vector in vector space 2000, referred to as a term embedding. Each vector may include the coordinates corresponding to a specific point in vector space 2000 (i.e., the endpoint of the vector). This is intended as an example, not a limitation. Figure 20 As shown, vectors 2010, 2020, and 2030 can be represented as points in vector space 2000. n-grams can be mapped to their corresponding vector representations. This is shown as an example, not a restriction, by applying functions defined by a dictionary. n-gram and n-gram They can be mapped to vectors in vector space 2000 respectively. and , making and As another example, and not a limitation, a dictionary trained to map text to vector representations can be utilized, or such a dictionary can be generated through training. As another example, and not a limitation, a word embedding model can be used to map n-grams to vector representations in vector space 2000. In a particular embodiment, n-grams can be mapped to vector representations in vector space 2000 using a machine learning model (e.g., a neural network). The machine learning model may have been trained using sequences of training data (e.g., corpora of multiple objects each comprising n-grams).

[0193] In a particular embodiment, an object may be represented as a vector in vector space 2000, which is referred to as a feature vector or object embedding. This is done as an example, not as a limitation, by applying a function. object and They can be mapped to vectors in vector space 2000 respectively. and , making and In certain embodiments, an object can be mapped to a vector based on one or more characteristics, attributes, or features of the object, the object's relationship to other objects, or any other suitable information associated with the object. As an example and not as a limitation, the function... Objects can be mapped to vectors through feature extraction, which can start from an initial measurement dataset and construct derived values ​​(e.g., features). As an example, and not a limitation, objects, including those in videos or images, can be mapped to vectors by using algorithms to detect or isolate various desired parts or shapes of the object. Features used to compute the vectors can be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transforms, edge orientation, change intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transforms (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information. As another example, and not a limitation, objects including audio data can be mapped to vectors based on features (e.g., spectral slope, pitch coefficient, audio spectral centroid, audio spectral envelope, Mel-frequency cepstrum, or any other suitable information). In certain embodiments, when the object has data that is too large to be efficiently processed or includes redundant data, the function... An object can be mapped to a vector using a transformed, streamlined feature set (e.g., feature selection). In a particular embodiment, the function... Can be based on objects e Associate one or more n-grams to group objects e Mapping to vector Although this disclosure describes the representation of n-grams or objects in a vector space in a particular manner, this disclosure contemplates the representation of n-grams or objects in a vector space in any suitable manner.

[0194] In a particular embodiment, the social network system 160 can compute a similarity measure of vectors in the vector space 2000. The similarity measure can be cosine similarity, Minkowski distance, Mahalanobis distance, Jaccard similarity coefficient, or any suitable similarity measure. This is provided as an example and not as a limitation. and The similarity measure can be cosine similarity. As another example, not as a limitation, and Similarity can be measured by Euclidean distance. A similarity measure between two vectors can represent the degree of similarity between two objects or n-grams corresponding to the two vectors, as measured by the distance between two vectors in vector space 2000. As an example, and not a limitation, vectors 2010 and 2020 can correspond to objects that are more similar to each other than the objects corresponding to vectors 2010 and 2030, based on the distance between the corresponding vectors. Although this disclosure describes the calculation of a similarity measure between vectors in a particular manner, this disclosure contemplates the calculation of a similarity measure between vectors in any suitable manner.

[0195] More information on vector spaces, embeddings, feature vectors, and similarity measures can be found in U.S. Patent Application No. 14 / 949436, filed November 23, 2015; U.S. Patent Application No. 15 / 286315, filed October 5, 2016; and U.S. Patent Application No. 15 / 365789, filed November 30, 2016, all of which are incorporated herein by reference.

[0196] Artificial Neural Networks Figure 21 An example artificial neural network ("ANN") 2100 is illustrated. In a particular embodiment, an ANN may refer to a computational model comprising one or more nodes. The example ANN 2100 may include an input layer 2110, hidden layers 2120, 2130, 2140, and an output layer 2150. Each layer of the ANN 2100 may include one or more nodes, such as node 2105 or node 2115. In a particular embodiment, each node of the ANN may be connected to another node of the ANN. By way of example and not limitation, each node of the input layer 2110 may be connected to one or more nodes of the hidden layer 2120. In a particular embodiment, one or more nodes may be bias nodes (e.g., nodes in a layer that are not connected to any node in the previous layer and do not receive input from them). In a particular embodiment, each node in each layer may be connected to one or more nodes in the previous or next layer. Although Figure 21 This disclosure describes a specific ANN with a specific number of layers, a specific number of nodes, and specific relationships between nodes; however, this disclosure contemplates any suitable ANN with any suitable number of layers, any suitable number of nodes, and any suitable relationships between nodes. This is provided as an example and not as a limitation, although... Figure 21 The relationships between each node in the input layer 2110 and each node in the hidden layer 2120 are depicted, but one or more nodes in the input layer 2110 may not be related to one or more nodes in the hidden layer 2120.

[0197] In a particular embodiment, the ANN may be a feedforward ANN (e.g., an ANN without loops or cycles, where communication between nodes flows in one direction starting from the input layer and progresses to successive layers). By way of example, and not limitation, the input to each node of hidden layer 2120 may include the outputs of one or more nodes of input layer 2110. By way of another example, and not limitation, the input to each node of output layer 2150 may include the outputs of one or more nodes of hidden layer 2140. In a particular embodiment, the ANN may be a deep neural network (e.g., a neural network including at least two hidden layers). In a particular embodiment, the ANN may be a deep residual network. A deep residual network may be a feedforward ANN, which includes hidden layers organized into residual blocks. The input to each residual block after the first residual block may be a function of the output of the previous residual block and the input of the previous residual block. By way of example, and not limitation, to the residual block... N The input can be ,in It can be a residual block N- Output of 1 x It can be to the residual block N- 1 is the input. Although this disclosure describes a specific ANN, this disclosure contemplates any suitable ANN.

[0198] In a particular embodiment, the activation function may correspond to each node of the ANN. The activation function of a node may define the node's output for a given input. In a particular embodiment, the node's input may include a set of inputs. By way of example and not limitation, the activation function may be an identity function, a binary step function, a logic function, or any other suitable function. By way of another example and not limitation, a node... k The activation function can be the sigmoid function. tangent function of hyperbola rectifier Or any other suitable function ,in It can be a node k Valid inputs. In a particular embodiment, the inputs to the activation functions corresponding to the nodes can be weighted. Each node can generate an output using the corresponding activation function based on the weighted inputs. In a particular embodiment, each relationship between nodes can be associated with a weight. As an example, and not a limitation, the relationship 2125 between nodes 2105 and 2115 can have a weighting coefficient of 0.4, which can indicate that the output of node 2105 multiplied by 0.4 is used as the input of node 2115. As another example, and not a limitation, nodes k Output It can be ,in It can correspond to a node k activation function, It can be a node k Valid input, It can be related to a node k nodes j The output, and It can be a node j and nodes k The weighting coefficients between them. In a particular embodiment, the input to a node in the input layer may be based on a vector representing the object. Although this disclosure describes specific inputs and outputs of nodes, this disclosure considers any suitable inputs and outputs of nodes. Furthermore, although this disclosure may describe specific relationships and weights between nodes, this disclosure considers any suitable relationships and weights between nodes.

[0199] In certain embodiments, training data can be used to train an ANN. By way of example, and not limitation, the training data may include the inputs and expected outputs of an ANN 2100. By way of another example, and not limitation, the training data may include vectors, each representing a training object and the expected label for each training object. In certain embodiments, training an ANN may include modifying the weights associated with the connections between nodes of the ANN by optimizing an objective function. By way of example, and not limitation, training methods (e.g., conjugate gradient, gradient descent, stochastic gradient descent) may be used to backpropagate the sum of squared errors as a distance measurement between each vector representing a training object (e.g., using a cost function that minimizes the sum of squared errors). In certain embodiments, dropout techniques may be used to train the ANN. By way of example, and not limitation, one or more nodes may be temporarily ignored during training (e.g., not receiving input and not generating output). For each training object, one or more nodes of the ANN may have a certain probability of being ignored. The nodes ignored for a particular training object may differ from the nodes ignored for other training objects (e.g., nodes may be temporarily ignored object-by-object). Although this disclosure describes training ANNs in a particular manner, this disclosure envisions training ANNs in any suitable manner.

[0200] privacy In certain embodiments, one or more objects of a computing system (e.g., content or other types of objects) may be associated with one or more privacy settings. One or more objects may be stored on or otherwise associated with any suitable computing system or application, such as, for example, a social networking system 160, a client system 130, an assistant system 140, a third-party system 170, a social networking application, an assistant application, a messaging application, a photo-sharing application, or any other suitable computing system or application. Although the examples discussed herein are in the context of an online social network, these privacy settings can be applied to any other suitable computing system. The object's privacy settings (or "access settings") may be stored in any suitable manner (e.g., in association with the object, at an index on an authorization server, in another suitable manner, or any suitable combination thereof). The object's privacy settings may specify how the object (or specific information associated with the object) can be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, performed, surfaced, or identified) within the online social network. An object can be described as "visible" relative to a specific user or other entity when its privacy settings allow access to that object by that user or other entity. As an example, and not a limitation, users of an online social network can specify privacy settings for their profile pages that identify a set of users who can access their work experience information on the profile page, thus excluding other users from accessing that information.

[0201] In certain embodiments, the privacy settings of an object may specify a "blocked list" of users or other entities that should not be allowed to access certain information associated with the object. In certain embodiments, the blacklist may include third-party entities. The blacklist may specify one or more users or entities to whom the object is not visible. As an example, and not a limitation, a user may specify a group of users who cannot access an album associated with that user, thereby excluding those users from accessing the album (while potentially allowing access to the album to some users not in the specified user group). In certain embodiments, privacy settings may be associated with a specific social graph element. The privacy settings of a social graph element (e.g., a node or edge) may specify how the online social network can be used to access the social graph element, information associated with the social graph element, or objects associated with the social graph element. As an example, and not a limitation, a specific conceptual node 1904 corresponding to a specific photo may have a privacy setting that specifies that the photo can only be accessed by the user tagged in the photo and the friends of the user tagged in the photo. In certain embodiments, privacy settings may allow users to opt in or out so that their content, information, or actions are stored / recorded by the social network system 160 or assistant system 140 or shared with other systems (e.g., third-party system 170). Although this disclosure describes the use of a particular privacy setting in a particular manner, this disclosure contemplates the use of any suitable privacy setting in any suitable manner.

[0202] In certain embodiments, privacy settings can be based on one or more nodes or edges of the social graph 1900. Privacy settings can be specified for one or more edges 1906 or edge types of the social graph 1900, or for one or more nodes 1902, 1904 or node types of the social graph 1900. Privacy settings applied to a specific edge 1906 connecting two nodes can control whether the relationship between two entities corresponding to those two nodes is visible to other users of the online social network. Similarly, privacy settings applied to a specific node can control whether a user or concept corresponding to that node is visible to other users of the online social network. As an example, and not a limitation, a first user can share an object with the social network system 160. The object can be associated with a concept node 1904 connected to the first user's user node 1902 via edge 1906. The first user can specify privacy settings applied to a specific edge 1906 connected to the concept node 1904 of the object, or can specify privacy settings applied to all edges 1906 connected to the concept node 1904. As another example, and not a limitation, a first user can share a collection of objects of a specific object type (e.g., a collection of images). The first user can specify privacy settings for all objects of that particular object type associated with the first user, such as specifying that all images posted by the first user are only visible to the first user's friends and / or users tagged in the images.

[0203] In a particular embodiment, the social networking system 160 may present a "privacy wizard" (e.g., within a webpage, module, one or more dialog boxes, or any other suitable interface) to a first user to help the first user specify one or more privacy settings. The privacy wizard may display instructions, appropriate privacy-related information, current privacy settings, one or more input fields for accepting changes or confirmations of the specified privacy settings from the first user, or any suitable combination thereof. In a particular embodiment, the social networking system 160 may provide a "dashboard" function to the first user, which displays the first user's current privacy settings. The dashboard function may be displayed to the first user at any appropriate time (e.g., after input from the first user who invoked the dashboard function, or after a specific event or triggering action occurs). The dashboard function may allow the first user to modify one or more of their current privacy settings at any time and in any suitable manner (e.g., redirecting the first user to the privacy wizard).

[0204] Privacy settings associated with an object can specify any suitable granularity for allowing or denying access. As an example, and not as a limitation, access can be specified for specific users (e.g., only me, my roommate, my boss), users within a specific separation (e.g., friends, friends of friends), user groups (e.g., gaming clubs, my family), user networks (e.g., employees of a specific employer, students or alumni of a specific university), all users ("public"), no users ("private"), users of third-party systems, specific applications (e.g., third-party applications, external websites), other suitable entities, or any suitable combination thereof. While this disclosure describes specific granularities for allowing or denying access, this disclosure contemplates any suitable granularity for allowing or denying access.

[0205] In a particular embodiment, one or more servers 162 may be authorization / privacy servers for implementing privacy settings. In response to a request from a user (or other entity) for a specific object stored in data storage 164, the social networking system 160 may send a request for that object to data storage 164. The request may identify the user associated with the request, and the object may only be sent to the user (or the user's client system 130) if the authorization server determines, based on the privacy settings associated with the object, that the user is authorized to access the object. If the requesting user is not authorized to access the object, the authorization server may prevent the requested object from being retrieved from data storage 164 or may prevent the requested object from being sent to the user. In a search-query context, an object may only be provided as a search result if the querying user is authorized to access the object, for example, if the object's privacy settings allow it to be displayed to the querying user, discovered by the querying user, or otherwise visible to the querying user. In a particular embodiment, the object may represent content visible to the user through the user's feed. By way of example and not limitation, one or more objects may be visible to a user's "Trending" page. In certain embodiments, an object may correspond to a specific user. The object may be content associated with a specific user, or it may be a specific user's account or information stored on a social networking system 160 or other computing system. As an example, and not a limitation, a first user may view one or more second users on an online social network through the "People You May Know" feature or by viewing the first user's friend list. As an example, and not a limitation, a first user may specify that they do not wish to see objects associated with a particular second user in their feed or friend list. If an object's privacy settings do not allow it to be exposed to, discovered by, or visible to a user, that object may be excluded from search results. Although this disclosure describes implementing privacy settings in a particular manner, this disclosure contemplates implementing privacy settings in any suitable manner.

[0206] In certain embodiments, different objects of the same type associated with a user may have different privacy settings. Different types of objects associated with a user may have different types of privacy settings. As an example, and not a limitation, a first user may specify that the first user's status updates are public, but any images shared by the first user are only visible to the first user's friends on an online social network. As another example, and not a limitation, a user may specify different privacy settings for different types of entities (e.g., individual users, friends of friends, followers, user groups, or corporate entities). As another example, and not a limitation, a first user may specify a group of users who can view videos posted by the first user, while preventing the videos from being visible to the first user's employer. In certain embodiments, different privacy settings may be provided for different user groups or user demographics. As an example, and not a limitation, a first user may specify that other users attending the same university as the first user can view the first user's photos, but other users who are family members of the first user cannot view those same photos.

[0207] In a particular embodiment, the social networking system 160 may provide one or more default privacy settings for each object of a specific object type. The privacy settings of an object set as the default can be changed by the user associated with that object. By way of example and not limitation, all images posted by a first user may have a default privacy setting that is visible only to the first user's friends, and for a particular image, the first user may change the privacy settings of that image to be visible to friends and friends of friends.

[0208] In certain embodiments, privacy settings may allow a first user to specify (e.g., by opting out or not opting in) whether the social networking system 160 or assistant system 140 may receive, collect, record, or store specific objects or information associated with the user for any purpose. In certain embodiments, privacy settings may allow a first user to specify whether a particular application or process may access, store, or use specific objects or information associated with the user. Privacy settings may allow the first user to opt in or out, allowing objects or information to be accessed, stored, or used by a particular application or process. The social networking system 160 or assistant system 140 may access such information to provide specific functionality or services to the first user, but the social networking system 160 or assistant system 140 may not access the information for any other purpose. Before accessing, storing, or using such objects or information, the social networking system 160 or assistant system 140 may prompt the user to provide privacy settings that specify which applications or processes (if any) may access, store, or use the objects or information before allowing any such action. As an example, and not as a limitation, a first user may transmit messages to a second user via an application associated with an online social network (e.g., a messaging app), and may specify privacy settings that the social network system 160 or the assistant system 140 should not store such messages.

[0209] In certain embodiments, a user may specify whether the social networking system 160 or the assistant system 140 can access, store, or use a specific type of object or information associated with a first user. As an example, and not a limitation, the first user may specify that images sent by the first user through the social networking system 160 or the assistant system 140 cannot be stored by the social networking system 160 or the assistant system 140. As another example, and not a limitation, the first user may specify that messages sent from the first user to a specific second user cannot be stored by the social networking system 160 or the assistant system 140. As yet another example, and not a limitation, the first user may specify that all objects sent via a specific application can be saved by the social networking system 160 or the assistant system 140.

[0210] In certain embodiments, privacy settings may allow a first user to specify whether specific objects or information associated with the first user can be accessed from a specific client system 130 or a third-party system 170. Privacy settings may allow the first user to opt in or out of accessing objects or information from a specific device (e.g., the user's phonebook on their smartphone), a specific application (e.g., a messaging app), or a specific system (e.g., an email server). The social networking system 160 or assistant system 140 may provide default privacy settings for each device, system, or application, and / or may prompt the first user to specify specific privacy settings for each context. As an example, and not a limitation, the first user may utilize the location service features of the social networking system 160 or assistant system 140 to provide recommendations for restaurants or other places near the user. The first user's default privacy settings may specify that the social networking system 160 or assistant system 140 may use location information provided from the first user's client device 130 to provide location-based services, but the social networking system 160 or assistant system 140 may not store the first user's location information or provide it to any third-party system 170. The first user can then update their privacy settings to allow third-party image-sharing apps to use location information to geotag photos.

[0211] In certain embodiments, privacy settings may allow a user to specify one or more geographic locations from which they can access an object. Access to or denial of access to an object may depend on the geographic location of the user attempting to access the object. As an example, and not a limitation, a user may share an object and specify that only users in the same city can access or view the object. As another example, and not a limitation, a first user may share an object and specify that the object is only visible to a second user when the first user is in a specific location. If the first user leaves that specific location, the object may no longer be visible to the second user. As another example, and not a limitation, a first user may specify that the object is only visible to second users within a threshold distance of the first user. If the first user subsequently changes location, the second user who originally had access to the object may lose access, and a new group of second users may gain access when they reach within the threshold distance of the first user.

[0212] In a particular embodiment, the social networking system 160 or assistant system 140 may have the capability to use a user's personal or biometric information as input for user authentication or experience personalization purposes. Users may choose to utilize these capabilities to enhance their experience on the online social network. By way of example, and not limitation, a user may provide personal or biometric information to the social networking system 160 or assistant system 140. A user's privacy settings may specify that such information is only available for specific processes (such as authentication) and that such information cannot be shared with any third-party system 170 or used for other processes or applications associated with the social networking system 160 or assistant system 140. By way of another example, and not limitation, the social networking system 160 may provide a user with the capability to provide a voiceprint record to the online social network. By way of example, and not limitation, if a user wishes to utilize this capability of the online social network, the user may provide a voice recording of their own voice to provide status updates on the online social network. The voice input record can be compared with the user's voiceprint to determine what words the user spoke. A user's privacy settings can specify that such voice recordings can only be used for voice input purposes (e.g., authenticating users, sending voice messages, improving voice recognition for using voice operation features on online social networks), and also specify that such voice recordings cannot be shared with any third-party system 170, or used by other processes or applications associated with the social network system 160. As another example, and not as a limitation, the social network system 160 can provide users with the ability to provide reference images (e.g., facial contours, retinal scans) to the online social network. The online social network can compare the reference image with later received image input (e.g., for user authentication, tagging users in photos). The user's privacy settings can specify that such images can only be used for limited purposes (e.g., authentication, tagging users in photos), and also specify that such images cannot be shared with any third-party system 170, or used by other processes or applications associated with the social network system 160.

[0213] Systems and Methods Figure 22 An example computer system 2200 is illustrated. In a particular embodiment, one or more computer systems 2200 perform one or more steps of one or more methods described or illustrated herein. In a particular embodiment, one or more computer systems 2200 provide the functionality described or illustrated herein. In a particular embodiment, software running on one or more computer systems 2200 performs one or more steps of one or more methods described or illustrated herein, or provides the functionality described or illustrated herein. Specific embodiments include one or more portions of one or more computer systems 2200. Throughout this document, references to computer systems may include computing devices and vice versa, where appropriate. Furthermore, references to computer systems may include one or more computer systems, where appropriate.

[0214] This disclosure contemplates any suitable number of computer systems 2200. The computer systems 2200 are contemplated to take any suitable physical form. By way of example and not limitation, the computer system 2200 may be an embedded computer system, a system-on-a-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system mesh, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, the computer system 2200 may include one or more computer systems 2200; may be monolithic or distributed; may span multiple locations; may span multiple machines; may span multiple data centers; or may reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 2200 may perform one or more steps of the methods described or shown herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 2200 may execute one or more steps of the methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 2200 may execute one or more steps of the methods described or illustrated herein at different times or at different locations.

[0215] In a particular embodiment, computer system 2200 includes a processor 2202, a memory 2204, a storage device 2206, an input / output (I / O) interface 2208, a communication interface 2210, and a bus 2212. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0216] In a particular embodiment, processor 2202 includes hardware for executing instructions, such as those constituting a computer program. By way of example, and not limitation, to execute instructions, processor 2202 may retrieve (or fetch) instructions from internal registers, internal caches, memory 2204, or storage device 2206; decode and execute them; and then write one or more results to internal registers, internal caches, memory 2204, or storage device 2206. In a particular embodiment, processor 2202 may include one or more internal caches for data, instructions, or addresses. Where appropriate, this disclosure contemplates that processor 2202 may include any suitable number of suitable internal caches. By way of example, and not limitation, processor 2202 may include one or more instruction caches, one or more data caches, and one or more translation lookup buffers (TLBs). Instructions in the instruction cache may be copies of instructions in memory 2204 or storage device 2206, and the instruction cache may accelerate the retrieval of those instructions by processor 2202. The data in the data cache may be: a copy of data in memory 2204 or storage device 2206 for operating instructions executed at processor 2202; the result of a previous instruction executed at processor 2202 for access by a subsequent instruction executed at processor 2202 or for writing to memory 2204 or storage device 2206; or other suitable data. The data cache can accelerate read or write operations performed by processor 2202. The TLB can accelerate virtual address translation with respect to processor 2202. In a particular embodiment, processor 2202 may include one or more internal registers for data, instructions, or addresses. Where appropriate, this disclosure contemplates that processor 2202 may include any suitable number of suitable internal registers. Where appropriate, processor 2202 may include one or more arithmetic logic units (ALUs); is a multi-core processor; or includes one or more processors 2202. Although this disclosure describes and illustrates particular processors, this disclosure contemplates any suitable processor.

[0217] In a particular embodiment, memory 2204 includes main memory for storing instructions for causing processor 2202 to execute or data for causing processor 2202 to operate. By way of example and not limitation, computer system 2200 may load instructions from storage device 2206 or another source (e.g., another computer system 2200) into memory 2204. Processor 2202 may then load instructions from memory 2204 into internal registers or internal caches. To execute instructions, processor 2202 may retrieve instructions from internal registers or internal caches and decode them. During or after the execution of instructions, processor 2202 may write one or more results (which may be intermediate or final results) to internal registers or internal caches. Processor 2202 may then write one or more of these results to memory 2204. In a particular embodiment, processor 2202 executes only instructions in one or more internal registers or internal caches or in memory 2204 (rather than memory device 2206 or elsewhere), and operates only on data in one or more internal registers or internal caches or in memory 2204 (rather than memory device 2206 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) couple processor 2202 to memory 2204. As described below, bus 2212 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) reside between processor 2202 and memory 2204 and facilitate access to memory 2204 requested by processor 2202. In a particular embodiment, memory 2204 includes random access memory (RAM). Where appropriate, the RAM may be volatile memory. Where appropriate, the RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Furthermore, where appropriate, the RAM may be single-port RAM or multi-port RAM. This disclosure contemplates any suitable RAM. Where appropriate, memory 2204 may include one or more memories 2204. Although this disclosure describes and illustrates specific memories, this disclosure contemplates any suitable memories.

[0218] In a particular embodiment, storage device 2206 includes a mass storage device for data or instructions. By way of example and not limitation, storage device 2206 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk drive, magneto-optical disk drive, magnetic tape drive, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, storage device 2206 may include removable or non-removable (or fixed) media. Where appropriate, storage device 2206 may be internal or external to computer system 2200. In a particular embodiment, storage device 2206 is a non-volatile solid-state memory. In a particular embodiment, storage device 2206 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM), or flash memory, or a combination of two or more of these. This disclosure contemplates a mass storage device 2206 employing any suitable physical form. Where appropriate, storage device 2206 may include one or more storage device control units that facilitate communication between processor 2202 and storage device 2206. Where appropriate, storage device 2206 may include one or more storage devices. Although this disclosure describes and illustrates specific storage devices, any suitable storage device is contemplated.

[0219] In a particular embodiment, I / O interface 2208 includes hardware, software, or both that provide one or more interfaces for communication between computer system 2200 and one or more I / O devices. Where appropriate, computer system 2200 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and computer system 2200. By way of example and not limitation, I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet computer, touchscreen, trackball, video camera, another suitable I / O device, or a combination of two or more of these. I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interface 2208 for them. Where appropriate, I / O interface 2208 may include one or more device or software drivers that enable processor 2202 to drive one or more of these I / O devices. Where appropriate, I / O interface 2208 may include one or more I / O interfaces 2208. Although this disclosure describes and illustrates specific I / O interfaces, this disclosure contemplates any suitable I / O interface.

[0220] In a particular embodiment, communication interface 2210 includes hardware, software, or both providing one or more interfaces for communication (e.g., packet-based communication) between computer system 2200 and one or more other computer systems 2200 or one or more networks. By way of example and not limitation, communication interface 2210 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired-based networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks (e.g., Wi-Fi networks). This disclosure contemplates any suitable network and any suitable communication interface 2210 for it. By way of example and not limitation, computer system 2200 may communicate with one or more portions of an ad hoc network, personal area network (PAN), local area network (LAN), wide area network (WAN), metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more portions of these networks may be wired or wireless. As an example, computer system 2200 may communicate with a wireless PAN (WPAN) (e.g., Bluetooth WPAN), a Wi-Fi network, a Wi-Fi Max network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Where appropriate, computer system 2200 may include any suitable communication interface 2210 for any of these networks. Where appropriate, communication interface 2210 may include one or more communication interfaces 2210. Although specific communication interfaces are described and shown in this disclosure, any suitable communication interface is contemplated in this disclosure.

[0221] In a particular embodiment, bus 2212 includes hardware, software, or both, that couple components of computer system 2200 to each other. By way of example and not limitation, bus 2212 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or any other suitable bus, or a combination of two or more of these. Where appropriate, bus 2212 may include one or more buses 2212. Although this disclosure describes and illustrates specific buses, this disclosure contemplates any suitable bus or interconnect.

[0222] In this document, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0223] Other miscellaneous items In this document, unless otherwise expressly indicated or indicated by the context, "or" is inclusive rather than exclusive. Therefore, in this document, unless otherwise expressly indicated or indicated by the context, "A or B" means "A, B, or both." Furthermore, unless otherwise expressly indicated or indicated by the context, "and" is both joint and separate. Therefore, in this document, unless otherwise expressly indicated or indicated by the context, "A and B" means "A and B, jointly or separately."

[0224] The scope of this disclosure includes all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or illustrated herein that will be understood by those skilled in the art. The scope of this disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although this disclosure describes and illustrates corresponding embodiments herein as including specific components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or substitution of any component, element, feature, function, operation, or step described or illustrated anywhere herein that will be understood by those skilled in the art. Furthermore, references in the appended claims to a means or system or component of a means or system suitable for, arranged to, capable of, configured to, implemented to, operable to, or operated to perform a particular function include that means, system, or component, whether or not it or that particular function is activated, turned on, or unlocked, provided that the means, system, or component is so adapted, arranged, enabled, configured, implemented, operable, or operated. Furthermore, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide some, all, or none of these advantages.

[0225] This disclosure may be further understood by referring to the following terms: Clause 1. A method comprising one or more computing systems: The assistant xbot receives user input, including natural language utterances, from the client system associated with the user; The semantic representation of the user input is determined based on the structural ontology, wherein the structural ontology defines a markup syntax for parsing the natural language discourse into semantic units including multiple actions, objects and attributes, and wherein the semantic representation embeds at least one object into at least one action and declares at least one attribute of the embedded object that will be subject to an action. Send a request to the agent to perform a task corresponding to the user input, wherein the request is based on the semantic representation; Receive the results of the executed task from the agent, wherein the results are mapped to a structure determined by the structure ontology; and The assistant xbot sends instructions to the client system to present a response to the user input, wherein the response is based on the result of a task being performed.

[0226] Clause 2. The method described in Clause 1, wherein the structural ontology definition Each of the plurality of actions operates on one or more of the plurality of objects; Each of the plurality of objects includes one or more of the plurality of attributes; or Each of the multiple attributes states a restriction on the action or object.

[0227] Clause 3. The method described in Clause 1, wherein the semantic representation is capable of being executed by each module associated with the assistant xbot configured to process the user input.

[0228] Clause 4. The method described in Clause 1, wherein the structural ontology defines the plurality of objects as being hierarchically organized into a plurality of supertypes and subtypes, each subtype inheriting one or more properties of its associated supertype.

[0229] Clause 5. The method according to Clause 1, wherein the value associated with each of the at least one action, the at least one object, and the at least one attribute is determined by each module that performs the semantic representation.

[0230] Clause 6. The method described in Clause 1 further includes: The response is generated by the natural language generation module associated with the assistant xbot based on the structural ontology and the results of the task performed.

[0231] Clause 7. According to the method described in Clause 1, the semantic unit defined by the structural ontology further includes a plurality of methods and enumerations, wherein each of the plurality of methods includes an action whose lifetime is bound to an object, and wherein each of the plurality of enumerations includes an object representing one of the plurality of objects.

[0232] Clause 8. The method according to Clause 1, wherein the structural ontology further defines a graph structure comprising one or more core subgraphs and one or more general subgraphs, wherein the one or more core subgraphs are inaccessible by a third-party proxy, and wherein the one or more general subgraphs are accessible by the third-party proxy.

[0233] Clause 9. The method described pursuant to Clause 1 further includes: The reasoning module associated with the assistant xbot resolves one or more entities associated with the at least one object based on at least one attribute of the semantic representation.

[0234] Clause 10. The method described in Clause 1, wherein the agent is a first-party agent associated with the assistant xbot.

[0235] Clause 11. The method according to Clause 1, wherein the agent is a third-party agent associated with an external computing system, and wherein the structure of the execution result includes one or more custom semantic units.

[0236] Clause 12. The method according to Clause 11, wherein sending the request to the agent and receiving the execution result from the agent are both via an application programming interface (API), wherein the API defines a mapping from the one or more custom semantic units to a structure determined by the structure ontology; The request sent to the agent includes the semantic representation modified by the dialogue manager associated with the assistant xbot, wherein the execution result includes the semantic representation modified by the agent, and wherein the request and the execution result use the markup syntax of the structure ontology, or Each of the semantic units is associated with a default namespace defined by the computing system, and one or more of the custom semantic units are associated with a personalized namespace defined by the third-party agent.

[0237] Clause 13. The method according to Clause 1, wherein the at least one action has one or more parameters, wherein each parameter specifies a constraint to be satisfied by the object, such that the at least one action can be applied to the object, and wherein at least one of the parameters specifies a constraint to be satisfied by the at least one object.

[0238] Clause 14. One or more computer-readable non-transitory storage media containing software that, when executed, is operable to perform the method described in any one of Clauses 1 to 13 or the following operations: The assistant xbot receives user input, including natural language utterances, from the client system associated with the user; The semantic representation of the user input is determined based on the structural ontology, wherein the structural ontology defines a markup syntax for parsing the natural language discourse into semantic units including multiple actions, objects and attributes, and wherein the semantic representation embeds at least one object into at least one action and declares at least one attribute of the embedded object that will be subject to an action. Send a request to the agent to perform a task corresponding to the user input, wherein the request is based on the semantic representation; Receive the results of the executed task from the agent, wherein the results are mapped to a structure determined by the structure ontology; and The assistant xbot sends instructions to the client system to present a response to the user input, wherein the response is based on the result of a task being performed.

[0239] Clause 15. A system comprising: one or more processors; and a non-transitory memory coupled to said processor, comprising instructions executable by said processor, said processor being operable, when executing said instructions, to perform the method of any one of Clauses 1 to 13 or the following operations: The assistant xbot receives user input, including natural language utterances, from the client system associated with the user; The semantic representation of the user input is determined based on the structural ontology, wherein the structural ontology defines a markup syntax for parsing the natural language discourse into semantic units including multiple actions, objects and attributes, and wherein the semantic representation embeds at least one object into at least one action and declares at least one attribute of the embedded object that will be subject to an action. Send a request to the agent to perform a task corresponding to the user input, wherein the request is based on the semantic representation; Receive the results of the executed task from the agent, wherein the results are mapped to a structure determined by the structure ontology; and The assistant xbot sends instructions to the client system to present a response to the user input, wherein the response is based on the result of a task being performed.< / attributename> < / attributename> < / attributename> < / attributename> < / attributename> < / attributename> < / attributename> < / attributeinitialization> < / attributename> < / attributeinitialization> < / attributename> < / attributename> < / object> < / object> < / attributename> < / attributename> < / attributename> < / action> < / attributename> < / object> < / attributename> < / t> < / attributename> < / attributename> < / attributename> < / methodname> < / attributename> < / methodname> < / attributename>

Claims

1. A method comprising one or more computing systems: The assistant system receives user input, including natural language utterances, from the client system associated with the user; The semantic representation of the user input is determined at least in part based on a structural ontology, wherein the structural ontology defines relationships between multiple actions, multiple objects, and multiple attributes; Send a request to the agent to perform a task corresponding to the user input, wherein the request is at least partially based on the semantic representation; Receive the results of the tasks performed from the agent, wherein the execution results are associated with the structure ontology; and The assistant system sends instructions to the client system to present a response to the user input, wherein the response is based on the result of a task being performed.

2. The method according to claim 1, wherein the structure ontology definition Each of the plurality of actions operates on one or more of the plurality of objects; Each of the plurality of objects includes one or more of the plurality of attributes; or Each of the multiple attributes states a restriction on the action or object.

3. The method of claim 1, wherein the semantic representation can be performed by each module associated with the assistant system configured to process the user input.

4. The method of claim 1, wherein the structural ontology defines the plurality of objects as hierarchically organized into a plurality of supertypes and subtypes, each subtype inheriting one or more properties of its associated supertype.

5. The method of claim 1, wherein the value associated with each of at least one action, at least one object, and at least one attribute is determined by each module performing the semantic representation.

6. The method according to claim 1, further comprising: The response is generated by a natural language generation module associated with the assistant system based on the structural ontology and the results of the task performed.

7. The method according to claim 1, wherein the structural ontology defining the semantic unit further includes multiple methods and multiple enumerations; Each of the plurality of methods includes an action whose lifetime is bound to the object; and Each of the plurality of enumerations includes an object representing one of the plurality of objects.

8. The method according to claim 1, further comprising: The reasoning module associated with the assistant system resolves one or more entities associated with at least one object based on at least one attribute of the semantic representation.

9. The method of claim 1, wherein the agent is a first-party agent associated with the assistant system.

10. The method of claim 1, wherein the agent is a third-party agent associated with an external computing system, and wherein the structure of the execution result includes one or more custom semantic units.

11. The method of claim 10, wherein sending the request to the agent and receiving the execution result from the agent are both via an application programming interface (API); The API definition therein is a mapping from the one or more custom semantic units to a structure determined by the structure ontology.

12. The method of claim 10, wherein the request sent to the agent includes the semantic representation modified by a dialogue manager associated with the assistant system; The execution result includes the semantic representation modified by the agent; and The request and the execution result are stated using the markup syntax of the structure ontology, or Each of the one or more custom semantic units is associated with a default namespace defined by the computing system, and each of the one or more custom semantic units is associated with a personalized namespace defined by the third-party agent.

13. The method of claim 1, wherein at least one action has one or more parameters, wherein each parameter specifies a constraint to be satisfied by the object, such that the at least one action can be applied to the object, and wherein at least one of the parameters specifies at least one constraint to be satisfied by the object.

14. One or more computer-readable non-transitory storage media comprising software that, when executed, is operable to perform the method of any one of claims 1 to 13 or the following operations: The assistant system receives user input, including natural language utterances, from the client system associated with the user; The semantic representation of the user input is determined at least in part based on a structural ontology, wherein the structural ontology defines relationships between multiple actions, multiple objects, and multiple attributes; Send a request to the agent to perform a task corresponding to the user input, wherein the request is at least partially based on the semantic representation; Receive the results of the tasks performed from the agent, wherein the results are associated with the structure ontology; and The assistant system sends instructions to the client system to present a response to the user input, wherein the response is based on the result of a task being performed.

15. A system comprising: One or more processors; and a non-transitory memory coupled to the processor, comprising instructions executable by the processor, wherein the processor, when executing the instructions, is operable to perform the method of any one of claims 1 to 13 or the following operations: The assistant system receives user input, including natural language utterances, from the client system associated with the user; The semantic representation of the user input is determined at least in part based on a structural ontology, wherein the structural ontology defines relationships between multiple actions, multiple objects, and multiple attributes; Send a request to the agent to perform a task corresponding to the user input, wherein the request is at least partially based on the semantic representation; Receive the results of the tasks performed from the agent, wherein the results are associated with the structure ontology; and The assistant system sends instructions to the client system to present a response to the user input, wherein the response is based on the result of a task being performed.

Images