Contextual autocompletion for assistant system

Abstract

In one embodiment, a method includes receiving user input including a partial request from a client system of a first user, analyzing the user input based on a personalized language model to generate one or more candidate hypotheses, wherein each candidate hypothesis includes one or more of intent suggestions or slot suggestions, sending an instruction to the client system for presenting one or more suggested auto-completions corresponding to the one or more candidate hypotheses, respectively, wherein each suggested auto-completion includes the partial request and the corresponding candidate hypothesis, receiving from the client system an indication of a selection of a first suggested auto-completion from the suggested auto-completions by the first user, and performing one or more tasks via one or more agents based on the first suggested auto-completion selected by the first user.

Description

Technical Field

[0001] The present disclosure relates generally to database and file management within a network environment, and particularly to hardware and software for a smart assistant system.

[0002] background

[0003] The assistant system can provide information or services on behalf of the user based on a combination of user input, location awareness, and the ability to access information from various online sources (such as weather conditions, traffic congestion, news, stock prices, user schedules, retail prices, etc.). User input can include text (e.g., online chat) (especially text in instant messaging applications or other applications), voice, images, or a combination thereof. The assistant system can perform concierge-type services (e.g., making dinner reservations, purchasing event tickets, arranging travel) or provide information based on user input. The assistant system can also perform management or data processing tasks based on online information and events without user initiation or interaction. Examples of those tasks that can be performed by the assistant system can include schedule management (e.g., sending a warning message to a dinner appointment that the user is going to be late due to traffic conditions, updating the schedules of both parties, and changing restaurant reservation times). The assistant system can be implemented through a combination of computing devices, application programming interfaces (APIs), and application proliferation on user devices.

[0004] A social networking system, which may include a social networking website, may enable its users (e.g., individuals or organizations) to interact with it and with each other through it. The social networking system may utilize input from a user to create and store a user profile associated with the user in the social networking system. The user profile may include demographic information, communication channel information, and information about personal interests of the user. The social networking system may also utilize input from the user to create and store a record of the user's relationships with other users of the social networking system, as well as provide services (e.g., profile / news feed postings, photo sharing, event organization, messaging, games, or advertising) to facilitate social interaction between or among users.

[0005] The social networking system may send content or messages related to its services to the user's mobile device or other computing device through one or more networks. The user may also install a software application on the user's mobile device or other computing device for accessing the user's user profile and other data within the social networking system. The social networking system may generate a set of personalized content objects to display to the user, such as a dynamic message of aggregated stories of other users connected to the user.

[0006] Overview of Certain Embodiments

[0007] In a particular embodiment, the assistant system can help users obtain information or services. The assistant system can enable users to interact with it through multimodal user input (such as sound, text, image, video) in stateful and multi-turn conversations to obtain help. The assistant system can create and store a user profile, which includes personal information and contextual information associated with the user. In a particular embodiment, the assistant system can use natural language understanding to analyze user input. The analysis can be based on the user profile to obtain a more personalized and context-aware understanding. The assistant system can resolve entities associated with user input based on the analysis. In a particular embodiment, the assistant system can interact with different agents to obtain information or services associated with the resolved entity. The assistant system can generate responses about information or services for users by using natural language generation. Through interaction with the user, the assistant system can use dialogue management technology to manage and forward the conversation flow with the user. In a particular embodiment, the assistant system can also assist users to digest the information obtained effectively and efficiently by summarizing the information. The assistant system can also help users better participate in online social networks by providing tools that help users interact with the online social network (e.g., create posts, comments, messages). The assistant system can additionally help users manage different tasks, such as keeping track of events. In certain embodiments, the assistant system can proactively perform tasks related to the user's interests and preferences based on the user profile without user input. In certain embodiments, the assistant system can check privacy settings to ensure that access to the user's profile or other user information and the performance of different tasks are allowed according to the user's privacy settings.

[0008] In a particular embodiment, the assistant system can suggest context-related, typeahead-like auto-completion to the user. The assistant system can receive user input in various modes, including audio, text, video, images, etc. In a public environment, some modes may be inconvenient and / or unsuitable for use (e.g., audio and video), and users may prefer to input text input into the assistant system to protect their privacy. One challenge of text input may be that keyboard entry is slower than the audio / voice input of a typical user. Accordingly, methods to increase the speed of keyboard entry will help improve the interaction between the user and the assistant system. In a particular embodiment, the assistant system can generate a suggested auto-completion using a personalized language model that predicts the next keyboard entry (e.g., character, word, phrase, sentence, etc.), which helps users complete their entries faster and more effortlessly. As an example and not as a limitation, if the user is interacting with the assistant system via a messaging interface and has typed "call...", the assistant system can determine that the input corresponds to the intent [IN: call (person)]. The personalized language model can then predict that the next keyboard entry is a person's name to fill the slot [SL: person (name)]. Entries for suggested auto-completion can be stored in a range trie, which indexes entries to allow efficient lookup for a given prefix. Accordingly, the personalized language model can select a list of entries for suggested auto-completion from the range trie. The user can also select suggested auto-completion, thereby reducing the number of keystrokes the user needs to enter to complete a request that can be executed by the assistant system. Although the present disclosure describes suggesting a particular auto-completion via a particular system in a particular manner, the present disclosure contemplates suggesting any suitable auto-completion via any suitable system in any suitable manner.

[0009] In a particular embodiment, the assistant system may receive user input from a first user from a client system associated with the first user. The user input may include a partial request. In a particular embodiment, the assistant system may analyze the user input based on a personalized language model to generate one or more candidate hypotheses corresponding to the partial request. Each of the one or more candidate hypotheses may include one or more of an intent suggestion or a slot suggestion. Each of the one or more candidate hypotheses may additionally correspond to a subsequent entry associated with the user input. In a particular embodiment, the assistant system may send an instruction to the client system for presenting one or more suggested auto-completions corresponding to the one or more candidate hypotheses, respectively. Each suggested auto-completion may include a partial request and a corresponding candidate hypothesis. In a particular embodiment, the assistant system may receive from the client system an indication of the selection of a first suggested auto-completion in one or more suggested auto-completions by the first user. The assistant system may also perform one or more tasks via one or more agents based on the first suggested auto-completion selected by the first user.

[0010] In order to achieve the goal of the assistant system suggesting the correct automatic completion in response to the partial request in the user input, there may be certain technical challenges. One technical challenge may include accurately determining the candidate hypothesis based on the partial request. The solution to the above challenge proposed by the embodiment disclosed herein is to use a personalized language model based on a recurrent neural network, and train the personalized language model based on various training data associated with the user. The personalized language model is distinguished when determining the candidate hypothesis because various information about the user is learned from the training data, resulting in a comprehensive understanding of the partial request. Another technical challenge may include presenting the most relevant candidate hypothesis to the user. The solution to this challenge proposed by the embodiment disclosed herein includes sorting the candidate hypotheses based on the dialogue state and the confidence score determined by the personalized language model, and dynamically updating the confidence score based on additional user input, which generates a sorted list of candidate hypotheses that more accurately reflects the user's intention in the current dialogue session. Another technical challenge may include generating candidate hypotheses in real time. The solution to this challenge proposed by the embodiment disclosed herein includes storing the candidate hypotheses in a range dictionary tree, because the assistant system can quickly find the candidate hypothesis in the range dictionary tree. Another technical challenge may include the sparsity of data associated with a single user. The solution proposed by the embodiments disclosed herein to address this challenge includes using a global language model trained based on data associated with multiple users, because these data are sufficient to learn a discriminative language model, which can improve the generation of candidate hypotheses by simply personalizing the language model.

[0011] Certain embodiments disclosed herein may provide one or more technical advantages. A technical advantage of an embodiment may include improving the user's interaction with the assistant system by helping the user complete their typing faster and with less effort. Another technical advantage of an embodiment may include guiding the user to tasks that the assistant system can perform (e.g., if the user is unaware of the capabilities of the assistant system) to improve the user experience of the assistant system. Certain embodiments disclosed herein may provide none, some, or all of the above technical advantages. In view of the drawings, description, and claims of the present disclosure, one or more other technical advantages may be apparent to those skilled in the art.

[0012] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to them. Specific embodiments may include all, some or none of the components, elements, features, functions, operations or steps of the embodiments disclosed herein. Embodiments according to the present invention are specifically disclosed in the attached claims relating to methods, storage media, systems, assistant systems and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or back references in the attached claims are selected only for formal reasons. However, any subject matter generated by intentional back references (especially multiple references) to any previous claim may also be claimed, so that any combination of claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features as set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any features of the appended claims.

[0013] In an embodiment, a method, particularly for use in an assistant system, for assisting a user in obtaining information or services by enabling the user to interact with the assistant system in a conversation using user input to obtain assistance, wherein the user input includes sound, text, image, or video, or any combination thereof, the assistant system being particularly implemented by a combination of a computing device, an application programming interface (API), and application proliferation on a user device, may include, by one or more computing systems:

[0014] receiving, from a client system associated with the first user, user input from the first user, wherein the user input comprises a portion of the request;

[0015] Analyzing user input based on the personalized language model to generate one or more candidate hypotheses corresponding to the partial request, wherein each of the one or more candidate hypotheses includes one or more of intent suggestions or slot suggestions;

[0016] sending instructions to the client system for presenting one or more suggested auto-completions corresponding to the one or more candidate hypotheses, respectively, wherein each suggested auto-completion comprises a partial request and a corresponding candidate hypothesis;

[0017] receiving, from the client system, an indication of a selection by the first user of a first suggested auto-completion of the one or more suggested auto-completions; and

[0018] Based on the auto-completion of the first suggestion selected by the first user, one or more tasks are performed via the one or more agents.

[0019] Analyzing the user input to generate one or more candidate hypotheses corresponding to the partial request may include:

[0020] The user input is analyzed based on the personalized language model to determine one or more candidate intents.

[0021] In an embodiment, a method may include:

[0022] sending one or more intent suggestions corresponding to the one or more candidate intents to the client system; and

[0023] A selection of one of the one or more intent suggestions by a first user is received from the client system, wherein the selected intent suggestion is provided as an intent suggestion of one of the candidate hypotheses.

[0024] In an embodiment, a method may include:

[0025] sending a request to the client system for additional information from the first user;

[0026] receiving additional user input from the first user in response to the request from the client system; and

[0027] Based on the additional user input, one or more candidate intents are disambiguated to determine a top candidate intent that is provided as an intent suggestion as one of the candidate hypotheses.

[0028] Analyzing the user input to generate one or more candidate hypotheses corresponding to the partial request may include:

[0029] Based on the personalized language model, the user input is analyzed to determine one or more candidate slots.

[0030] In an embodiment, a method may include:

[0031] sending one or more slot suggestions corresponding to the one or more possible slots to the client system; and

[0032] A selection of one of the one or more slot suggestions by a first user is received from the client system, wherein the selected slot suggestion is provided as a slot suggestion for one of the candidate hypotheses.

[0033] In an embodiment, a method may include:

[0034] sending a request to the client system for additional information about the first user;

[0035] receiving additional user input from the first user in response to the request from the client system; and

[0036] Based on the additional user input, one or more candidate slots are disambiguated to determine a top candidate slot that is provided as a slot suggestion for one of the candidate hypotheses.

[0037] The personalized language model may be trained based on a plurality of training data, the plurality of training data including one or more of the following:

[0038] News Feed posts associated with the first user;

[0039] News Feed comments associated with the first user;

[0040] messages in one or more messaging interfaces associated with the first user;

[0041] Data representing one or more domains;

[0042] a conversation state of one or more conversation sessions associated with the first user;

[0043] user profile data associated with the user;

[0044] A task status associated with one or more tasks.

[0045] The one or more candidate hypotheses may be ranked based on a dialog state of the dialog session associated with the user input.

[0046] The one or more candidate hypotheses may be respectively associated with one or more confidence scores, wherein the one or more confidence scores may be calculated by the personalized language model, and the one or more candidate hypotheses may be ranked based on their respective confidence scores.

[0047] In an embodiment, a method may include:

[0048] receiving additional user input from the client system, wherein the additional user input is appended to the user input;

[0049] updating one or more confidence scores for one or more candidate hypotheses based on the additional user input; and

[0050] One or more candidate hypotheses are re-ranked based on the updated confidence scores.

[0051] In an embodiment, a method may include:

[0052] A sliding window is applied to the user input, wherein a length of the sliding window determines a percentage of the user input to be used as a model input for the personalized language model.

[0053] In an embodiment, a method may include:

[0054] determining whether at least one of the one or more confidence scores associated with the one or more candidate hypotheses is less than a threshold score; and

[0055] After determining that at least one confidence score is less than a threshold score, the length of the sliding window is adjusted.

[0056] Analyzing the user input to generate one or more candidate hypotheses corresponding to the partial request may be based on one or more context-specific language models.

[0057] In an embodiment, a method may include:

[0058] Accessing, by the dialog engine, a dialog state of the dialog session associated with the user input;

[0059] selecting a particular context-specific language model from one or more context-specific language models based on the dialog state; and

[0060] One or more candidate hypotheses are generated based on the personalized language model and the selected context-specific model.

[0061] One or more context-specific language models may be trained based on context-specific data, the context-specific data comprising one or more of the following:

[0062] data associated with the presence of the first user at a particular location;

[0063] data associated with the first user's interaction with the particular user; or

[0064] Data associated with the first user's registration of a particular event.

[0065] Analyzing the user input to generate one or more candidate hypotheses corresponding to the partial request may be based on one or more global language models.

[0066] One or more global language models may be trained based on data associated with a plurality of users of an online social network.

[0067] Analyzing the user input to generate one or more candidate hypotheses corresponding to the partial request may be based on one or more global context-specific language models.

[0068] The personalized language model can be based on a recurrent neural network.

[0069] Each of the one or more candidate hypotheses may correspond to a subsequent entry associated with the user input.

[0070] In an embodiment, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to:

[0071] receiving, from a client system associated with the first user, user input from the first user, wherein the user input comprises a portion of the request;

[0072] Analyzing user input based on the personalized language model to generate one or more candidate hypotheses corresponding to the partial request, wherein each of the one or more candidate hypotheses includes one or more of intent suggestions or slot suggestions;

[0073] sending instructions to the client system for presenting one or more suggested auto-completions corresponding to the one or more candidate hypotheses, respectively, wherein each suggested auto-completion comprises a partial request and a corresponding candidate hypothesis;

[0074] receiving, from the client system, an indication of a selection by the first user of a first suggested auto-completion of the one or more suggested auto-completions; and

[0075] Based on the auto-completion of the first suggestion selected by the first user, one or more tasks are performed via the one or more agents.

[0076] In an embodiment, a system may include: one or more processors; and a non-transitory memory coupled to the processor, the non-transitory memory including processor-executable instructions, the processor being operable when executing the instructions to:

[0077] receiving, from a client system associated with the first user, user input from the first user, wherein the user input comprises a portion of the request;

[0078] Analyzing user input based on the personalized language model to generate one or more candidate hypotheses corresponding to the partial request, wherein each of the one or more candidate hypotheses includes one or more of intent suggestions or slot suggestions;

[0079] sending instructions to the client system for presenting one or more suggested auto-completions corresponding to the one or more candidate hypotheses, respectively, wherein each suggested auto-completion comprises a partial request and a corresponding candidate hypothesis;

[0080] receiving, from the client system, an indication of a selection by the first user of a first suggested auto-completion of the one or more suggested auto-completions; and

[0081] Based on the auto-completion of the first suggestion selected by the first user, one or more tasks are performed via the one or more agents.

[0082] In an embodiment, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to perform a method according to the present invention or any of the above-mentioned embodiments.

[0083] In an embodiment, a system may include: one or more processors; and at least one memory coupled to the processor and including instructions executable by the processor, the processor being operable to perform a method according to the present invention or any of the above-mentioned embodiments when executing the instructions.

[0084] In an embodiment, a computer program product, preferably comprising a computer readable non-transitory storage medium, is operable to perform a method according to the invention or any of the above mentioned embodiments when executed on a data processing system.

[0085] In an embodiment, an assistant system is used to help a user obtain information or services by enabling the user to interact with the assistant system using user input in a conversation to obtain help, wherein the user input includes sound, text, image or video or any combination thereof. The assistant system is specifically implemented through a combination of a computing device, an application programming interface (API), and a proliferation of applications on a user device. The system may include: one or more processors; and a non-volatile memory coupled to the processor, the non-volatile memory including instructions executable by the processor, and the processor is operable to perform a method according to the present invention or any of the above-mentioned embodiments when executing the instructions.

[0086] In an embodiment, the assistant system may assist the user by performing at least one or more of the following features or steps:

[0087] - Create and store a user profile that includes personal and contextual information associated with the user

[0088] - Use natural language understanding to analyze user input, where the analysis can be based on user profiles for a more personalized and context-aware understanding

[0089] - Resolve entities associated with user input based on analysis

[0090] -Interact with different agents to obtain information or services associated with the resolved entity

[0091] - Generate responses for users regarding information or services by using natural language generation

[0092] - Manage and forward conversation flows with users through interaction with them using conversation management techniques

[0093] - Help users digest the information they have obtained effectively and efficiently by aggregating the information

[0094] - Help users better participate in online social networks by providing tools that help users interact with online social networks (e.g., create posts, comments, messages)

[0095] - Helps users manage different tasks, such as keeping track of events

[0096] - Proactively perform pre-authorized tasks related to the user's interests and preferences based on the user profile at a time relevant to the user and without user input

[0097] - Checking privacy settings whenever necessary to ensure that accessing a user profile and performing different tasks adheres to the user's privacy settings.

[0098] In an embodiment, the assistant system may include at least one or more of the following components:

[0099] a messaging platform for receiving a text-based modality user input and / or receiving an image or video-based modality user input from a client system associated with the user and processing the image or video-based modality user input within the messaging platform using optical character recognition technology to convert the user input into text,

[0100] an audio speech recognition (ASR) module for receiving audio modality-based user input from a client system associated with the user (e.g., the user may speak to it or send a video including voice) and converting the audio modality-based user input into text,

[0101] - Assistant xbot, which is used to receive the output of the messaging platform or the ASR module.

[0102] In an embodiment, a system may include:

[0103] at least one client system (130), in particular an electronic device,

[0104] At least one assistant system (140) according to the present invention or any embodiment herein,

[0105] The client system and the assistant system are connected to each other in particular via a network (110),

[0106] wherein the client system includes an assistant application (136) for allowing a user of the client system (130) to interact with the assistant system (140),

[0107] wherein the assistant application (136) transmits the user input to the assistant system (140), and based on the user input, the assistant system (140) generates a response and sends the generated response to the assistant application (136), and the assistant application (136) presents the response to the user of the client system (130),

[0108] In particular, the user input is audio or verbal, and the response may be textual or also audio or verbal.

[0109] In an embodiment, a system may include a social networking system (160),

[0110] The client system particularly includes a social networking application (134) for accessing a social networking system (160). BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 An example network environment associated with an assistant system is shown.

[0113] Figure 2 An example architecture for an assistant system is shown.

[0114] Figure 3 An example flow chart of an assistant system responding to a user request is shown.

[0115] Figure 4 Shown based on Figure 2 An example flowchart of suggested auto-completion in the example architecture of the assistant system.

[0116] Figure 5A An example interaction with a user for suggested auto-completion in a messaging interface is shown.

[0117] Figure 5B An example interaction with a user for suggested auto-completions in a messaging interface is shown after the user selects a previously suggested auto-completion.

[0118] Figure 6 An example method for suggesting auto-completion is shown.

[0119] Figure 7 An example social graph is shown.

[0120] Figure 8 An example view of the embedding space is shown.

[0121] Fig. 9 An example artificial neural network is shown.

[0122] Fig.10 An example computer system is shown.

[0123] Description of Example Embodiments

[0124] Systematic review

[0125] Figure 1 An example network environment 100 associated with an assistant system is shown. The network environment 100 includes a client system 130, an assistant system 140, a social networking system 160, and a third-party system 170 connected to each other via a network 110. Figure 1 A particular arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110 is shown, but the present disclosure contemplates any suitable arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110. By way of example and not limitation, two or more of client system 130, social networking system 160, assistant system 140, and third-party system 170 may be directly connected to each other, bypassing network 110. As another example, two or more of client system 130, assistant system 140, social networking system 160, and third-party system 170 may be physically or logically co-located with each other, in whole or in part. Furthermore, although Figure 1 A particular number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110 are shown, but this disclosure contemplates any suitable number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110. By way of example and not limitation, network environment 100 may include a plurality of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110.

[0126] The present disclosure contemplates any suitable network 110. By way of example and not limitation, one or more portions of network 110 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 110 may include one or more networks 110.

[0127] Links 150 may connect client systems 130, assistant systems 140, social networking systems 160, and third-party systems 170 to communication network 110 or to each other. The present disclosure contemplates any suitable links 150. In certain embodiments, one or more links 150 include one or more wired (e.g., digital subscriber line (DSL) or cable-based data service interface specification (DOCSIS)) links, wireless (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or optical (e.g., synchronous optical network (SONET) or synchronous digital hierarchy (SDH)) links. In certain embodiments, one or more links 150 each include a self-organizing network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 150, or a combination of two or more such links 150. Links 150 need not necessarily be the same throughout network environment 100. One or more first links 150 may differ from one or more second links 150 in one or more aspects.

[0128] In certain embodiments, client system 130 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more such components, and is capable of performing appropriate functions implemented or supported by client system 130. By way of example and not limitation, client system 130 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a smart speaker, other suitable electronic devices, or any suitable combination thereof. In certain embodiments, client system 130 may be a smart assistant device. More information about smart assistant devices can be found in U.S. Patent Application No. 15 / 949,011, filed April 9, 2018, U.S. Patent Application No. 62 / 655,751, filed April 10, 2018, U.S. Design Patent Application No. 29 / 631,910, filed January 3, 2018, U.S. Design Patent Application No. 29 / 631,747, filed January 2, 2018, U.S. Design Patent Application No. 29 / 631,913, filed January 3, 2018, and U.S. Design Patent Application No. 29 / 631,914, filed January 3, 2018, each of which is incorporated by reference. The present disclosure contemplates any suitable client system 130. Client system 130 can enable network users at client system 130 to access network 110. Client system 130 can enable its users to communicate with other users at other client systems 130.

[0129] In a particular embodiment, the client system 130 may include a web browser 132, such as MICROSOFTINTERNET EXPLORER, GOOGLE CHROME or MOZILLA FIREFOX, and may have one or more add-ons, plug-ins or other extensions, such as TOOLBAR or YAHOO TOOLBAR. A user at the client system 130 may enter a uniform resource locator (URL) or other address that directs the web browser 132 to a specific server (such as server 162 or a server associated with a third-party system 170), and the web browser 132 may generate a hypertext transfer protocol (HTTP) request and pass the HTTP request to the server. The server may accept the HTTP request and pass one or more hypertext markup language (HTML) files to the client system 130 in response to the HTTP request. The client system 130 may display a web interface (such as a web page) based on an HTML file from a server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not as a limitation, a web interface may be displayed according to an HTML file, an extensible hypertext markup language (XHTML) file or an extensible markup language (XML) file according to specific needs. Such an interface may also execute scripts, such as, for example and without limitation, scripts written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, a combination of markup language and script (e.g., AJAX (Asynchronous JAVASCRIPT and XML)), etc. Herein, references to a web interface include one or more corresponding source files (which a browser may use to render the web interface), and vice versa, where appropriate.

[0130] In certain embodiments, client system 130 may include social networking application 134 installed on client system 130. A user at client system 130 may use social networking application 134 to access an online social network. A user at client system 130 may use social networking application 134 to communicate with the user's social connections (e.g., friends, followers, followed accounts, contacts, etc.). A user at client system 130 may also use social networking application 134 to interact with a plurality of content objects (e.g., posts, news articles, temporary content, etc.) on the online social network. As an example and not by way of limitation, a user may use social networking application 134 to browse trending topics and breaking news.

[0131] In certain embodiments, the client system 130 may include an assistant application 136. The user of the client system 130 may use the assistant application 136 to interact with the assistant system 140. In certain embodiments, the assistant application 136 may include an independent application. In certain embodiments, the assistant application 136 may be integrated into the social network application 134 or another suitable application (e.g., a messaging application). In certain embodiments, the assistant application 136 may also be integrated into the client system 130, the assistant hardware device, or any other suitable hardware device. In certain embodiments, the assistant application 136 may be accessed via a web browser 132. In certain embodiments, the user may provide input via different modalities. As an example and not as a limitation, the modality may include audio, text, image, video, etc. The assistant application 136 may transmit the user input to the assistant system 140. Based on the user input, the assistant system 140 may generate a response. The assistant system 140 may send the generated response to the assistant application 136. Then, the assistant application 136 may present the response to the user of the client system 130. The presented response may be based on different modalities, such as audio, text, image, and video. As an example and not by way of limitation, a user may verbally ask assistant application 136 for traffic information (i.e., via an audio modality). Assistant application 136 may then transmit the request to assistant system 140. Assistant system 140 may generate results accordingly and send them back to assistant application 136. Assistant application 136 may also present the results to the user in text.

[0132] In a particular embodiment, the assistant system 140 can help the user retrieve information from different sources. The assistant system 140 can also help the user request services from different service providers. In a particular embodiment, the assistant system 140 can receive a user's request for information or services via the assistant application 136 in the client system 130. The assistant system 140 can use natural language understanding to analyze the user request based on the user profile and other relevant information. The results of the analysis may include different entities associated with online social networks. The assistant system 140 can then retrieve information or request services associated with these entities. In a particular embodiment, when retrieving information or requesting services for the user, the assistant system 140 can interact with the social networking system 160 and / or the third-party system 170. In a particular embodiment, the assistant system 140 can generate personalized communication content for the user using natural language generation technology. The personalized communication content may include, for example, the status of the retrieved information or the requested service. In a particular embodiment, the assistant system 140 can enable the user to interact with it about information or services in a stateful and multi-round conversation by using dialogue management technology. In the following Figure 2 The functionality of assistant system 140 is described in more detail in the discussion of .

[0133] In certain embodiments, social networking system 160 may be a network-addressable computing system that may host an online social network. Social networking system 160 may generate, store, receive, and transmit social networking data (such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to an online social network). Social networking system 160 may be accessed by other components of network environment 100 directly or via network 110. As an example and not by way of limitation, client system 130 may access social networking system 160 directly or via network 110 using a web browser 132 or a native application associated with social networking system 160 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In certain embodiments, social networking system 160 may include one or more servers 162. Each server 162 may be a unitary server or a distributed server across multiple computers or multiple data centers. The server 162 may be of various types, such as, without limitation, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 162 may include a hardware, software, or embedded logic component, or a combination of two or more such components for performing the appropriate functions implemented or supported by the server 162. In a particular embodiment, the social networking system 160 may include one or more data storage devices 164. The data storage device 164 may be used to store various types of information. In a particular embodiment, the information stored in the data storage device 164 may be organized according to a particular data structure. In a particular embodiment, each data storage device 164 may be a relational database, a columnar database, a correlation database, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. Particular embodiments may provide an interface that enables client system 130 , social networking system 160 , or third-party system 170 to manage, retrieve, modify, add, or delete information stored in data store 164 .

[0134] In certain embodiments, social networking system 160 may store one or more social graphs in one or more data stores 164. In certain embodiments, a social graph may include a plurality of nodes, which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept), and a plurality of edges connecting the nodes. Social networking system 160 may provide users of an online social network with the ability to communicate and interact with other users. In certain embodiments, a user may join an online social network via social networking system 160 and then add connections (e.g., relationships) with a plurality of other users in social networking system 160 to whom they want to be connected. Herein, the term "friend" may refer to any other user of social networking system 160 with whom the user forms a connection, association, or relationship via social networking system 160.

[0135] In particular embodiments, social-networking system 160 may provide users with the ability to take actions on various types of items or objects supported by social-networking system 160. By way of example and not limitation, items and objects may include groups or social networks to which users of social-networking system 160 may belong, events or calendar entries that may be of interest to users, computer-based applications that users may use, transactions that allow users to buy or sell goods via a service, interactions with advertisements that users may perform, or other suitable items or objects. Users may interact with anything that can be represented in social-networking system 160 or by an external system such as third-party system 170 that is separate from social-networking system 160 and coupled to social-networking system 160 via network 110.

[0136] In certain embodiments, social networking system 160 can link various entities. As an example and not by way of limitation, social networking system 160 can enable users to interact with each other and receive content from third-party systems 170 or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0137] In certain embodiments, third-party systems 170 may include one or more types of servers, one or more data stores, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable components (e.g., the server may communicate with these components). Third-party systems 170 may be operated by a different entity than the entity that operates social-networking system 160. However, in certain embodiments, social-networking system 160 and third-party systems 170 may operate in conjunction with each other to provide social-networking services to users of social-networking system 160 or third-party systems 170. In this sense, social-networking system 160 may provide a platform or backbone that other systems (e.g., third-party systems 170) may use to provide social-networking services and functionality to users across the Internet.

[0138] In certain embodiments, third-party system 170 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be delivered to client system 130. For example, and not by way of limitation, a content object may include information about things or activities of interest to a user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example and not by way of limitation, a content object may include an incentive content object (e.g., a coupon, discount voucher, gift certificate, or other suitable incentive object).

[0139] In certain embodiments, social networking system 160 also includes user-generated content objects that can enhance a user's interaction with social networking system 160. User-generated content can include any content that a user can add, upload, send, or "post" to social networking system 160. As an example and not by way of limitation, a user transmits a post from client system 130 to social networking system 160. Posts can include data, such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to social networking system 160 by third parties through "communication channels" such as dynamic messages or streams.

[0140] In certain embodiments, the social networking system 160 may include various servers, subsystems, programs, modules, logs, and data storage. In certain embodiments, the social networking system 160 may include one or more of the following: a web server, an action recorder, an API request server, a correlation and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertisement-targeting module, a user interface module, a user profile storage, a relationship storage, a third-party content storage, or a location storage. The social networking system 160 may also include suitable components, such as a network interface, a security mechanism, a load balancer, a failover server, a management and network operation console, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 160 may include one or more user profile storages for storing user profiles. A user profile may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, affinities, or locations). Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not by way of limitation, if a user "likes" an article about a brand of shoes, the category may be the brand, or the general category of "shoes" or "clothing". The association storage may be used to store association information about users. The association information may indicate users with similar or common work experience, group membership, hobbies, educational history, or who are related or share common attributes in any way. The association information may also include user-defined associations between different users and content (internal and external). The web server may be used to link the social networking system 160 to one or more client systems 130 or one or more third-party systems 170 via the network 110. The web server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 160 and one or more client systems 130. The API request server may allow a third-party system 170 to access information from the social networking system 160 by calling one or more APIs. The action recorder may be used to receive communications from the web server about the user's actions on or outside the social networking system 160. In conjunction with the action log, a third-party content object log of user exposure to third-party content objects can be maintained. The notification controller can provide information about content objects to the client system 130. The information can be pushed to the client system 130 as a notification, or the information can be pulled from the client system 130 in response to a request received from the client system 130.The authorization server may be used to enforce one or more privacy settings of users of the social networking system 160. The privacy settings of the user determine how certain information associated with the user may be shared. The authorization server may allow the user to opt-in or opt-out of having their actions recorded by the social networking system 160 or shared with other systems (e.g., third-party systems 170), such as by setting appropriate privacy settings. A third-party content object store may be used to store content objects received from a third party (e.g., third-party system 170). A location store may be used to store location information associated with the user received from the client system 130. The advertising pricing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0141] Assistant System

[0142] Figure 2An example architecture of an assistant system 140 is shown. In a particular embodiment, the assistant system 140 can help users obtain information or services. The assistant system 140 can enable users to interact with it in a stateful and multi-turn conversation with multimodal user input (such as sound, text, image, video) to obtain help. The assistant system 140 can create and store a user profile including personal information and contextual information associated with the user. In a particular embodiment, the assistant system 140 can use natural language understanding to analyze user input. The analysis can be based on the user profile to obtain a more personalized and context-aware understanding. The assistant system 140 can parse entities associated with the user input based on the analysis. In a particular embodiment, the assistant system 140 can interact with different agents to obtain information or services associated with the parsed entity. The assistant system 140 can generate responses about information or services for the user by using natural language generation. Through interaction with the user, the assistant system 140 can use dialogue management technology to manage and forward the conversation flow with the user. In a particular embodiment, the assistant system 140 can also help users digest the information obtained effectively and efficiently by summarizing the information. The assistant system 140 can also help users better participate in online social networks by providing tools that help users interact with online social networks (e.g., create posts, comments, messages). The assistant system 140 can also help users manage different tasks, such as keeping track of events. In certain embodiments, the assistant system 140 can proactively perform pre-authorized tasks related to user interests and preferences based on the user profile at a time relevant to the user without user input. In certain embodiments, the assistant system 140 can check privacy settings to ensure that access to the user's profile or other user information and the performance of different tasks are allowed according to the user's privacy settings. More information about helping users according to privacy settings can be found in U.S. Patent Application No. 62 / 675,090, filed on May 22, 2018, which is incorporated by reference.

[0143] In certain embodiments, the assistant system 140 may receive user input from an assistant application 136 in a client system 130 associated with a user. In certain embodiments, the user input may be a user-generated input that is sent to the assistant system 140 in a single round. If the user input is based on a text modality, the assistant system 140 may receive it at the messaging platform 205. If the user input is based on an audio modality (e.g., the user may speak to the assistant application 136 or send a video including voice to the assistant application 136), the assistant system 140 may process it using an audio speech recognition (ASR) module 210 to convert the user input into text. If the user input is based on an image or video modality, the assistant system 140 may process it using optical character recognition technology within the messaging platform 205 to convert the user input into text. The output of the messaging platform 205 or the ASR module 210 may be received at the assistant xbot 215. More information on processing user input based on different modalities may be found in U.S. Patent Application No. 16 / 053600 filed on August 2, 2018, which is incorporated by reference.

[0144] In a particular embodiment, assistant xbot 215 can be a type of chatbot. Assistant xbot 215 can include a programmable service channel, which can be a software code, logic or routine used as a user personal assistant. Assistant xbot 215 can serve as a user portal of assistant system 140. Therefore, assistant xbot 215 can be considered as a type of conversational agent. In a particular embodiment, assistant xbot 215 can send text user input to natural language understanding (NLU) module 220 to interpret user input. In a particular embodiment, NLU module 220 can obtain information from user context engine 225 and semantic information aggregator 230 to accurately understand user input. User context engine 225 can store user profiles of users. User profiles of users can include user profile data, which includes demographic information, social information and context information associated with users. User profile data can also include user interests and preferences for multiple topics aggregated by dynamic messages, search logs, messaging platforms 205, etc. The use of user profiles may be protected by the privacy check module 245 to ensure that the user's information can only be used for his / her benefit and not shared with anyone else. More information about user profiles can be found in U.S. Patent Application No. 15 / 967,239 filed on April 30, 2018, which is incorporated by reference. The semantic information aggregator 230 may provide ontology data associated with a plurality of predefined domains, intents, and slots to the NLU module 220. In a particular embodiment, the domain may represent the social context of the interaction, e.g., education. The intent may be an element in a predefined classification of semantic intents, which may indicate the purpose of the user's interaction with the assistant system 140. In a particular embodiment, if the user input includes text / voice input, the intent may be an output of the NLU module 220. The NLU module 220 may classify the text / voice input as a member of a predefined classification, e.g., for the input "play Beethoven's Fifth Symphony", the NLU module 220 may classify the input as having the intent [intent: play_music]. In certain embodiments, a domain can be conceptually a namespace for a set of intents, e.g., music. A slot can be a named substring with user input that represents a basic semantic entity. For example, a slot for "pizza" can be [slot:dish]. In certain embodiments, the set of valid or expected named slots can be based on a categorized intent. As an example and not by way of limitation, for [intent:play_music], the slot can be [slot:song_name].The semantic information aggregator 230 may also extract information from social graphs, knowledge graphs, and concept graphs, and retrieve user profiles from the user context engine 225. The semantic information aggregator 230 may also process information from these different sources by determining what information to aggregate, annotating n-grams of user input, sorting n-grams with confidence scores based on the aggregated information, and formulating the sorted n-grams into features that can be used by the NLU module 220 to understand the user input. More information about aggregated semantic information can be found in U.S. Patent Application No. 15 / 967,342 filed on April 30, 2018, which is incorporated by reference. Based on the output of the user context engine 225 and the semantic information aggregator 230, the NLU module 220 may identify domains, intents, and one or more slots from the user input in a personalized and context-aware manner. As an example and not by way of limitation, the user input may include "show me how to get to the Starbucks". The NLU module 220 may identify a specific Starbucks that the user wants to go to based on the user's personal information and associated context information. In a particular embodiment, the NLU module 220 may include a lexicon of language, a parser, and grammatical rules to divide sentences into internal representations. The NLU module 220 may also include one or more programs that use pragmatics to perform naive semantics or random semantic analysis to understand user input. In a particular embodiment, the parser may be based on a deep learning architecture including multiple long short-term memory (LSTM) networks. As an example and not as a limitation, the parser may be based on a recurrent neural network grammar (RNNG) model, which is a type of recursive and cyclic LSTM algorithm. More information about natural language understanding can be found in U.S. Patent Application No. 16 / 011062 filed on June 18, 2018, U.S. Patent Application No. 16 / 025317 filed on July 2, 2018, and U.S. Patent Application No. 16 / 038120 filed on July 17, 2018, each of which is incorporated by reference.

[0145] In a particular embodiment, the identified domain, intent, and one or more slots from the NLU module 220 may be sent to the conversation engine 235. In a particular embodiment, the conversation engine 235 may manage the conversation flow and conversation state between the user and the assistant xbot 215. The conversation engine 235 may also store previous conversations between the user and the assistant xbot 215. In a particular embodiment, the conversation engine 235 may communicate with the entity resolution module 240 to resolve entities associated with one or more slots, which supports the conversation engine 235 to forward the conversation flow between the user and the assistant xbot 215. In a particular embodiment, the entity resolution module 240 may access social graphs, knowledge graphs, and concept graphs when resolving entities. An entity may include, for example, a unique user or concept, each of which may have a unique identifier (ID). As an example and not as a limitation, a knowledge graph may include multiple entities. Each entity may include a single record associated with one or more attribute values. A specific record may be associated with a unique entity identifier. For an attribute of an entity, each record may have a different value. Each attribute value may be associated with a confidence probability. The confidence probability of an attribute value indicates the probability that the value is accurate for a given attribute. Each attribute value may also be associated with a semantic weight. The semantic weight of an attribute value may indicate the degree to which the value is semantically suitable for a given attribute, taking into account all available information. For example, a knowledge graph may include an entity of the movie "The Martian" (2015), which includes information that has been extracted from multiple content sources (e.g., Facebook, Wikipedia, movie review sources, media databases, and entertainment content sources), and then deduped, parsed, and fused to generate a single unique record of a knowledge graph. An entity may be associated with a spatial attribute value indicating the genre of the movie "The Martian" (2015). More information about the knowledge graph may be found in U.S. Patent Application No. 16 / 048049 filed on July 27, 2018 and U.S. Patent Application No. 16 / 048101 filed on July 27, 2018, each of which is incorporated by reference. The entity resolution module 240 may additionally request a user profile of a user associated with a user input from the user context engine 225. In certain embodiments, entity resolution module 240 may communicate with privacy check module 245 to ensure that the resolution of the entity does not violate the privacy policy. In certain embodiments, privacy check module 245 may use an authorization / privacy server to enforce the privacy policy. As an example and not by way of limitation, the entity to be resolved may be another user who has specified in his / her privacy settings that his / her identity should not be searchable on an online social network, and therefore entity resolution module 240 may respond to the request without returning an identifier for that user.Based on information obtained from social graphs, knowledge graphs, concept graphs, and user profiles, and in compliance with applicable privacy policies, the entity resolution module 240 can therefore accurately resolve entities associated with user input in a personalized and context-aware manner. In a particular embodiment, each resolved entity can be associated with one or more identifiers hosted by the social networking system 160. As an example and not by way of limitation, the identifier can include a unique user identifier (ID). In a particular embodiment, each resolved entity can also be associated with a confidence score. More information about resolved entities can be found in U.S. Patent Application No. 16 / 048049, filed on July 27, 2018, and U.S. Patent Application No. 16 / 048072, filed on July 27, 2018, each of which is incorporated by reference.

[0146] In a particular embodiment, the conversation engine 235 can communicate with different agents based on the identified intent and domain and the resolved entity. In a particular embodiment, the agent can be an implementation of a broker between multiple content providers acting as a domain. A content provider can be an entity responsible for performing an action associated with an intent or completing a task associated with an intent. As an example and not as a limitation, multiple device-specific implementations (e.g., real-time calls to a messaging application on or to a client system 130) can be handled internally by a single agent. Alternatively, these device-specific implementations can be handled by multiple agents associated with multiple domains. In a particular embodiment, the agent may include a first-party agent 250 and a third-party agent 255. In a particular embodiment, the first-party agent 250 may include an internal agent (e.g., an agent associated with a service provided by an online social network (Messenger, Instagram)) that can be accessed and controlled by the assistant system 140. In a particular embodiment, the third-party agent 255 may include an external agent (e.g., a music streaming agent (Spotify), a ticket sales agent (Ticketmaster)) that the assistant system 140 cannot control. The first party agent 250 may be associated with a first party provider 260 that provides content objects and / or services hosted by the social networking system 160. The third party agent 255 may be associated with a third party provider 265 that provides content objects and / or services hosted by the third party system 170.

[0147] In certain embodiments, communications from the conversation engine 235 to the first party agent 250 may include a request for a specific content object and / or service provided by the first party provider 260. Thus, the first party agent 250 may retrieve the requested content object from the first party provider 260 and / or execute a task that instructs the first party provider 260 to perform the requested service. In certain embodiments, communications from the conversation engine 235 to the third party agent 255 may include a request for a specific content object and / or service provided by the third party provider 265. Thus, the third party agent 255 may retrieve the requested content object from the third party provider 265 and / or execute a task that instructs the third party provider 265 to perform the requested service. The third party agent 255 may access the privacy check module 245 to ensure that there is no privacy violation before interacting with the third party provider 265. As an example and not by way of limitation, a user associated with the user input may specify in his / her privacy settings that his / her profile information is not visible to any third party content provider. Thus, when retrieving a content object associated with a user input from a third party provider 265 , the third party agent 255 may complete the retrieval without revealing to the third party provider 265 which user is requesting the content object.

[0148] In certain embodiments, each of the first party agent 250 or the third party agent 255 may be designated for a specific domain. By way of example and not limitation, the domain may include weather, transportation, music, etc. In certain embodiments, the assistant system 140 may use multiple agents in concert to respond to user input. By way of example and not limitation, the user input may include "direct me to my next meeting". The assistant system 140 may use a calendar agent to retrieve the location of the next meeting. The assistant system 140 may then use a navigation agent to direct the user to the next meeting.

[0149] In certain embodiments, each of the first party agent 250 or the third party agent 255 may retrieve a user profile from the user context engine 225 to perform the task in a personalized and context-aware manner. By way of example and not limitation, the user input may include "book me a ride to the airport". The transportation agent may perform the task of booking a ride. The transportation agent may retrieve the user profile of the user from the user context engine 225 before booking the ride. For example, the user profile may indicate that the user prefers taxis, so the transportation agent may book a taxi for the user. As another example, context information associated with the user profile may indicate that the user is in a hurry, so the transportation agent may book a ride for the user from a ride-sharing service (e.g., Uber, Lyft) because getting a ride from a ride-sharing service may be faster than a taxi company. In certain embodiments, each of the first party agent 250 or the third party agent 255 may consider other factors when performing the task. By way of example and not limitation, the other factors may include price, ratings, efficiency, partnerships with online social networks, etc.

[0150] In certain embodiments, the dialog engine 235 may communicate with a conversation understanding writer (CU writer) 270. The dialog engine 235 may send the requested content object and / or the status of the requested service to the CU writer 270. In certain embodiments, the dialog engine 235 may send the requested content object and / or the status of the requested service to the CU writer 270.<k,c,u,d> Tuples are sent, where k indicates a knowledge source, c indicates a communication target, u indicates a user model, and d indicates a discourse model. In a specific embodiment, the CU writer 270 may include a natural language generator (NLG) 271 and a user interface (UI) payload generator 272. The natural language generator 271 may generate communication content based on the output of the dialogue engine 235. In a specific embodiment, the NLG 271 may include a content determination component, a sentence planner, and a surface realization component. The content determination component may determine the communication content based on the knowledge source, the communication target, and the user's expectations. As an example and not as a limitation, the determination may be based on description logic. The description logic may include, for example, three basic notions, which are individuals (representing objects in a domain), concepts (describing individual sets), and roles (representing binary relationships between individuals or concepts). The description logic may be characterized by a set of constructors that allow the natural language generator 271 to build complex concepts / roles from atomic concepts / roles. In a specific embodiment, the content determination component may perform the following tasks to determine the communication content. The first task may include a translation task, in which the input to the natural language generator 271 can be translated into concepts. The second task may include a selection task, in which relevant concepts can be selected from the concepts generated by the translation task based on the user model. The third task may include a verification task, in which the consistency of the selected concept can be verified. The fourth task may include an instantiation task, in which the verified concept can be instantiated into an executable file that can be processed by the natural language generator 271. The sentence planner can determine the organization of the communication content so that it is understandable to people. The surface implementation component can determine the specific words to be used, the order of sentences, and the style of the communication content. The UI payload generator 272 can determine the preferred modality of the communication content to be presented to the user. In a specific embodiment, the CU writer 270 can communicate with the privacy check module 245 to ensure that the generation of the communication content follows the privacy policy. In a specific embodiment, when generating the communication content and determining the modality of the communication content, the CU writer 270 can retrieve the user profile from the user context engine 225. Therefore, for the user, the communication content can be more natural, personalized, and context-aware. As an example and not by way of limitation, a user profile may indicate that the user prefers short sentences in conversation, and thus the generated communication content may be based on short sentences.As another example and not by way of limitation, context information associated with the user profile may indicate that the user is using a device that only outputs audio signals, so the UI payload generator 272 may determine the modality of the communication content as audio. More information about natural language generation can be found in U.S. Patent Application No. 15 / 967,279, filed on April 30, 2018, and U.S. Patent Application No. 15 / 966,455, filed on April 30, 2018, each of which is incorporated by reference.

[0151] In certain embodiments, the CU writer 270 may send the generated communication content to the assistant xbot 215. In certain embodiments, the assistant xbot 215 may send the communication content to the messaging platform 205. The messaging platform 205 may also send the communication content to the client system 130 via the assistant application 136. In an alternative embodiment, the assistant xbot 215 may send the communication content to a text-to-speech (TTS) module 275. The TTS module 275 may convert the communication content into an audio clip. The TTS module 275 may also send the audio clip to the client system 130 via the assistant application 136.

[0152] In certain embodiments, assistant xbot 215 may interact with proactive inference layer 280 without receiving user input. Proactive inference layer 280 may infer user interests and preferences based on a user profile retrieved from user context engine 225. In certain embodiments, proactive inference layer 280 may also communicate with proactive agent 285 regarding inferences. Proactive agent 285 may perform proactive tasks based on inferences. As an example and not as a limitation, proactive tasks may include sending content objects to users or providing services. In certain embodiments, each proactive task may be associated with an agenda item. Agenda items may include recurring items, such as daily summaries. Agenda items may also include one-time items. In certain embodiments, proactive agent 285 may retrieve a user profile from user context engine 225 when performing proactive tasks. Thus, proactive agent 285 may perform proactive tasks in a personalized and context-aware manner. As an example and not as a limitation, the proactive inference layer may infer that a user likes the band Maroon 5, and proactive agent 285 may generate recommendations for new songs / albums of Maroon 5 for the user.

[0153] In a particular embodiment, the active agent 285 can generate candidate entities associated with the active task based on the user profile. The generation can be based on a direct back-end query that retrieves the candidate entity from a structured data storage using a deterministic filter. Alternatively, the generation can be based on a machine learning model that is trained based on the user profile, entity attributes, and the correlation between the user and the entity. As an example and not as a limitation, the machine learning model can be based on a support vector machine (SVM). As another example and not as a limitation, the machine learning model can be based on a regression model. As another example and not as a limitation, the machine learning model can be based on a deep convolutional neural network (DCNN). In a particular embodiment, the active agent 285 can also sort the generated candidate entities based on the user profile and the content associated with the candidate entity. The sorting can be based on the similarity between the user's interest and the candidate entity. As an example and not as a limitation, the assistant system 140 can generate a feature vector representing the user's interest and a feature vector representing the candidate entity. The assistant system 140 can then calculate a similarity score (e.g., based on cosine similarity) between the feature vector representing the user's interest and the feature vector representing the candidate entity. Alternatively, the ranking may be based on a ranking model that is trained based on user feedback data.

[0154] In certain embodiments, the active task may include recommending a candidate entity to a user. The active agent 285 may schedule the recommendation, thereby associating the recommendation time with the recommended candidate entity. The recommended candidate entity may also be associated with a priority and an expiration time. In certain embodiments, the recommended candidate entity may be sent to an active scheduler. The active scheduler may determine the actual time to send the recommended candidate entity to the user based on the priority associated with the task and other relevant factors (e.g., clicks and impressions of the recommended candidate entity). In certain embodiments, the active scheduler may then send the recommended candidate entity with the determined actual time to an asynchronous tier. The asynchronous tier may temporarily store the recommended candidate entity as a job. In certain embodiments, the asynchronous tier may send the job to the conversation engine 235 at the determined actual time for execution. In alternative embodiments, the asynchronous tier may execute the job by sending it to other surface layers (e.g., other notification services associated with the social networking system 160). In certain embodiments, the conversation engine 235 may identify the conversation intent, state, and history associated with the user. Based on the conversation intent, the conversation engine 235 may select some candidate entities from the recommended candidate entities to send to the client system 130. In a particular embodiment, the conversation state and history may indicate whether the user is involved in an ongoing conversation with assistant xbot 215. If the user is involved in an ongoing conversation and the priority of the recommended task is low, the conversation engine 235 may communicate with the active scheduler to reschedule the time to send the selected candidate entity to the client system 130. If the user is involved in an ongoing conversation and the priority of the recommended task is high, the conversation engine 235 may initiate a new conversation session with the user, in which the selected candidate entity may be presented. Therefore, the interruption of the ongoing conversation may be prevented. When it is determined that sending the selected candidate entity will not interrupt the user, the conversation engine 235 may send the selected candidate entity to the CU writer 270 to generate personalized and context-aware communication content including the selected candidate entity according to the user's privacy settings. In a particular embodiment, the CU writer 270 may send the communication content to the assistant xbot 215, which may then send it to the client system 130 via the messaging platform 205 or the TTS module 275. More information about proactively helping users can be found in U.S. Patent Application No. 15 / 967,193, filed on April 30, 2018, and U.S. Patent Application No. 16 / 036,827, filed on July 16, 2018, each of which is incorporated by reference.

[0155] In certain embodiments, assistant xbot 215 may communicate with proactive agent 285 in response to user input. As an example and not by way of limitation, a user may ask assistant xbot 215 to set a reminder. Assistant xbot 215 may request proactive agent 285 to set such a reminder, and proactive agent 285 may later proactively perform the task of reminding the user.

[0156] In a particular embodiment, the assistant system 140 may include a summarizer 290. The summarizer 290 may provide a customized dynamic message summary to the user. In a particular embodiment, the summarizer 290 may include multiple metaagents. Multiple metaagents may use first-party agents 250, third-party agents 255, or active agents 285 to generate dynamic message summaries. In a particular embodiment, the summarizer 290 may retrieve user interests and preferences from the active inference layer 280. The summarizer 290 may then retrieve entities associated with user interests and preferences from the entity resolution module 240. The summarizer 290 may also retrieve a user profile from the user context engine 225. Based on information from the active inference layer 280, the entity resolution module 240, and the user context engine 225, the summarizer 290 may generate a personalized and context-aware summary for the user. In a particular embodiment, the summarizer 290 may send the summary to the CU writer 270. The CU writer 270 may process the summary and send the processing result to the assistant xbot 215. Assistant xbot 215 may then send the processed summary to client system 130 via messaging platform 205 or TTS module 275. More information about summarization can be found in U.S. Patent Application No. 15 / 967,290, filed April 30, 2018, which is incorporated by reference.

[0157] Figure 3An example flow chart of the assistant system 140 responding to a user request is shown. In a particular embodiment, the assistant xbot 215 can access the request manager 305 when receiving a user request. The request manager 305 can include a context extractor 306 and a conversation understanding object generator (CU object generator) 307. The context extractor 306 can extract context information associated with the user request. The context extractor 306 can also update the context information based on the assistant application 136 executed on the client system 130. As an example and not as a limitation, the update of the context information can include displaying a content item on the client system 130. As another example and not as a limitation, the update of the context information can include setting an alarm on the client system 130. As another example and not as a limitation, the update of the context information can include playing a song on the client system 130. The CU object generator 307 can generate a specific content object related to the user request. The content object can include conversation session data and features associated with the user request, which can be shared with all modules of the assistant system 140. In particular embodiments, request manager 305 may store context information and generated content objects in data store 310 , which is a particular data store implemented in assistant system 140 .

[0158] In a particular embodiment, the request manager 305 may send the generated content object to the NLU module 220. The NLU module 220 may perform a number of steps to process the content object. In step 221, the NLU module 220 may generate a whitelist of content objects. In a particular embodiment, the whitelist may include interpretation data that matches the user request. In step 222, the NLU module 220 may perform characterization based on the whitelist. In step 223, the NLU module 220 may perform domain classification / selection on the user request based on the features generated by the characterization to classify the user request into a predefined domain. The domain classification / selection results may also be further processed based on two related processes. In step 224a, the NLU module 220 may use an intent classifier to process the domain classification / selection results. The intent classifier may determine the user intent associated with the user request. In a particular embodiment, each domain may have an intent classifier to determine the most likely intent in a given domain. As an example and not by way of limitation, the intent classifier may be based on a machine learning model that may take the domain classification / selection results as input and calculate the probability that the input is associated with a particular predefined intent. In step 224b, the NLU module may use a meta-intent classifier to process the domain classification / selection results. The meta-intent classifier may determine a category that describes the user's intent. In a particular embodiment, intents common to multiple domains may be processed by the meta-intent classifier. As an example and not by way of limitation, the meta-intent classifier may be based on a machine learning model that may take the domain classification / selection results as input and calculate the probability that the input is associated with a specific predefined meta-intent. In step 225a, the NLU module 220 may use a slot tagger to annotate one or more slots associated with the user request. In a particular embodiment, the slot tagger may annotate one or more slots for n-grams of the user request. In step 225b, the NLU module 220 may use a meta-slot tagger to annotate one or more slots for the classification results from the meta-intent classifier. In a particular embodiment, the meta-slot tagger may tag common slots such as references to items (e.g., the first one), slot types, slot values, and the like. By way of example and not limitation, a user request may include "change 500 dollars in my account to Japanese yen". The intent classifier may take as input the user request and formulate it into a vector. The intent classifier may then calculate the probability that the user request is associated with different predefined intents based on a vector comparison between the vector representing the user request and the vectors representing different predefined intents. In a similar manner, the slot tagger may take as input the user request and formulate each word into a vector.The intent classifier can then calculate the probability of each word being associated with different predefined slots based on a vector comparison between the vector representing the word and the vector representing the different predefined slots. The user's intent can be classified as "changing money". The slots requested by the user can include "500", "dollars", "account", and "Japanese yen". The user's meta-intent can be classified as "financial service". The meta-slot can include "finance".

[0159] In certain embodiments, the NLU module 220 may improve the domain classification / selection of content objects by extracting semantic information from the semantic information aggregator 230. In certain embodiments, the semantic information aggregator 230 may aggregate semantic information in the following manner. The semantic information aggregator 230 may first retrieve information from the user context engine 225. In certain embodiments, the user context engine 225 may include an offline aggregator 226 and an online inference service 227. The offline aggregator 226 may process a plurality of data associated with the user collected from a previous time window. As an example and not by way of limitation, the data may include dynamic message posts / comments, interactions with dynamic message posts / comments, Instagram posts / comments, search history, etc. collected from a previous 90-day window. The processing results may be stored in the user context engine 225 as part of the user profile. The online inference service 227 may analyze the session data associated with the user received by the assistant system 140 at the current time. The analysis results may also be stored in the user context engine 225 as part of the user profile. In certain embodiments, both the offline aggregator 226 and the online inference service 227 may extract personalized features from the plurality of data. The extracted personalized features can be used by other modules of the assistant system 140 to better understand the user input. In a particular embodiment, the semantic information aggregator 230 can then process the information retrieved from the user context engine 225, i.e., the user profile, in the following steps. In step 231, the semantic information aggregator 230 can process the information retrieved from the user context engine 225 based on natural language processing (NLP). In a particular embodiment, the semantic information aggregator 230 can: tokenize the text by text normalization, extract syntax features from the text, and extract semantic features from the text based on NLP. The semantic information aggregator 230 can also extract features from context information, which is accessed from the conversation history between the user and the assistant system 140. The semantic information aggregator 230 can also perform global word embedding, domain-specific embedding, and / or dynamic embedding based on the context information. In step 232, the processing results can be annotated with entities by the entity tagger. In step 233, based on the annotations, the semantic information aggregator 230 can generate a dictionary for the retrieved information. In certain embodiments, the dictionary may include a global dictionary feature that may be dynamically updated offline. At step 234, the semantic information aggregator 230 may rank the entities tagged by the entity tagger. In certain embodiments, the semantic information aggregator 230 may communicate with different graphs 330 including social graphs, knowledge graphs, and concept graphs to extract ontology data related to the information retrieved from the user context engine 225. In certain embodiments, the semantic information aggregator 230 may aggregate the user profile, the ranked entities, and the information from the graphs 330.The semantic information aggregator 230 may then send the aggregated information to the NLU module 220 to facilitate domain classification / selection.

[0160] In certain embodiments, the output of the NLU module 220 may be sent to a co-reference module 315 to interpret the references of the content object associated with the user request. In certain embodiments, the co-reference module 315 may be used to identify the item to which the user request refers. The co-reference module 315 may include reference creation 316 and reference resolution 317. In certain embodiments, the reference creation 316 may create references for the entities determined by the NLU module 220. The reference resolution 317 may accurately resolve these references. As an example and not by way of limitation, the user request may include "find me the nearest Walmart and direct me there". The co-reference module 315 may interpret "there" as "the nearest Walmart". In certain embodiments, the co-reference module 315 may access the user context engine 225 and the dialog engine 235, if necessary, to interpret the references with improved accuracy.

[0161] In a particular embodiment, the identified domains, intents, meta-intents, slots and meta-slots, and resolved references may be sent to entity resolution module 240 to resolve related entities. Entity resolution module 240 may perform general and domain-specific entity resolution. In a particular embodiment, entity resolution module 240 may include domain entity resolution 241 and general entity resolution 242. Domain entity resolution 241 may resolve entities by categorizing slots and meta-slots into different domains. In a particular embodiment, entities may be resolved based on ontology data extracted from graph 330. Ontology data may include structural relationships between different slots / meta-slots and domains. Ontology may also include information on how slots / meta-slots may be grouped, related, and subdivided according to similarities and differences within a hierarchy that includes domains at a higher level. General entity resolution 242 may resolve entities by categorizing slots and meta-slots into different general topics. In a particular embodiment, resolution may also be based on ontology data extracted from graph 330. Ontology data may include structural relationships between different slots / meta-slots and general topics. The ontology may also include information about how slots / meta-slots may be grouped, related, and subdivided according to similarities and differences within a hierarchy that includes topics at a higher level. As an example and not by way of limitation, in response to an input of a query about the advantages of a Tesla car, the general entity resolution 242 may resolve a Tesla car to a vehicle, and the domain entity resolution 241 may resolve a Tesla car to an electric car.

[0162] In a particular embodiment, the output of the entity resolution module 240 may be sent to the conversation engine 235 to forward the conversation flow with the user. The conversation engine 235 may include a conversation intent resolution 236 and a conversation state updater / sorter 237. In a particular embodiment, the conversation intent resolution 236 may resolve the user intent associated with the current conversation session based on the conversation history between the user and the assistant system 140. The conversation intent resolution 236 may map the intent determined by the NLU module 220 to different conversation intents. The conversation intent resolution 236 may also sort the conversation intent based on the signals from the NLU module 220, the entity resolution module 240, and the conversation history between the user and the assistant system 140. In a particular embodiment, the conversation state updater / sorter 237 may update / sort the conversation state of the current conversation session. As an example and not as a limitation, if the conversation session ends, the conversation state updater / sorter 237 may update the conversation state to "completed". As another example and not as a limitation, the conversation state updater / sorter 237 may sort the conversation state based on the priority associated with the conversation state.

[0163] In certain embodiments, the dialog engine 235 may communicate with the task completion module 335 regarding the dialog intent and the associated content object. In certain embodiments, the task completion module 335 may rank different dialog hypotheses for different dialog intents. The task completion module 335 may include an action selection component 336. In certain embodiments, the dialog engine 235 may additionally check against the dialog strategy 320 regarding the dialog state. In certain embodiments, the dialog strategy 320 may include a data structure describing the action execution plan of the agent 340. The agent 340 may select from the registered content providers to complete the action. The data structure may be constructed by the dialog engine 235 based on the intent and one or more slots associated with the intent. The dialog strategy 320 may also include multiple goals that are interrelated by logical operators. In certain embodiments, the goal may be the output result of a part of the dialog strategy, and it may be constructed by the dialog engine 235. The goal may be represented by an identifier (e.g., a string) having one or more named parameters that parameterize the goal. As an example and not as a limitation, a goal and its associated goal parameter can be represented as {confirm_artist, parameter: {artist: "Madonna"}}. In a particular embodiment, the dialogue strategy can be represented based on a tree structure, in which the goal is mapped to the leaves. In a particular embodiment, the dialogue engine 235 can execute the dialogue strategy 320 to determine the next action to be performed. The dialogue strategy 320 may include a general strategy 321 and a domain-specific strategy 322, both of which can guide how to select the next system action based on the dialogue state. In a particular embodiment, the task completion module 335 can communicate with the dialogue strategy 320 to obtain guidance for the next system action. In a particular embodiment, the action selection component 336 can therefore select an action based on the dialogue intention, the associated content object, and the guidance from the dialogue strategy 320.

[0164] In a particular embodiment, the output of the task completion module 335 may be sent to the CU writer 270. In an alternative embodiment, the selected action may require one or more agents 340 to participate. Therefore, the task completion module 335 may notify the agent 340 of the selected action. At the same time, the conversation engine 235 may receive an instruction to update the conversation state. As an example and not as a limitation, the update may include waiting for the response of the agent. In a particular embodiment, the CU writer 270 may generate communication content for the user using NLG 271 based on the output of the task completion module 335. In a particular embodiment, the NLG 271 may use different language models and / or language templates to generate natural language output. The generation of natural language output may be application-specific. The generation of natural language output may also be personalized for each user. The CU writer 270 may also use the UI payload generator 272 to determine the modality of the generated communication content. Since the generated communication content may be considered as a response to a user request, the CU writer 270 may additionally use a response sorter 273 to sort the generated communication content. As an example and not as a limitation, the sorting may indicate the priority of the response.

[0165] In a particular embodiment, the output of the CU writer 270 may be sent to a response manager 325. The response manager 325 may perform different tasks, including storing / updating the dialog state 326 retrieved from the data storage 310 and generating a response 327. In a particular embodiment, the output of the CU writer 270 may include one or more of a natural language string, a voice, or an action with parameters. Therefore, the response manager 325 may determine what task to perform based on the output of the CU writer 270. In a particular embodiment, the generated response and communication content may be sent to the assistant xbot 215. In an alternative embodiment, if the modality of the determined communication content is audio, the output of the CU writer 270 may be sent to the TTS module 275 in addition. Then, the voice generated by the TTS module 275 and the response generated by the response manager 325 may be sent to the assistant xbot 215.

[0166] Contextual autocompletion for assistant system

[0167] In certain embodiments, assistant system 140 may suggest context-related, pre-input-like auto-completion to the user. Assistant system 140 may receive user input in various modalities, including audio, text, video, images, etc. In a public environment, some modalities may be inconvenient and / or unsuitable for use (e.g., audio and video), and users may prefer to enter text input into assistant system 140 to protect their privacy. One challenge of text input may be that keyboard entry is slower than the audio / voice input of a typical user. Accordingly, methods to increase keyboard entry speed will be beneficial to improving the user's interaction with assistant system 140. In certain embodiments, assistant system 140 may generate suggested auto-completion using a personalized language model that predicts the next keyboard entry (e.g., character, word, phrase, sentence, etc.), which helps users complete their entries faster and with less effort. As an example and not as a limitation, if a user is interacting with assistant system 140 via a messaging interface and has typed "call...", assistant system 140 may determine that the input corresponds to the intent [IN: call (person)]. The personalized language model can then predict that the next keyboard entry is a person's name to fill the slot [SL:person(name)]. Entries for suggested auto-completion can be stored in a scope dictionary tree, which indexes entries to allow efficient lookup for a given prefix. Accordingly, the personalized language model can select a list of entries for suggested auto-completion from the scope dictionary tree. The user can also select suggested auto-completion, thereby reducing the number of keystrokes the user needs to enter to complete a request that can be executed by assistant system 140. Although the present disclosure describes suggesting a particular auto-completion in a particular manner via a particular system, the present disclosure contemplates suggesting any suitable auto-completion in any suitable manner via any suitable system.

[0168] In a particular embodiment, the assistant system 140 may receive user input from the first user from the client system 130 associated with the first user. The user input may include a partial request. In a particular embodiment, the assistant system 140 may analyze the user input based on the personalized language model to generate one or more candidate hypotheses corresponding to the partial request. Each of the one or more candidate hypotheses may include one or more of the intent suggestions or slot suggestions. Each of the one or more candidate hypotheses may additionally correspond to a subsequent entry associated with the user input. In a particular embodiment, the assistant system 140 may send an instruction to the client system 130 for presenting one or more suggested auto-completion corresponding to the one or more candidate hypotheses, respectively. Each suggested auto-completion may include the partial request and the corresponding candidate hypothesis. In a particular embodiment, the assistant system 140 may receive from the client system 130 an indication of the first user's selection of the first suggested auto-completion in the one or more suggested auto-completions. The assistant system 140 may also perform one or more tasks via one or more agents based on the first suggested auto-completion selected by the first user.

[0169] Figure 4 Shown based on Figure 2. In a particular embodiment, the assistant xbot 215 may receive user input 405 from a first user from a client system 130 associated with the first user. As an example and not as a limitation, the user input may include a string. In a particular embodiment, the user input 405 may include a partial request. The partial request may include an unexecutable request for which the assistant system 140 may not immediately determine the tasks associated with the request and perform the tasks. In a particular embodiment, the assistant xbot 215 may send the user input 405 to the dialogue engine 235. The dialogue engine 235 may analyze the user input 405 based on the personalized language model 410 to generate one or more candidate hypotheses 415 corresponding to the partial request. The generation of one or more candidate hypotheses 415 may also be based on one or more context-specific language models 420, one or more global language models 425, and one or more global context-specific language models 430. In a particular embodiment, the dialogue engine 235 may train the personalized language model 410 based on the training data 435 accessed from the user context engine 225. The dialogue engine 235 may train the context-specific language model 420 based on the context-specific data 440 accessed from the user context engine 225. In certain embodiments, the dialogue engine 235 may train the global language model 445 based on the global user data 450 accessed from the data storage 164 of the social networking system 160. The dialogue engine 235 may train the global context-specific language model 430 based on the global context-specific user data 455 accessed from the data storage 164 of the social networking system 160. In certain embodiments, the global user data 450 and the global context-specific user data 455 are data associated with multiple users on the online social network. In certain embodiments, the dialogue engine 235 may store the trained personalized language model 410 and the context-specific language model 420 locally on the user's client system 130 to protect the user's privacy. When analyzing the user input 405 to generate the candidate hypothesis 415, the dialogue engine 235 may then access the stored model from the user's client system 130 according to the user's permission. In certain embodiments, the dialog engine 235 can store the trained global language model 425 and the global context-specific language model 430 in one or more data stores 310 of the assistant system 140, because these models do not contain privacy-sensitive information of a particular user. When analyzing the user input 405 to generate the candidate hypothesis 415, the dialog engine 235 can then access the stored models from the data store 310. In certain embodiments, the generated candidate hypothesis 415 can be sent back to the assistant xbot 215. The assistant xbot 215 can also generate suggested auto-completion 460 corresponding to the candidate hypothesis 415.Each suggested auto-completion 460 may include a partial request and a corresponding candidate hypothesis 415. The assistant xbot 215 may also send instructions to the client system 130 for presenting the suggested auto-completion 460 to the first user. In a particular embodiment, the first user may select a suggested auto-completion 460 from the suggested auto-completion 460. The selection of the auto-completion 460 may convert the partial request into a complete request. Accordingly, the assistant system 140 may then perform the task associated with the complete request. Therefore, the assistant system 140 may have a technical advantage of improving the user's interaction with the assistant system 140 by helping the user complete their typing faster and with less effort. Although the present disclosure describes the use of a particular module in a particular manner to suggest a particular auto-completion, the present disclosure contemplates the use of any suitable module in any suitable manner to suggest any suitable auto-completion.

[0170] In a particular embodiment, each of the one or more candidate hypotheses 415 may include one or more of an intent suggestion or a slot suggestion. The assistant system 140 may analyze the user input 405 to generate one or more candidate hypotheses 415, which include intent suggestions corresponding to the partial request in the following manner. The assistant system 140 may first analyze the user input 405 based on the personalized language model 410 to determine one or more candidate intents. In a particular embodiment, the assistant system 140 may then send one or more intent suggestions corresponding to the one or more candidate intents to the client system 130. The assistant system 140 may also receive a selection of one of the one or more intent suggestions from the first user from the client system 130. The selected intent suggestion may then be provided as an intent suggestion of one of the candidate hypotheses 415. In an alternative embodiment, the assistant system 140 may provide intent suggestions after determining one or more candidate intents, as described below. The assistant system 140 may send a request for additional information from the first user to the client system 130. The assistant system 140 may then receive additional user input from the first user in response to the request from the client system 130. Assistant system 140 can also disambiguate one or more candidate intents based on additional user input to determine a top candidate intent to provide as an intent suggestion for one of candidate hypotheses 415. Although this disclosure describes providing particular intent suggestions in a particular manner, this disclosure contemplates providing any suitable intent suggestions in any suitable manner.

[0171] In a particular embodiment, assistant system 140 may analyze user input 405 to generate one or more candidate hypotheses 415 in the following manner, the one or more candidate hypotheses 415 including slot suggestions corresponding to the partial request. Assistant system 140 may first analyze user input 405 based on personalized language model 410 to determine one or more candidate slots. In a particular embodiment, assistant system 140 may then send one or more slot suggestions corresponding to one or more candidate slots to client system 130. Assistant system 140 may also receive a selection of one of the one or more slot suggestions from the first user from client system 130. The selected slot suggestion may then be provided as a slot suggestion for one of candidate hypotheses 415. In an alternative embodiment, assistant system 140 may provide slot suggestions after determining one or more candidate slots, as described below. Assistant system 140 may send a request for additional information from the first user to client system 130. Assistant system 140 may then receive additional user input from the first user in response to the request from client system 130. Assistant system 140 can also disambiguate one or more candidate slots based on additional user input to determine a top candidate slot to provide as a slot suggestion for one of candidate hypotheses 415. Although this disclosure describes providing particular slot suggestions in a particular manner, this disclosure contemplates providing any suitable slot suggestions in any suitable manner.

[0172] In certain embodiments, each of the one or more candidate hypotheses 415 may additionally correspond to a subsequent entry associated with the user input 405. As an example and not by way of limitation, an entry may include a character, a word, a phrase, or a sentence. For example, the user input 405 may be "turn", and the subsequent entry may be "down the light" or "on the light". In certain embodiments, the assistant system 140 may store multiple entries in a range dictionary tree in the client system 130 or the data storage 310 of the assistant system 140. The range dictionary tree is a type of search tree, i.e., an ordered tree data structure for storing dynamic sets or associative arrays. All descendants of a node have a common prefix of the string associated with the node, and the root node is associated with an empty string. Each entry in the range dictionary tree may be indexed by a prefix, thereby enabling the assistant system 140 to efficiently find candidate entries. In certain embodiments, the range dictionary tree may be an additional model input to the personalized language model 410, which may speed up the assistant system 140's search for candidate entries. Storing candidate hypotheses 415 in a scope trie where assistant system 140 can quickly look up candidate hypotheses 415 may be an effective solution to the technical challenge of generating candidate hypotheses 415 in real time. Although this disclosure describes particular entries stored in a particular trie in a particular manner, this disclosure contemplates any suitable entries stored in any suitable trie in any suitable manner.

[0173] In a particular embodiment, the personalized language model 410 may be based on a recurrent neural network. A recurrent neural network is a class of artificial neural networks that can characterize the dynamic temporal behavior of a time series. A recurrent neural network can use its internal state (memory) to process an input sequence, thereby being suitable for language analysis such as speech recognition. In a particular embodiment, training a recurrent neural network may include one or more of the following: selecting the size of each hidden layer of the recurrent neural network, adjusting one or more weights of the recurrent neural network, or validating the recurrent neural network by evaluating the performance of the recurrent neural network against a threshold accuracy level. The recurrent neural network may calculate the probability that a candidate hypothesis 415 matches a subsequent entry of the user's intent after the user's initial input 405. In a particular embodiment, the personalized language model 410 may be trained based on a plurality of training data 435, the training data 435 including one or more of the following: a dynamic message post associated with a first user, a dynamic message comment associated with a first user, a message in one or more messaging interfaces associated with a first user, data representing one or more domains, a conversation state of one or more conversation sessions associated with a first user, user profile data associated with a first user, a task state associated with one or more tasks, any suitable data, or any combination thereof. As an example and not by way of limitation, the user profile data may include the user's contact information, such as a phone number (e.g., 650-123-4567). Thus, when user input "6..." is received, the personalized language model 410 may generate a candidate hypothesis 415 for the phone number. In a particular embodiment, the task status may indicate the status of the task, such as "completed", "pending", "failed", etc. Thus, the personalized language model 410 trained based on the task status may determine a higher probability for a task with a "completed" task status, but a lower probability for a task with a "failed" task status. Training the personalized language model 410 based on the task status may produce a technical advantage of directing users to tasks that the assistant system 140 excels at, to improve the user experience of the assistant system 140, because the top-ranked candidate hypotheses 415 tend to correspond to successful tasks previously performed by the assistant system 140. In certain embodiments, using a personalized language model 410 based on a recurrent neural network may be an effective solution to the technical challenge of accurately determining a candidate hypothesis 415 based on a partial request (the personalized language model 410 is trained based on various training data 435 associated with a user), because the personalized language model 410 is discriminating in determining the candidate hypothesis 415 by learning various information about the user from the training data 435, thereby fully understanding the partial request.Although this disclosure describes training a particular language model based on particular training data in a particular manner, this disclosure contemplates training any suitable language model based on any suitable training data in any suitable manner.

[0174] In certain embodiments, one or more candidate hypotheses 415 may be ranked based on the dialog state of the dialog session associated with the user input 405. In certain embodiments, the dialog state may indicate what the user wants from the assistant system 140. The dialog state may include all states used when the assistant system 140 decides what to convey to the user next. In certain embodiments, one or more candidate hypotheses 415 may be associated with one or more confidence scores, respectively. The confidence score may indicate the likelihood that the candidate hypothesis 415 matches the subsequent entry of the user's intent. In certain embodiments, one or more confidence scores may be calculated by the personalized language model 410. Accordingly, one or more candidate hypotheses 415 may be ranked based on their respective confidence scores. In certain embodiments, the assistant system 140 may receive additional user input from the client system 130. The additional user input may be appended to the initial user input 405. As an example and not as a limitation, the initial user input 405 may include a string, and the additional user input may include additional characters. The additional characters may be added to the string of the initial user input 405. In certain embodiments, the assistant system 140 may then update one or more confidence scores for one or more candidate hypotheses 415 based on the additional user input. The assistant system 140 may also re-rank one or more candidate hypotheses 415 based on the updated confidence scores. In certain embodiments, the assistant system 140 may continuously update the ranked list of candidate hypotheses 415 with the newly scored candidate hypotheses 415 until the user selects a candidate hypothesis 415 of a subsequent entry that matches the user's intent. Sorting the candidate hypotheses 415 based on the dialog state and the confidence scores determined by the personalized language model 410, and dynamically updating the confidence scores based on the additional user input to generate a ranked list of candidate hypotheses 415, may be an effective solution to the technical challenge of presenting the most relevant candidate hypotheses 415 to the user that more accurately reflect the user's intent in the current dialog session. Although the present disclosure describes ranking particular hypotheses in a particular manner, the present disclosure contemplates ranking any suitable hypothesis in any suitable manner.

[0175] In certain embodiments, the assistant system 140 may apply a sliding window to the user input 405. The length of the sliding window may determine the percentage of the user input 405 to be used as a model input for the personalized language model 410. As an example and not by way of limitation, the assistant system 140 may apply a sliding window to take the most recent 20 characters or 7 words as a model input for the personalized language model 410 to generate a candidate hypothesis 415. In certain embodiments, the assistant system 140 may apply sliding windows of different lengths to dynamically adjust how the user input 405 is analyzed based on the confidence scores of the candidate hypotheses 415. In certain embodiments, the assistant system 140 may determine whether at least one of the one or more confidence scores associated with one or more candidate hypotheses 415 is less than a threshold score. Upon determining that at least one confidence score is less than the threshold score, the assistant system 140 may adjust the length of the sliding window. In certain embodiments, adjusting the length of the sliding window may include increasing the length of the sliding window. In alternative embodiments, adjusting the length of the sliding window may include decreasing the length of the sliding window. As an example and not by way of limitation, the user input 405 may include "te" and the initial length of the sliding window may be one character. Therefore, the assistant system 140 can determine several candidate hypotheses 415, including "turn on...", "take...", "tell...", "text...", etc. One of these candidate hypotheses 415 may have a confidence score less than the threshold score. Therefore, the assistant system 140 can adjust the length by increasing the length to two characters. Accordingly, the assistant system 140 can determine several candidate hypotheses 415, including "teach me how to...", "tell..." and "text...", etc., wherein the confidence scores of all candidate hypotheses are greater than the threshold score. As another example and not by way of limitation, the user input 405 can include "reso", and the initial length of the sliding window can be four characters. Accordingly, the assistant system 140 can determine several candidate hypotheses 415, including "resolve..." and "resort to...", etc. However, "reso" may be a typing error from the user, for which the confidence scores associated with at least these candidate hypotheses 415 may be less than the threshold score. For example, if the user did not mention anything about a problem or a question in the conversation session with assistant system 140, it is reasonable that the aforementioned candidate hypotheses 415 have small confidence scores. Therefore, assistant system 140 may adjust the length by reducing the length to three characters.Accordingly, assistant system 140 may determine several candidate hypotheses 415, including “reserve dinner at…”, “reserve seat at…”, etc., where the confidence scores of all candidate hypotheses are greater than the threshold score. For example, the user has been talking about celebrating his / her partner's birthday, and it is reasonable for assistant system 140 to assign a high confidence score to the aforementioned candidate hypotheses 415 based on “res” input to personalized language model 410. Although the present disclosure describes dynamically analyzing specific user input in a specific manner, the present disclosure contemplates dynamically analyzing any suitable user input in any suitable manner.

[0176] In certain embodiments, the assistant system 140 may also analyze the user input 405 based on one or more context-specific language models 420 to generate one or more candidate hypotheses 415 corresponding to the partial request. The assistant system 140 may first access the dialog state of the dialog session associated with the user input 405 through the dialog engine 235. The assistant system 140 may then select a specific context-specific language model 420 from the one or more context-specific language models 420 based on the dialog state. The assistant system 140 may also generate one or more candidate hypotheses 415 based on the personalized language model 410 and the selected context-specific language model 420. In certain embodiments, the one or more context-specific language models 420 may be trained based on context-specific data 440. The context-specific data 440 may include one or more of the following: data associated with the presence of the first user at a specific location, data associated with the interaction of the first user with a specific user, data associated with the registration of the first user in a specific event, any suitable data, or any combination thereof. As an example and not by way of limitation, a home-specific language model 420 may be trained based on data captured when a user is within a specific radius of his / her home address. As another example and not by way of limitation, a work-specific language model 420 may be trained based on data captured when a user interacts with work colleagues via a messaging interface or within a specific radius of his / her work address. Accordingly, when the user is at home, assistant system 140 may utilize home-specific language model 420, and when the user is at work, assistant system 140 may utilize work-specific language model 420. Although this disclosure describes particular context-specific language models in particular ways, this disclosure contemplates any context-specific language model in any suitable way.

[0177] In certain embodiments, the assistant system 140 may also analyze the user input 405 based on one or more global language models 425 to generate one or more candidate hypotheses 415 corresponding to the partial request. The one or more global language models 425 may be trained based on global user data 450, which is data associated with multiple users of an online social network. In certain embodiments, the assistant system 140 may also analyze the user input 405 based on one or more global context-specific language models 430 to generate one or more candidate hypotheses 415 corresponding to the partial request. The one or more context-specific language models 430 may be trained based on global context-specific data 455, which is data associated with multiple users of an online social network. As an example and not by way of limitation, the assistant system 140 may train a global language model 425 or a global context-specific language model 430 based on data associated with multiple users in a particular region or from a particular demographic. In certain embodiments, the assistant system 140 may utilize the personalized language model 410 in conjunction with the global language model 425 and / or the global context-specific language model 430. As an example and not by way of limitation, if the user is traveling and enters "hotels...", assistant system 140 may determine the intent of user input 405 based on global language model 425 [IN:book_hotel(hotel)]. Dialogue engine 235 may then access region-specific (i.e., context-specific) global language model 430 to generate a list of candidate slots [SL:hotel(name)]. Dialogue engine 235 may also utilize personalized language model 410 to sort the list of candidate slots. Continuing with the previous example, dialogue engine 235 may additionally use global language model 425 to determine another intent [IN:get_directions(location)]. Dialogue engine 235 may also use personalized language model 410 to generate sorted lists of candidate slots [SL:location(coordinates)] that correspond to the list of candidate slots [SL:hotel(name)], respectively. Using a global language model 425 and / or a global language context-specific model 430 may be an effective solution to the technical challenge of data sparsity associated with a single user, because data associated with multiple users is sufficient to learn a discriminative language model that can improve the generation of candidate hypotheses 415 simply by personalizing the language model 410. Although this disclosure describes a particular global language model in a particular manner, this disclosure contemplates any suitable global language model in any suitable manner.

[0178] In certain embodiments, assistant system 140 may identify a category of user input 405 and accordingly generate candidate hypotheses 415 based on personalized language model 410. As an example and not by way of limitation, user input 405 may include “s” for which assistant system 140 may identify its category as “sending message”. Assistant system 140 may then generate candidate hypothesis 415 as “sending a message to [SL: person (name)] at [SL: time (time)]”. If the user selects this candidate hypothesis 415, assistant system 140 may also generate a list of highly relevant friends / family members for slot [SL: person (name)] and some possible times for slot [SL: time (time)]. As another example and not by way of limitation, user input 405 may include “r” for which assistant system 140 may identify its category as “reminder”. Assistant system 140 may then generate candidate hypotheses 415 including “Remind me to call [SL: person (name)] when I arrive at [SL: location (name)]” and “Remind me to call my mom on [SL: holiday (name)]. If the user selects “Remind me to call [SL: person (name)] when I arrive at [SL: location (name)],” assistant system 140 may also generate a list of highly relevant friends / family members for the slot [SL: person (name)] and several most likely locations for the slot [SL: location (time)]. As another example and not limitation, user input 405 may include “m,” for which assistant system 140 may identify its category as “reservation.” Assistant system 140 may then generate candidate hypothesis 415 as “Make a reservation at [SL: restaurant (name)] on [SL: date (date)]. If the user selects the candidate hypothesis 415, assistant system 140 may also generate a list of nearby and / or good restaurants for slot [SL: restaurant (name)] and some possible dates for slot [SL: date (date)]. Although this disclosure describes generating particular candidate hypotheses corresponding to particular categories of user input in particular manners, this disclosure contemplates generating any suitable candidate hypotheses corresponding to any suitable categories of user input in any suitable manner.

[0179] In particular embodiments, assistant system 140 may provide a user with null state candidate hypotheses 415. In particular embodiments, before the user begins entering user input 405, assistant system 140 may generate candidate hypotheses 415 based on information from previous user interactions with assistant system 140. As an example and not by way of limitation, previous user interactions may include information stored in user context engine 225, the user's search history, the user's request history, etc. Null state candidate hypotheses 415 may be useful for attracting users to use assistant system 140. Although this disclosure describes providing null state candidate hypotheses in a particular manner, this disclosure contemplates providing null state candidate hypotheses in any suitable manner.

[0180] Figure 5A-Figure 5B An example interaction with a user for suggested auto-completion in messaging interface 500 is shown. Figure 5A An example interaction with a user for suggested auto-completion in messaging interface 500 is shown. Figure 5A As shown, a user can interact with the assistant xbot 215 by using a keyboard 505 on a client system 130 associated with the user to generate user input 405. As an example and not by way of limitation, the user input can be "s". The assistant system 140 can suggest some auto-completions 510, including "send a message to", "set a timer", and "share". The assistant system 140 can present the suggested auto-completions 510 in a drop-down menu 515, and the user can select one of these suggested auto-completions 510. For example, the user can select "set a timer", for which the assistant system 140 can perform the corresponding task via an agent. In some cases, the suggested auto-completion 510 selected by the user may not be easy to execute, and the assistant system 140 may require additional user input from the user, which may be necessary in some cases. Figure 5B Example in. Figure 5B An example interaction with a user for a suggested auto-completion 510 in a messaging interface 500 is shown after the user has selected a previously suggested auto-completion 510. The user may have selected the previously suggested "Send message to" 520. Assistant system 140 may need to determine to whom the user wants to send the message before performing the corresponding task. Figure 5BAs shown, assistant xbot 215 may receive additional user input 525 “r” from the user. Assistant system 140 may accordingly suggest auto-completion 510, including “send a message to Raymond” and “send a message to Roger”. Assistant system 140 may present two suggested auto-completion 510 in a drop-down menu 515, from which the user may select one. For example, the user may select “send a message to Roger,” for which assistant system 140 may invoke an agent to send a message to Roger. Although the present disclosure describes particular interaction examples for suggested auto-completion conducted with a user in a particular manner, the present disclosure contemplates any suitable interaction examples for suggested auto-completion conducted with a user in any suitable manner.

[0181] Figure 6 An example method 600 for suggesting auto-completion 510 is shown. The method may begin at step 610, where assistant system 140 may receive user input 405 from a first user from a client system 130 associated with the first user, wherein user input 405 includes a partial request. At step 620, assistant system 140 may analyze user input 405 based on personalized language model 410 to generate one or more candidate hypotheses 415 corresponding to the partial request, wherein each of the one or more candidate hypotheses 415 includes one or more of an intent suggestion or a slot suggestion. At step 630, assistant system 140 may send instructions to client system 130 for presenting one or more suggested auto-completion 510 corresponding to the one or more candidate hypotheses 415, respectively, wherein each suggested auto-completion 510 includes a partial request and a corresponding candidate hypothesis 415. At step 640, assistant system 140 may receive from client system 130 an indication of a selection of a first suggested auto-completion 510 by the first user of the one or more suggested auto-completions 510. At step 650, assistant system 140 may perform one or more tasks via one or more agents based on the first suggested auto-completion 510 selected by the first user. Particular embodiments may repeat Figure 6 One or more steps of the method. Although the present disclosure describes and illustrates Figure 6 The specific steps of the method are preferably performed in a specific order, but the present disclosure contemplates Figure 6 Any suitable steps of the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates an example method for suggesting automatic completion including Figure 6method, but the present disclosure contemplates any suitable method for suggesting automatic completion including any suitable steps, which, where appropriate, may include Figure 6 All, some, or none of the steps in the method Figure 6 Furthermore, although the present disclosure describes and illustrates the steps of performing Figure 6 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Figure 6 Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0182] Social Graph

[0183] Figure 7 An example social graph 700 is shown. In certain embodiments, social networking system 160 may store one or more social graphs 700 in one or more data stores. In certain embodiments, social graph 700 may include a plurality of nodes, which may include a plurality of user nodes 702 or a plurality of concept nodes 704, and a plurality of edges 706 connecting the nodes. Each node may be associated with a unique entity (i.e., a user or a concept), and each entity may have a unique identifier (ID), such as a unique number or username. For teaching purposes, Figure 7 An example social graph 700 is shown in a two-dimensional visual map representation in FIG. 1 . In certain embodiments, social networking system 160, client system 130, assistant system 140, or third-party system 170 may access social graph 700 and related social graph information for use in appropriate applications. The nodes and edges of social graph 700 may be stored as data objects in, for example, a data store (e.g., a social graph database). Such a data store may include one or more searchable or queryable indexes of the nodes or edges of social graph 700.

[0184] In certain embodiments, user nodes 702 may correspond to users of social networking system 160 or assistant system 140. As an example and not by way of limitation, a user may be an individual (human user), an entity (e.g., a business, company, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with or through social networking system 160 or assistant system 140. In certain embodiments, when a user registers an account with social networking system 160, social networking system 160 may create a user node 702 corresponding to the user and store the user node 702 in one or more data stores. Users and user nodes 702 described herein may refer to registered users and user nodes 702 associated with registered users, where appropriate. Additionally or alternatively, users and user nodes 702 described herein may refer to users who are not registered with social networking system 160, where appropriate. In certain embodiments, user nodes 702 may be associated with information provided by a user or information collected by various systems, including social networking system 160. As an example and not by way of limitation, a user may provide his or her name, profile picture, contact information, date of birth, gender, marital status, family status, occupation, educational background, preferences, interests, or other demographic information. In certain embodiments, user node 702 may be associated with one or more data objects corresponding to information associated with the user. In certain embodiments, user node 702 may correspond to one or more web interfaces.

[0185] In certain embodiments, concept nodes 704 may correspond to concepts. By way of example and not limitation, a concept may correspond to a place (such as, for example, a movie theater, a restaurant, a landmark, or a city); a website (such as, for example, a website associated with social networking system 160 or a third-party website associated with a web application server); an entity (such as, for example, a person, a business, a group, a sports team, or a celebrity); a resource (such as, for example, an audio file, a video file, a digital photo, a text file, a structured document, or an application), which may be located within social networking system 160 or on an external server (e.g., a web application server); real estate or intellectual property (such as, for example, a sculpture, a painting, a movie, a game, a song, an idea, a photo, or a written work); a game; an activity; an idea or theory; another suitable concept; or two or more such concepts. Concept nodes 704 may be associated with information about the concept provided by a user or information collected by various systems, including social networking system 160 and assistant system 140. By way of example and not limitation, information for a concept may include a name or title; one or more images (e.g., an image of a book cover); a location (e.g., an address or geographic location); a website (which may be associated with a URL); contact information (e.g., a phone number or email address); other suitable concept information; or any suitable combination of such information. In particular embodiments, a concept node 704 may be associated with one or more data objects that correspond to the information associated with the concept node 704. In particular embodiments, a concept node 704 may correspond to one or more web interfaces.

[0186] In certain embodiments, a node in social graph 700 may represent a web interface (which may be referred to as a "profile interface") or be represented by a web interface. The profile interface may be hosted by social networking system 160 or assistant system 170 or accessible to social networking system 160 or assistant system 170. The profile interface may also be hosted on a third-party website associated with third-party system 170. As an example and not as a limitation, a profile interface corresponding to a particular external web interface may be a particular external web interface, and the profile interface may correspond to a particular concept node 704. The profile interface may be viewable by all or a selected subset of other users. As an example and not as a limitation, user node 702 may have a corresponding user profile interface, where the corresponding user may add content, make statements, or otherwise express him or herself. As another example and not as a limitation, concept node 704 may have a corresponding concept profile interface, where one or more users may add content, make statements, or express themselves, particularly with respect to the concept corresponding to concept node 704.

[0187] In certain embodiments, concept node 704 may represent a third-party web interface or resource hosted by third-party system 170. The third-party web interface or resource may include content representing an action or activity, selectable icons or other icons or other interactive objects (which may be implemented, for example, with JavaScript, AJAX, or PHP code), and other elements. As an example and not by way of limitation, the third-party web interface may include selectable icons such as “Like,” “Check-in,” “Eat,” “Recommend,” or other suitable actions or activities. A user viewing the third-party web interface may perform an action by selecting one of the icons (e.g., “Check-in”), causing client system 130 to send a message to social-networking system 160 indicating the user's action. In response to the message, social-networking system 160 may create an edge (e.g., a check-in-type edge) between user node 702 corresponding to the user and concept node 704 corresponding to the third-party web interface or resource, and store edge 706 in one or more data stores.

[0188] In particular embodiments, a pair of nodes in the social graph 700 may be connected to each other via one or more edges 706. An edge 706 connecting a pair of nodes may represent a relationship between the pair of nodes. In particular embodiments, an edge 706 may include or represent one or more data objects or attributes corresponding to the relationship between a pair of nodes. As an example and not by way of limitation, a first user may indicate that a second user is a "friend" of the first user. In response to the indication, the social networking system 160 may send a "friend request" to the second user. If the second user confirms the "friend request", the social networking system 160 may create an edge 706 in the social graph 700 connecting the first user's user node 702 to the second user's user node 702, and store the edge 706 as social graph information in one or more data stores 167. Figure 7In the example of , the social graph 700 includes an edge 706 indicating a friend relationship between the user nodes 702 of user "A" and user "B", and an edge indicating a friend relationship between the user nodes 702 of user "C" and user "B". Although the present disclosure describes or illustrates a particular edge 706 with a particular attribute connecting a particular user node 702, the present disclosure contemplates any suitable edge 706 with any suitable attribute connecting the user nodes 702. As an example and not by way of limitation, the edge 706 may represent a friendship, a family relationship, a business or employment relationship, a fan relationship (including, for example, likes, etc.), a follower relationship, a visitor relationship (including, for example, visits, views, check-ins, shares, etc.), a subscriber relationship, a superior / subordinate relationship, a reciprocal relationship, a non-reciprocal relationship, another suitable type of relationship, or two or more such relationships. In addition, although the present disclosure generally describes nodes as being connected, the present disclosure also describes users or concepts as being connected. Herein, references to connected users or concepts may refer to nodes corresponding to those users or concepts connected by one or more edges 706 in social graph 700 , where appropriate.

[0189] In certain embodiments, an edge 706 between a user node 702 and a concept node 704 may represent a particular action or activity performed by a user associated with the user node 702 toward a concept associated with the concept node 704. By way of example and not limitation, Figure 7 As shown, a user can "Like," "Attend," "Play," "Listen to," "Cook," "Work at," or "Watch" a concept, each of which can correspond to an edge type or subtype. The concept profile interface corresponding to concept node 704 may include, for example, a selectable "Check-in" icon (such as, for example, a clickable "Check-in" icon) or a selectable "Add to Favorites" icon. Similarly, after the user clicks on these icons, social networking system 160 may create a "Favorites" edge or a "Check-in" edge in response to a user action corresponding to the corresponding action. As another example and not by way of limitation, a user (user "C") may use a particular application (SPOTIFY, which is an online music application) to listen to a particular song ("Imagine"). In this case, social networking system 160 may create a "Listen" edge 706 and a "Use" edge (such as, for example, a clickable "Check-in" icon) between user node 702 corresponding to the user and concept nodes 704 corresponding to the song and application. Figure 7 ), to indicate that the user listened to the song and used the application. In addition, social networking system 160 can create a "play" edge 706 (as shown in FIG. 7 ) between concept nodes 704 corresponding to the song and the application. Figure 7as shown), to indicate that a particular song is played by a particular application. In this case, the "played" edge 706 corresponds to an action performed by an external application (Spotify) on an external audio file (the song "Imagine"). Although this disclosure describes a particular edge 706 having particular properties that connects a user node 702 and a concept node 704, this disclosure contemplates any suitable edge 706 having any suitable properties that connects a user node 702 and a concept node 704. Additionally, although this disclosure describes an edge between a user node 702 and a concept node 704 that represents a single relationship, this disclosure contemplates an edge between a user node 702 and a concept node 704 that represents one or more relationships. By way of example and not limitation, the edge 706 can represent that the user likes and uses a particular concept. Alternatively, another edge 706 can represent each type of relationship (or multiple single relationships) between a user node 702 and a concept node 704 (as Figure 7 shown, between the user node 702 of user "E" and the concept node 704 of "Spotify").

[0190] In a particular embodiment, the social networking system 160 can create an edge 706 between a user node 702 and a concept node 704 in the social graph 700. By way of example and not limitation, a user who views a concept profile interface (such as, for example, by using a web browser or a dedicated application hosted by the user's client system 130) can indicate that he or she likes the concept represented by the concept node 704 by clicking or selecting a "like" icon, which can cause the user's client system 130 to send a message to the social networking system 160 indicating that the user likes the concept associated with the concept profile interface. In response to the message, the social networking system 160 can create an edge 706 between the user node 702 associated with the user and the concept node 704, as shown by the "like" edge 706 between the user node and the concept node 704. In a particular embodiment, the social networking system 160 can store the edge 706 in one or more data stores. In a particular embodiment, the edge 706 can be automatically formed by the social networking system 160 in response to a particular user action. By way of example and not limitation, if a first user uploads a picture, watches a movie, or listens to a song, an edge 706 can be formed between the user node 702 corresponding to the first user and the concept nodes 704 corresponding to those concepts. Although this disclosure describes forming a particular edge 706 in a particular manner, this disclosure contemplates forming any suitable edge 706 in any suitable manner.

[0191] Vector Spaces and Embeddings

[0192] Figure 8An example view of vector space 800 is shown. In certain embodiments, objects or n-grams may be represented in a d-dimensional vector space, where d represents any suitable number of dimensions. Although vector space 800 is shown as a three-dimensional space, this is for illustrative purposes only, as vector space 800 may have any suitable dimensions. In certain embodiments, n-grams may be represented in vector space 800 as vectors, which are referred to as term embeddings. Each vector may include coordinates corresponding to a particular point in vector space 800 (i.e., the end point of the vector). By way of example and not limitation, Figure 8 As shown, vectors 810, 820, and 830 can be represented as points in vector space 800. n-grams can be mapped to corresponding vector representations. By way of example and not limitation, by applying a function defined by a dictionary n-gramst 1 and t 2 can be mapped to vectors in vector space 800 respectively and Make and As another example and not by way of limitation, a dictionary trained to map text to a vector representation may be utilized, or such a dictionary itself may be generated by training. As another example and not by way of limitation, a model (e.g., Word2vec) may be used to map n-grams to vector representations in vector space 800. In certain embodiments, n-grams may be mapped to vector representations in vector space 800 using a machine learning model (e.g., a neural network). The machine learning model may have been trained using a sequence of training data (e.g., a corpus of multiple objects each including n-grams).

[0193] In certain embodiments, an object may be represented in vector space 800 as a vector, which is referred to as a feature vector or object embedding. By way of example and not limitation, by applying the function Object e 1 and e 2 can be mapped to vectors in vector space 800 respectively and Make and In certain embodiments, objects may be mapped to vectors based on one or more properties, attributes, or characteristics of the object, the object's relationship to other objects, or any other suitable information associated with the object. By way of example and not limitation, the function Objects can be mapped to vectors by feature extraction, which can start from an initial measurement data set and construct derived values ​​(e.g., features). As an example and not by way of limitation, objects including videos or images can be mapped to vectors by using algorithms to detect or isolate various desired parts or shapes of objects. The features used to calculate the vectors can be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transforms, edge direction, varying intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transforms (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information. As another example and not by way of limitation, objects including audio data can be mapped to vectors based on features (e.g., spectral slope, pitch coefficient, audio spectrum centroid, audio spectrum envelope, Mel-frequency cepstrum, or any other suitable information). In a particular embodiment, when an object has data that is too large to be effectively processed or includes redundant data, the function The transformed reduced feature set (e.g., feature selection) can be used to map objects to vectors. In certain embodiments, the function Objects can be mapped to vectors based on one or more n-grams associated with the object e. Although this disclosure describes representing n-grams or objects in a vector space in a particular manner, this disclosure contemplates representing n-grams or objects in a vector space in any suitable manner.

[0194] In certain embodiments, social networking system 160 may calculate a similarity metric for the vectors in vector space 800. The similarity metric may be cosine similarity, Minkowski distance, Mahalanobis distance, Jaccard similarity coefficient, or any suitable similarity metric. By way of example and not limitation, and The similarity measure can be cosine similarity As another example and not by way of limitation, and The similarity measure can be the Euclidean distance The similarity measure of two vectors may represent how similar two objects or n-grams corresponding to the two vectors, respectively, are to each other, as measured by the distance between the two vectors in vector space 800. As an example and not by way of limitation, vector 810 and vector 820 may correspond to objects that are more similar to each other than objects corresponding to vector 810 and vector 830, based on the distance between the respective vectors. Although this disclosure describes calculating a similarity measure between vectors in a particular manner, this disclosure contemplates calculating a similarity measure between vectors in any suitable manner.

[0195] More information about vector spaces, embeddings, feature vectors, and similarity measures can be found in U.S. patent application Ser. No. 14 / 949,436, filed on Nov. 23, 2015, U.S. patent application Ser. No. 15 / 286,315, filed on Oct. 5, 2016, and U.S. patent application Ser. No. 15 / 365,789, filed on Nov. 30, 2016, each of which is incorporated by reference.

[0196] Artificial Neural Networks

[0197] Fig. 9 An example artificial neural network ("ANN") 900 is shown. In particular embodiments, an ANN may refer to a computational model that includes one or more nodes. The example ANN 900 may include an input layer 910, hidden layers 920, 930, 960, and an output layer 950. Each layer of the ANN 900 may include one or more nodes, such as node 905 or node 915. In particular embodiments, each node of the ANN may be connected to another node of the ANN. As an example and not by way of limitation, each node of the input layer 910 may be connected to one or more nodes of the hidden layer 920. In particular embodiments, one or more nodes may be bias nodes (e.g., nodes in a layer that are not connected to any node in the previous layer and do not receive input from it). In particular embodiments, each node in each layer may be connected to one or more nodes of the previous or next layer. Although Fig. 9 A particular ANN having a particular number of layers, a particular number of nodes, and particular connections between nodes is depicted, but the present disclosure contemplates any suitable ANN having any suitable number of layers, any suitable number of nodes, and any suitable connections between nodes. By way of example and not limitation, although Fig. 9 The connection between each node of the input layer 910 and each node of the hidden layer 920 is depicted, but one or more nodes of the input layer 910 may not be connected to one or more nodes of the hidden layer 920.

[0198] In certain embodiments, the ANN may be a feedforward ANN (e.g., an ANN without cycles or loops, where communication between nodes flows in one direction starting from the input layer and proceeding to successive layers). As an example and not by way of limitation, the input to each node of the hidden layer 920 may include the output of one or more nodes of the input layer 910. As another example and not by way of limitation, the input to each node of the output layer 950 may include the output of one or more nodes of the hidden layer 960. In certain embodiments, the ANN may be a deep neural network (e.g., a neural network including at least two hidden layers). In certain embodiments, the ANN may be a deep residual network. The deep residual network may be a feedforward ANN including hidden layers organized into residual blocks. The input to each residual block after the first residual block may be a function of the output of the previous residual block and the input of the previous residual block. As an example and not by way of limitation, the input to residual block N may be F(x)+x, where F(x) may be the output of residual block N-1 and x may be the input to residual block N-1. Although the present disclosure describes a particular ANN, the present disclosure contemplates any suitable ANN.

[0199] In certain embodiments, an activation function may correspond to each node of the ANN. The activation function of a node may define the output of the node for a given input. In certain embodiments, the input of a node may include a set of inputs. As an example and not by way of limitation, the activation function may be an identity function, a binary step function, a logistic function, or any other suitable function. As another example and not by way of limitation, the activation function of node k may be a sigmoid function Hyperbolic tangent function Rectifier F k (s k )=max(0,s k ) or any other suitable function F k (s k ), where s k can be an effective input to node k. In certain embodiments, the inputs to the activation functions corresponding to the nodes can be weighted. Each node can generate an output using a corresponding activation function based on the weighted inputs. In certain embodiments, each connection between nodes can be associated with a weight. As an example and not by way of limitation, a connection 925 between node 905 and node 915 can have a weighting factor of 0.4, which can indicate that the output of node 905 multiplied by 0.4 is used as an input to node 915. As another example and not by way of limitation, the output y of node k is k It can be k =F k (s k ), where F kcan be the activation function corresponding to node k, s k =∑ j (w jk x j ) can be a valid input to node k, x j can be the output of node j connected to node k, and w jk It can be a weighted coefficient between node j and node k. In a specific embodiment, the input of the node of the input layer can be based on a vector representing an object. Although the present disclosure describes specific inputs and outputs of nodes, the present disclosure contemplates any suitable inputs and outputs of nodes. In addition, although the present disclosure can describe specific connections and weights between nodes, the present disclosure contemplates any suitable connections and weights between nodes.

[0200] In a particular embodiment, the ANN may be trained using training data. As an example and not as a limitation, the training data may include the input and expected output of the ANN 900. As another example and not as a limitation, the training data may include vectors, each vector representing a training object and an expected label for each training object. In a particular embodiment, the training ANN may include modifying the weights associated with the connections between the nodes of the ANN by optimizing the objective function. As an example and not as a limitation, a training method (e.g., a conjugate gradient method, a gradient descent method, a stochastic gradient descent) may be used to back propagate the sum of squared errors (e.g., using a cost function that minimizes the sum of squared errors) as distance measurements between each vector representing the training object. In a particular embodiment, the ANN may be trained using a discarding technique. As an example and not as a limitation, one or more nodes may be temporarily ignored during training (e.g., not receiving input and not generating output). For each training object, one or more nodes of the ANN may have a certain probability of being ignored. The nodes ignored for a particular training object may be different from the nodes ignored for other training objects (e.g., nodes may be temporarily ignored on an object-by-object basis). Although this disclosure describes training the ANN in a particular manner, this disclosure contemplates training the ANN in any suitable manner.

[0201] privacy

[0202] In a particular embodiment, one or more objects (e.g., content or other types of objects) of a computing system may be associated with one or more privacy settings. One or more objects may be stored on or otherwise associated with any suitable computing system or application, such as, for example, a social networking system 160, a client system 130, an assistant system 140, a third-party system 170, a social networking application, an assistant application, a messaging application, a photo sharing application, or any other suitable computing system or application. Although the examples discussed herein are in the context of an online social network, these privacy settings may be applied to any other suitable computing system. The privacy settings (or "access settings") of an object may be stored in any suitable manner, such as, for example, associated with an object, indexed on an authorization server, in another suitable manner, or any suitable combination thereof. Privacy settings about an object may specify how the object (or specific information associated with the object) may be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, executed, displayed, or identified) in an online social network. An object may be described as being "visible" to a particular user or other entity when the privacy settings of the object allow that user or other entity to access the object. As an example and not by way of limitation, a user of an online social network may specify privacy settings on a user profile page that identify a set of users that may access work experience information on the user profile page, thereby excluding other users from accessing that information.

[0203] In certain embodiments, the privacy settings of an object may specify a "blocked list" of users or other entities that should not be allowed to access certain information associated with the object. In certain embodiments, the blacklist may include third-party entities. The blacklist may specify one or more users or entities to which the object is invisible. As an example and not by way of limitation, a user may specify a set of users that may not access an album associated with the user, thereby excluding those users from accessing the album (while also potentially allowing certain users not in the specified set of users to access the album). In certain embodiments, privacy settings may be associated with certain social graph elements. The privacy settings of a social graph element (e.g., a node or edge) may specify how the social graph element, information associated with the social graph element, or an object associated with the social graph element may be accessed using an online social network. As an example and not by way of limitation, a specific concept node 704 corresponding to a specific photo may have a privacy setting that specifies that the photo can only be accessed by the user tagged in the photo and the friends of the user tagged in the photo. In certain embodiments, privacy settings may allow users to opt-in or opt-out of having their content, information, or actions stored / recorded by the social networking system 160 or assistant system 140 or shared with other systems (e.g., third-party systems 170). Although this disclosure describes using particular privacy settings in particular manners, this disclosure contemplates using any suitable privacy settings in any suitable manner.

[0204] In certain embodiments, privacy settings may be based on one or more nodes or edges of the social graph 700. Privacy settings may be specified for one or more edges 706 or edge types of the social graph 700, or for one or more nodes 702, 704 or node types of the social graph 700. A privacy setting applied to a particular edge 706 connecting two nodes may control whether the relationship between two entities corresponding to the two nodes is visible to other users of the online social network. Similarly, a privacy setting applied to a particular node may control whether a user or concept corresponding to the node is visible to other users of the online social network. As an example and not by way of limitation, a first user may share an object to the social networking system 160. The object may be associated with a concept node 704 connected to the user node 702 of the first user by an edge 706. The first user may specify a privacy setting that applies to a particular edge 706 of the concept node 704 connected to the object, or may specify a privacy setting that applies to all edges 706 connected to the concept node 704. As another example and not by way of limitation, the first user may share a collection of objects (e.g., a collection of images) of a particular object type. The first user may specify privacy settings for all objects of that particular object type associated with the first user to have a particular privacy setting (e.g., specifying that all images posted by the first user are visible only to the first user's friends and / or users tagged in the images).

[0205] In certain embodiments, social-networking system 160 may present a "privacy wizard" to the first user (e.g., within a web page, module, one or more dialog boxes, or any other suitable interface) to assist the first user in specifying one or more privacy settings. The privacy wizard may display instructions, appropriate privacy-related information, current privacy settings, one or more input fields for accepting one or more inputs from the first user (which specifies a change or confirmation of a privacy setting), or any suitable combination thereof. In certain embodiments, social-networking system 160 may provide the first user with a "dashboard" function that may display the first user's current privacy settings to the first user. The dashboard function may be displayed to the first user at any appropriate time (e.g., after input from the first user invoking the dashboard function, after a particular event or triggering action occurs). The dashboard function may allow the first user to modify one or more current privacy settings of the first user at any time in any suitable manner (e.g., redirecting the first user to the privacy wizard).

[0206] The privacy settings associated with an object may specify any suitable granularity at which access is allowed or denied. As an example and not by way of limitation, access or denial of access may be specified for specific users (e.g., only me, my roommate, my boss), users within a particular degree of separation (e.g., friends, friends of friends), user groups (e.g., a gaming club, my family), user networks (e.g., employees of a particular employer, students or alumni of a particular university), all users ("public"), no users ("private"), users of third-party systems 170, specific applications (e.g., third-party applications, external websites), other suitable entities, or any suitable combination thereof. Although the present disclosure describes a specific granularity at which access is allowed or denied, the present disclosure contemplates any suitable granularity at which access is allowed or denied.

[0207] In certain embodiments, one or more servers 162 may be authorization / privacy servers for implementing privacy settings. In response to a request from a user (or other entity) for a particular object stored in data storage 164, social networking system 160 may send a request for the object to data storage 164. The request may identify a user associated with the request, and the object may be sent to the user (or the user's client system 130) only if the authorization server determines that the user is authorized to access the object based on the privacy settings associated with the object. If the requesting user is not authorized to access the object, the authorization server may prevent the requested object from being retrieved from data storage 164, or may prevent the requested object from being sent to the user. In a search-query context, an object may be provided as a search result only if the querying user is authorized to access the object, for example, if the privacy settings of the object allow it to be revealed to the querying user, discovered by the querying user, or otherwise visible to the querying user. In certain embodiments, the object may represent content visible to the user through the user's dynamic message. As an example and not by way of limitation, one or more objects may be visible to the user's "Trending" page. In a particular embodiment, an object may correspond to a particular user. An object may be content associated with a particular user, or may be an account of a particular user or information stored on a social networking system 160 or other computing system. As an example and not by way of limitation, a first user may view one or more second users of an online social network through a “People You May Know” feature of the online social network or by viewing a list of friends of the first user. As an example and not by way of limitation, a first user may specify that they do not wish to see objects associated with a particular second user in their dynamic messages or friend lists. If the privacy settings of an object do not allow it to be revealed to the user, discovered by the user, or visible to the user, the object may be excluded from the search results. Although the present disclosure describes implementing privacy settings in a particular manner, the present disclosure contemplates implementing privacy settings in any suitable manner.

[0208] In certain embodiments, different objects of the same type associated with a user may have different privacy settings. Different types of objects associated with a user may have different types of privacy settings. As an example and not as a limitation, a first user may specify that the first user's status update is public, but any images shared by the first user are only visible to the first user's friends on an online social network. As another example and not as a limitation, a user may specify different privacy settings for different types of entities (such as individual users, friends of friends, followers, user groups, or corporate entities). As another example and not as a limitation, a first user may specify a group of users who can view a video posted by the first user while preventing the video from being visible to the first user's employer. In certain embodiments, different privacy settings may be provided for different user groups or user demographics. As an example and not as a limitation, a first user may specify that other users who attend the same university as the first user can view the first user's photos, but other users who are family members of the first user cannot view those same photos.

[0209] In certain embodiments, social networking system 160 may provide one or more default privacy settings for each object of a certain object type. The privacy settings of an object that are set as default may be changed by a user associated with the object. As an example and not by way of limitation, all images posted by a first user may have a default privacy setting of only visible to the first user's friends, and for a certain image, the first user may change the privacy setting of the image to be visible to friends and friends of friends.

[0210] In certain embodiments, privacy settings may allow a first user to specify (e.g., by opting out, by not opting in) whether social-networking system 160 or assistant system 140 may receive, collect, record, or store certain objects or information associated with the user for any purpose. In certain embodiments, privacy settings may allow a first user to specify whether certain applications or processes may access, store, or use certain objects or information associated with the user. Privacy settings may allow a first user to opt in or opt out of having an object or information accessed, stored, or used by a certain application or process. Social-networking system 160 or assistant system 140 may access such information in order to provide certain functionality or services to the first user, while social-networking system 160 or assistant system 140 may not access the information for any other purpose. Prior to accessing, storing, or using such an object or information, social-networking system 160 or assistant system 140 may prompt the user to provide a privacy setting that specifies which applications or processes, if any, may access, store, or use the object or information before any such action is allowed. As an example and not by way of limitation, a first user may transmit a message to a second user via an application associated with an online social network (e.g., a messaging app), and may specify a privacy setting that social networking system 160 or assistant system 140 should not store such messages.

[0211] In certain embodiments, a user may specify whether a particular type of object or information associated with a first user may be accessed, stored, or used by social networking system 160 or assistant system 140. As an example and not by way of limitation, a first user may specify that images sent by the first user via social networking system 160 or assistant system 140 may not be stored by social networking system 160 or assistant system 140. As another example and not by way of limitation, a first user may specify that messages sent from the first user to a particular second user may not be stored by social networking system 160 or assistant system 140. As yet another example and not by way of limitation, a first user may specify that all objects sent via a particular application may be saved by social networking system 160 or assistant system 140.

[0212] In certain embodiments, the privacy settings may allow the first user to specify whether a particular object or information associated with the first user may be accessed from a particular client system 130 or third-party system 170. The privacy settings may allow the first user to opt-in or opt-out of accessing objects or information from a particular device (e.g., a phone book on the user's smartphone), from a particular application (e.g., a messaging app), or from a particular system (e.g., an email server). The social networking system 160 or assistant system 140 may provide a default privacy setting for each device, system, or application, and / or may prompt the first user to specify a specific privacy setting for each context. As an example and not by way of limitation, the first user may utilize a location services feature of the social networking system 160 or assistant system 140 to provide recommendations for restaurants or other places near the user. The first user's default privacy settings may specify that the social networking system 160 or assistant system 140 may use location information provided from the first user's client device 130 to provide location-based services, but the social networking system 160 or assistant system 140 may not store the first user's location information or provide it to any third-party system 170. The first user may then update the privacy settings to allow the third-party image sharing application to use the location information to geotag the photos.

[0213] In certain embodiments, privacy settings may allow a user to specify one or more geographic locations from which an object may be accessed. Access or denial of access to an object may depend on the geographic location of the user attempting to access the object. As an example and not as a limitation, a user may share an object and specify that only users in the same city may access or view the object. As another example and not as a limitation, a first user may share an object and specify that the object is visible to a second user only when the first user is in a particular location. If the first user leaves the particular location, the object may no longer be visible to the second user. As another example and not as a limitation, the first user may specify that the object is visible only to a second user within a threshold distance from the first user. If the first user subsequently changes location, the original second user who could access the object may lose access, while a new group of second users may gain access when they enter within a threshold distance of the first user.

[0214] In certain embodiments, social networking system 160 or assistant system 140 may have functionality that can use a user's personal or biometric information as input for user authentication or experience personalization purposes. Users may choose to utilize these features to enhance their experience on online social networks. As an example and not by way of limitation, a user may provide personal or biometric information to social networking system 160 or assistant system 140. The user's privacy settings may specify that such information may only be used for specific processes (such as authentication) and may also specify that such information may not be shared with any third-party system 170 or may not be used for other processes or applications associated with social networking system 160 or assistant system 140. As another example and not by way of limitation, social networking system 160 may provide a user with the ability to provide a voiceprint recording to an online social network. As an example and not by way of limitation, if a user wishes to utilize this functionality of an online social network, the user may provide a voice recording of his or her own voice to provide a status update on the online social network. The recording of the voice input may be compared to the user's voiceprint to determine what words the user said. The user's privacy settings may specify that such voice recordings may be used only for voice input purposes (e.g., authenticating the user, sending voice messages, improving voice recognition for use of voice manipulation features of the online social network), and further specify that such voice recordings may not be shared with any third-party system 170, or may not be used by other processes or applications associated with the social networking system 160. As another example and not by way of limitation, the social networking system 160 may provide the user with the ability to provide a reference image (e.g., a facial profile, a retinal scan) to the online social network. The online social network may compare the reference image with an image input received later (e.g., for authenticating the user, tagging the user in a photo). The user's privacy settings may specify that such voice recordings may be used only for limited purposes (e.g., authentication, tagging the user in a photo), and further specify that such voice recordings may not be shared with any third-party system 170, or may not be used by other processes or applications associated with the social networking system 160.

[0215] System and method

[0216] Fig.10An example computer system 1000 is shown. In a particular embodiment, one or more computer systems 1000 perform one or more steps of one or more methods described or shown herein. In a particular embodiment, one or more computer systems 1000 provide functionality described or shown herein. In a particular embodiment, software running on one or more computer systems 1000 performs one or more steps of one or more methods described or shown herein, or provides functionality described or shown herein. A particular embodiment includes one or more portions of one or more computer systems 1000. Herein, references to computer systems may include computing devices, and vice versa, where appropriate. In addition, references to computer systems may include one or more computer systems, where appropriate.

[0217] The present disclosure contemplates any suitable number of computer systems 1000. The present disclosure contemplates that computer system 1000 takes any suitable physical form. By way of example and not limitation, computer system 1000 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 1000 may include one or more computer systems 1000; be monolithic or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not as a limitation, one or more computer systems 1000 may perform one or more steps of one or more methods described or shown herein in real time or in batch mode. Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or shown herein at different times or at different locations.

[0218] In a particular embodiment, computer system 1000 includes a processor 1002, a memory 1004, a storage device 1006, an input / output (I / O) interface 1008, a communication interface 1010, and a bus 1012. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0219] In a particular embodiment, the processor 1002 includes hardware for executing instructions (e.g., those that constitute a computer program). As an example and not by way of limitation, to execute instructions, the processor 1002 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 1004, or a storage device 1006; decode them and execute them; and then write one or more results to an internal register, an internal cache, a memory 1004, or a storage device 1006. In a particular embodiment, the processor 1002 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates that the processor 1002 includes any suitable number of any suitable internal caches. As an example and not by way of limitation, the processor 1002 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1004 or the storage device 1006, and the instruction cache may speed up the retrieval of those instructions by the processor 1002. The data in the data cache may be: a copy of data in the memory 1004 or storage device 1006 for operating the instructions executed at the processor 1002; the result of a previous instruction executed at the processor 1002 for access by a subsequent instruction executed at the processor 1002 or for writing to the memory 1004 or storage device 1006; or other suitable data. The data cache may speed up read or write operations performed by the processor 1002. The TLB may speed up virtual address translations for the processor 1002. In a particular embodiment, the processor 1002 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates that the processor 1002 includes any suitable number of any suitable internal registers. Where appropriate, the processor 1002 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1002. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0220] In a particular embodiment, the memory 1004 includes a main memory for storing instructions for the processor 1002 to execute or data for the processor 1002 to operate. As an example and not by way of limitation, the computer system 1000 may load instructions from the storage device 1006 or another source (e.g., another computer system 1000) to the memory 1004. The processor 1002 may then load the instructions from the memory 1004 to an internal register or an internal cache. To execute the instructions, the processor 1002 may retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 1002 may write one or more results (which may be intermediate results or final results) to the internal register or the internal cache. The processor 1002 may then write one or more of these results to the memory 1004. In a particular embodiment, the processor 1002 executes only instructions in one or more internal registers or internal caches or in the memory 1004 (rather than elsewhere in the storage device 1006), and operates only on data in one or more internal registers or internal caches or in the memory 1004 (rather than in the storage device 1006 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 1002 to the memory 1004. As described below, the bus 1012 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 1002 and the memory 1004 and facilitate access to the memory 1004 requested by the processor 1002. In a particular embodiment, the memory 1004 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1004 may include one or more memories 1004. Although this disclosure describes and illustrates particular memories, this disclosure contemplates any suitable memories.

[0221] In a particular embodiment, the storage device 1006 includes a large-capacity storage device for data or instructions. As an example and not as a limitation, the storage device 1006 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the storage device 1006 may include a removable or non-removable (or fixed) medium. Where appropriate, the storage device 1006 may be inside or outside the computer system 1000. In a particular embodiment, the storage device 1006 is a non-volatile solid-state memory. In a particular embodiment, the storage device 1006 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM), or a flash memory, or a combination of two or more of these. The present disclosure contemplates a large-capacity storage device 1006 in any suitable physical form. Where appropriate, storage 1006 may include one or more storage control units that facilitate communications between processor 1002 and storage 1006. Where appropriate, storage 1006 may include one or more storage devices 1006. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0222] In a particular embodiment, I / O interface 1008 includes hardware, software, or both that provide one or more interfaces for communication between computer system 1000 and one or more I / O devices. Where appropriate, computer system 1000 may include one or more of these I / O devices. One or more of these I / O devices may enable communication between a person and computer system 1000. As an example and not as a limitation, I / O devices may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a static camera, a stylus, a tablet computer, a touch screen, a tracking ball, a video camera, another suitable I / O device, or a combination of two or more of these. I / O devices may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1008 for them. Where appropriate, I / O interface 1008 may include one or more devices or software drivers that enable processor 1002 to drive one or more of these I / O devices. Where appropriate, I / O interface 1008 may include one or more I / O interfaces 1008. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0223] In a particular embodiment, the communication interface 1010 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1000 and one or more other computer systems 1000 or one or more networks. As an example and not a limitation, the communication interface 1010 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network (e.g., a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 1010 for it. As an example and not a limitation, the computer system 1000 may communicate with one or more parts of a self-organizing network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1000 may communicate with a wireless PAN (WPAN) (e.g., a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Where appropriate, the computer system 1000 may include any suitable communication interface 1010 for any of these networks. Where appropriate, the communication interface 1010 may include one or more communication interfaces 1010. Although this disclosure describes and illustrates particular communication interfaces, this disclosure contemplates any suitable communication interface.

[0224] In a particular embodiment, bus 1012 includes hardware, software, or both that couples components of computer system 1000 to each other. By way of example and not limitation, bus 1012 may include an accelerated graphics port (AGP) or other graphics bus, an extended industry standard architecture (EISA) bus, a front side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (Extended) (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or any other suitable bus, or a combination of two or more of these. Where appropriate, bus 1012 may include one or more buses 1012. Although the present disclosure describes and illustrates a particular bus, the present disclosure contemplates any suitable bus or interconnect.

[0225] Herein, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital (SECURE DIGITAL) cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, the computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0226] Other Miscellaneous

[0227] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated by context. Furthermore, "and" is both conjunctive and several, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated by context.

[0228] The scope of the present disclosure includes all changes, substitutions, variations, alterations and modifications to the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the corresponding embodiments herein as including specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or replacement of any components, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, references to devices or systems or components of devices or systems that are suitable for, arranged to, capable of, configured to, implemented to, operable to, or operated to perform specific functions in the appended claims include the device, system, component, whether it or that specific function is activated, turned on or unlocked, as long as the device, system or component is adjusted, arranged, enabled, configured, implemented, operable, or operated in this way. In addition, although the present disclosure describes or illustrates specific embodiments as providing specific advantages, specific embodiments may provide some, all or none of these advantages.

Claims

1. A method for contextual auto-completion of an assistant system, comprising: receiving a first user input from a first user from a client system associated with the first user, wherein: The first user input includes a partial request; sending, to the client system, instructions for presenting one or more suggested intent auto-completions corresponding to one or more first candidate hypotheses, the one or more first candidate hypotheses corresponding to the partial request, wherein each of the one or more first candidate hypotheses comprises an intent suggestion, and wherein each suggested intent auto-completion comprises the partial request and a corresponding candidate hypothesis; receiving, from the client system, an indication of a selection by the first user of a first suggested intent auto-completion of the one or more suggested intent auto-completions and a second user input; sending, to the client system, instructions for presenting one or more suggested slot auto-completions corresponding to one or more second candidate hypotheses, the one or more second candidate hypotheses corresponding to the second user input, wherein each of the one or more second candidate hypotheses comprises a slot suggestion, and wherein each suggested slot auto-completion comprises the second user input and a corresponding candidate hypothesis; receiving, from the client system, an indication of a selection by the first user of a first suggested slot auto-completion of the one or more suggested slot auto-completions; and Based on the first suggested intent auto-completion and the first suggested slot auto-completion selected by the first user, one or more tasks are performed via one or more agents.

2. The method according to claim 1, further comprising analyzing the first user input based on a personalized language model to generate the one or more first candidate hypotheses corresponding to the partial request, wherein The analysis included: The first user input is analyzed based on the personalized language model to determine one or more candidate intents.

3. The method according to claim 2, further comprising: sending a request to the client system for additional information from the first user; receiving, from the client system, additional user input from the first user in response to the request; as well as Based on the additional user input, the one or more candidate intents are disambiguated to determine a top candidate intent provided as a suggested intent auto-completion of one of the first candidate hypotheses.

4. The method according to claim 1, further comprising analyzing the second user input based on a personalized language model to generate the one or more second candidate hypotheses corresponding to the second user input, wherein The analysis also includes: The second user input is analyzed based on the personalized language model to determine one or more candidate slots.

5. The method according to claim 4, further comprising: sending a request to the client system for additional information of the first user; receiving, from the client system, additional user input from the first user in response to the request; as well as The one or more candidate slots are disambiguated based on the additional user input to determine a top candidate slot provided as a suggested slot auto-completion for one of the second candidate hypotheses.

6. The method according to claim 2, wherein: The personalized language model is trained based on a plurality of training data, wherein the plurality of training data includes one or more of the following: News Feed posts associated with the first user; dynamic feed comments associated with the first user; messages in one or more messaging interfaces associated with the first user; Data representing one or more domains; a conversation state of one or more conversation sessions associated with the first user; user profile data associated with the user; A task status associated with one or more tasks.

7. The method according to claim 1, wherein: The one or more first candidate hypotheses are ranked based on a dialog state of a dialog session associated with the first user input.

8. The method according to claim 1, wherein: The one or more first candidate hypotheses are respectively associated with one or more confidence scores, wherein the one or more confidence scores are calculated by a personalized language model, and wherein the one or more first candidate hypotheses are ranked based on the confidence scores of the one or more candidate hypotheses.

9. The method according to claim 8, further comprising: receiving additional user input from the client system, wherein the additional user input is appended to the first user input; updating the one or more confidence scores for the one or more first candidate hypotheses based on the additional user input; and The one or more first candidate hypotheses are re-ranked based on the updated confidence scores.

10. The method according to claim 8, further comprising: A sliding window is applied to the first user input, wherein a length of the sliding window determines a percentage of the first user input to be used as a model input for the personalized language model.

11. The method according to claim 10, further comprising: determining whether at least one of the one or more confidence scores associated with the one or more first candidate hypotheses is less than a threshold score; as well as After determining that at least one confidence score is less than the threshold score, adjusting the length of the sliding window.

12. The method according to claim 1, wherein: Analyzing the first user input to generate the one or more first candidate hypotheses corresponding to the partial request is also based on one or more context-specific language models.

13. The method according to claim 12, further comprising: accessing, by a dialog engine, a dialog state of a dialog session associated with the first user input; selecting a particular context-specific language model from the one or more context-specific language models based on the dialog state; and The one or more first candidate hypotheses are generated based on the personalized language model and the selected context-specific model.

14. The method according to claim 12, wherein: The one or more context-specific language models are trained based on context-specific data, the context-specific data comprising one or more of: data associated with the presence of the first user at a particular location; data associated with the first user's interaction with a particular user; or Data associated with registration of the first user for a particular event.

15. The method according to claim 1, further comprising analyzing the first user input based on a personalized language model to generate the one or more first candidate hypotheses corresponding to the partial request, wherein The analysis is also based on one or more global language models.

16. The method according to claim 15, wherein: The one or more global language models are trained based on data associated with a plurality of users of the online social network.

17. The method of claim 1, further comprising analyzing the first user input based on a personalized language model to generate the one or more first candidate hypotheses corresponding to the partial request, wherein The analysis is also based on one or more global context specific language models.

18. The method according to claim 2, wherein: The personalized language model is based on a recurrent neural network.

19. The method according to claim 1, wherein: Each of the one or more first candidate hypotheses corresponds to a subsequent entry associated with the first user input.

20. One or more computer-readable non-transitory storage media embodying software that, when executed, is operable to: receiving a first user input from a first user from a client system associated with the first user, wherein: The first user input includes a partial request; sending, to the client system, instructions for presenting one or more suggested intent auto-completions corresponding to one or more first candidate hypotheses, the one or more first candidate hypotheses corresponding to the partial request, wherein each of the one or more first candidate hypotheses comprises an intent suggestion, and wherein each suggested intent auto-completion comprises the partial request and a corresponding candidate hypothesis; receiving, from the client system, an indication of a selection by the first user of a first suggested intent auto-completion of the one or more suggested intent auto-completions and a second user input; sending, to the client system, instructions for presenting one or more suggested slot auto-completions corresponding to one or more second candidate hypotheses, the one or more second candidate hypotheses corresponding to the second user input, wherein each of the one or more second candidate hypotheses comprises a slot suggestion, and wherein each suggested slot auto-completion comprises the second user input and a corresponding candidate hypothesis; receiving, from the client system, an indication of a selection by the first user of a first suggested slot auto-completion of the one or more suggested slot auto-completions; and Based on the first suggested intent auto-completion and the first suggested slot auto-completion selected by the first user, one or more tasks are performed via one or more agents.

21. A system for contextual auto-completion of an assistant system, comprising: one or more processors; and a non-transitory memory coupled to the processor, the non-transitory memory comprising instructions executable by the processor, the processor being operable, when executing the instructions, to: receiving, from a client system associated with a first user, a first user input from the first user, wherein the first user input comprises a partial request; sending, to the client system, instructions for presenting one or more suggested intent auto-completions corresponding to one or more first candidate hypotheses, the one or more first candidate hypotheses corresponding to the partial request, wherein each of the one or more first candidate hypotheses comprises an intent suggestion, and wherein each suggested intent auto-completion comprises the partial request and a corresponding candidate hypothesis; receiving, from the client system, an indication of a selection by the first user of a first suggested intent auto-completion of the one or more suggested intent auto-completions and a second user input; sending, to the client system, instructions for presenting one or more suggested slot auto-completions corresponding to one or more second candidate hypotheses, the one or more second candidate hypotheses corresponding to the second user input, wherein each of the one or more second candidate hypotheses comprises a slot suggestion, and wherein each suggested slot auto-completion comprises the second user input and a corresponding candidate hypothesis; receiving, from the client system, an indication of a selection by the first user of a first suggested slot auto-completion of the one or more suggested slot auto-completions; and Based on the first suggested intent auto-completion and the first suggested slot auto-completion selected by the first user, one or more tasks are performed via one or more agents.

22. A method for contextual auto-completion of an assistant system, the method being used to help a user obtain information or services by enabling the user to interact with the assistant system in a session using user input to obtain help, wherein: The user input includes voice, text, image or video or any combination thereof, and the assistant system is implemented through a combination of a computing device, an application programming interface (API), and a proliferation of applications on the user device. The method includes, by one or more computing systems: receiving, from a client system associated with a first user, a first user input from the first user, wherein the first user input comprises a partial request; sending, to the client system, instructions for presenting one or more suggested intent auto-completions corresponding to one or more first candidate hypotheses, the one or more first candidate hypotheses corresponding to the partial request, wherein each of the one or more first candidate hypotheses comprises an intent suggestion, and wherein each suggested intent auto-completion comprises the partial request and a corresponding candidate hypothesis; receiving, from the client system, an indication of a selection by the first user of a first suggested intent auto-completion of the one or more suggested intent auto-completions and a second user input; sending, to the client system, instructions for presenting one or more suggested slot auto-completions corresponding to one or more second candidate hypotheses, the one or more second candidate hypotheses corresponding to the second user input, wherein each of the one or more second candidate hypotheses comprises a slot suggestion, and wherein each suggested slot auto-completion comprises the second user input and a corresponding candidate hypothesis; receiving, from the client system, an indication of a selection by the first user of a first suggested slot auto-completion of the one or more suggested slot auto-completions; and Based on the first suggested intent auto-completion and the first suggested slot auto-completion selected by the first user, one or more tasks are performed via one or more agents.

23. The method of claim 22, further comprising analyzing the first user input based on a personalized language model to generate the one or more first candidate hypotheses corresponding to the partial request, wherein The analysis included: The first user input is analyzed based on the personalized language model to determine one or more candidate intents.

24. The method according to claim 23, further comprising: sending a request to the client system for additional information from the first user; receiving, from the client system, additional user input from the first user in response to the request; as well as Based on the additional user input, the one or more candidate intents are disambiguated to determine a top candidate intent provided as a suggested intent auto-completion of one of the first candidate hypotheses.

25. The method according to any one of claims 22-24, further comprising analyzing the second user input based on a personalized language model to generate the one or more second candidate hypotheses corresponding to the second user input, wherein: The analysis also includes: Based on the personalized language model, the second user input is analyzed to determine one or more candidate slots.

26. The method according to claim 25, further comprising: sending a request to the client system for additional information of the first user; receiving, from the client system, additional user input from the first user in response to the request; as well as The one or more candidate slots are disambiguated based on the additional user input to determine a top candidate slot provided as a suggested slot auto-completion for one of the second candidate hypotheses.

27. The method according to any one of claims 23-24 and 26, wherein: The personalized language model is trained based on a plurality of training data, wherein the plurality of training data includes one or more of the following: News Feed posts associated with the first user; dynamic feed comments associated with the first user; messages in one or more messaging interfaces associated with the first user; Data representing one or more domains; a conversation state of one or more conversation sessions associated with the first user; user profile data associated with the user; A task status associated with one or more tasks.

28. The method according to any one of claims 22 to 24 and 26, wherein: The one or more first candidate hypotheses are ranked based on a dialog state of a dialog session associated with the first user input.

29. The method according to any one of claims 22 to 24 and 26, wherein: The one or more first candidate hypotheses are respectively associated with one or more confidence scores, wherein the one or more confidence scores are calculated by a personalized language model, and wherein the one or more first candidate hypotheses are ranked based on the confidence scores of the one or more candidate hypotheses.

30. The method of claim 29, further comprising: receiving additional user input from the client system, wherein the additional user input is appended to the first user input; updating the one or more confidence scores for the one or more first candidate hypotheses based on the additional user input; and re-ranking the one or more first candidate hypotheses based on the updated confidence scores; and / or The method further comprises: applying a sliding window to the first user input, wherein a length of the sliding window determines a percentage of the first user input to be used as a model input for the personalized language model; The method further comprises: determining whether at least one of the one or more confidence scores associated with the one or more first candidate hypotheses is less than a threshold score; and After determining that at least one confidence score is less than the threshold score, adjusting the length of the sliding window.

31. The method according to any one of claims 22 to 24, 26 and 30, wherein: Analyzing the first user input to generate the one or more first candidate hypotheses corresponding to the partial request is also based on one or more context-specific language models.

32. The method of claim 31 further comprising: accessing, by a dialog engine, a dialog state of a dialog session associated with the user input; selecting a particular context-specific language model from the one or more context-specific language models based on the dialog state; and generating one or more first candidate hypotheses based on the personalized language model and the selected context-specific model; and / or The one or more context-specific language models are trained based on context-specific data, the context-specific data comprising one or more of the following: data associated with the presence of the first user at a particular location; data associated with the first user's interaction with a particular user; or Data associated with registration of the first user for a particular event.

33. The method of any one of claims 22-24, 26, 30, and 32, further comprising analyzing the first user input based on a personalized language model to generate the one or more first candidate hypotheses corresponding to the partial request is also based on one or more global language models; Wherein the one or more global language models are trained based on data associated with a plurality of users of the online social network.

34. The method of any one of claims 22-24, 26, 30 and 32, further comprising analyzing the first user input based on a personalized language model to generate the one or more first candidate hypotheses corresponding to the partial request also based on one or more global context-specific language models.

35. The method according to any one of claims 23-24, 26, 30 and 32, wherein: The personalized language model is based on a recurrent neural network.

36. The method according to any one of claims 22-24, 26, 30 and 32, wherein: Each of the one or more candidate hypotheses corresponds to a subsequent entry associated with the first user input.

37. One or more computer-readable non-transitory storage media embodying software which, when executed, is operable to perform the method according to any one of claims 22 to 36.

38. An assistant system for assisting a user in obtaining information or services by enabling the user to interact with the assistant system in a session using user input to obtain assistance, wherein: The user input includes sound, text, image or video or any combination thereof, and the assistant system is implemented by a combination of a computing device, an application programming interface (API), and a proliferation of applications on a user device, and the system includes: one or more processors; and a non-volatile memory coupled to the processor, the non-volatile memory including instructions executable by the processor, and the processor is operable to perform a method according to any one of claims 22 to 36 when executing the instructions.

39. The assistant system of claim 38, configured to assist a user by performing at least one or more of the following features or steps: - Create and store a user profile that includes personal and contextual information associated with the user - analyzing the user input using natural language understanding, wherein, The analysis can be based on the user profile to gain a more personalized and context-aware understanding - resolving entities associated with the user input based on the analysis -Interact with different agents to obtain information or services associated with the resolved entity - Generate responses for users regarding said information or services by using natural language generation - Manage and forward conversation flows with users through interaction with them using conversation management techniques - Help users digest the information they have obtained effectively and efficiently by aggregating the information - Helping users better participate in online social networks by providing tools that help users interact with online social networks - Help users manage different tasks, - Proactively perform pre-authorized tasks related to the user's interests and preferences based on the user profile at a time relevant to the user and without user input - Checking privacy settings whenever necessary to ensure that accessing a user profile and performing different tasks adheres to the user's privacy settings.

40. The assistant system according to claim 38 or 39, comprising at least one or more of the following components: a messaging platform for receiving a text-based user input and / or receiving an image or video-based user input from a client system associated with a user and processing the image or video-based user input within the messaging platform using optical character recognition techniques to convert the user input into text, - an audio speech recognition (ASR) module for receiving a user input based on an audio modality from said client system associated with a user and converting said user input based on said audio modality into text, - an assistant xbot for receiving the output of said messaging platform or said ASR module.

41. A system for contextual auto-completion of an assistant system, comprising: At least one client system, At least one assistant system according to any one of claims 38 to 40, The client system and the assistant system are connected to each other via a network, wherein the client system includes an assistant application for allowing a user of the client system to interact with the assistant system, wherein the assistant application transmits user input to the assistant system, and based on the user input, the assistant system generates a response and sends the generated response to the assistant application, and the assistant application presents the response to the user of the client system, Wherein, the user input is audio or verbal, and the response can be text or can also be audio or verbal.

42. The system of claim 41 further comprising a social networking system, in, The client system includes a social networking application for accessing the social networking system.

Images