Processing multi-modal user input for assistant system

Abstract

The invention relates to processing multi-modal user input for an assistant system. In one embodiment, a method includes receiving a user input based on one or more modalities from a client system associated with a first user, where at least one modality is a visual modality, identifying one or more subjects associated with the user input based on the visual modalities based on one or more machine learning models, and transmitting the identified subjects to the client system. Determining one or more attributes associated with the one or more subjects based on the one or more machine learning models, respectively, parsing one or more entities corresponding to the one or more subjects based on the determined one or more attributes, one or more parsed entities are provided, one or more tasks associated with the one or more parsed entities are executed, and instructions are sent to a client system associated with the first user to present communication content in response to user input, the communication content including information associated with the executed one or more tasks.

Description

[0001] This application is a divisional application of an application filed on August 3, 2018, with application number 201880094828.4 and invention name “Processing multimodal user input for an assistant system”. Technical Field

[0002] The present disclosure relates generally to database and file management within a network environment, and particularly to hardware and software for a smart assistant system. background

[0004] The assistant system can provide information or services on behalf of the user based on a combination of user input, location awareness, and the ability to access information from various online sources (such as weather conditions, traffic congestion, news, stock prices, user schedules, retail prices, etc.). User input can include text (e.g., online chat) (especially text in instant messaging applications or other applications), voice, images, or a combination thereof. The assistant system can perform concierge-type services (e.g., making dinner reservations, purchasing event tickets, arranging travel) or provide information based on user input. The assistant system can also perform management or data processing tasks based on online information and events without user initiation or interaction. Examples of those tasks that can be performed by the assistant system can include schedule management (e.g., sending a warning message to a dinner appointment that the user is going to be late due to traffic conditions, updating the schedules of both parties, and changing restaurant reservation times). The assistant system can be implemented through a combination of computing devices, application programming interfaces (APIs), and application proliferation on user devices.

[0005] A social networking system, which may include a social networking website, may enable its users (e.g., individuals or organizations) to interact with it and with each other through it. The social networking system may utilize input from a user to create and store a user profile associated with the user in the social networking system. The user profile may include demographic information, communication channel information, and information about personal interests of the user. The social networking system may also utilize input from the user to create and store a record of the user's relationships with other users of the social networking system, as well as provide services (e.g., profile / news feed postings, photo sharing, event organization, messaging, games, or advertising) to facilitate social interaction between or among users.

[0006] The social networking system may send content or messages related to its services to the user's mobile device or other computing device through one or more networks. The user may also install a software application on the user's mobile device or other computing device for accessing the user's user profile and other data within the social networking system. The social networking system may generate a set of personalized content objects to display to the user, such as a dynamic message of aggregated stories of other users connected to the user.

[0007] Overview of Certain Embodiments

[0008] In a particular embodiment, the assistant system can help users obtain information or services. The assistant system can enable users to interact with it through multimodal user input (such as sound, text, image, video) in stateful and multi-turn conversations to obtain help. The assistant system can create and store a user profile, which includes personal information and contextual information associated with the user. In a particular embodiment, the assistant system can use natural language understanding to analyze user input. The analysis can be based on the user profile to obtain a more personalized and context-aware understanding. The assistant system can resolve entities associated with user input based on the analysis. In a particular embodiment, the assistant system can interact with different agents to obtain information or services associated with the resolved entity. The assistant system can generate responses about information or services for users by using natural language generation. Through interaction with the user, the assistant system can use dialogue management technology to manage and forward the conversation flow with the user. In a particular embodiment, the assistant system can also assist users to digest the information obtained effectively and efficiently by summarizing the information. The assistant system can also help users better participate in online social networks by providing tools that help users interact with online social networks (e.g., create posts, comments, messages). The assistant system can additionally help users manage different tasks, such as keeping track of events. In certain embodiments, the assistant system can proactively perform tasks related to the user's interests and preferences based on the user profile without user input. In certain embodiments, the assistant system can check privacy settings to ensure that access to the user's profile or other user information and the performance of different tasks are allowed according to the user's privacy settings.

[0009] In a particular embodiment, the assistant system may receive multimodal user input (e.g., sound, image, video, text) from a client system associated with the user. The assistant system may process the multimodal user input using different modules of the assistant system based on the different modalities of the user input. The assistant system may then understand the user input based on an entity resolution module and a co-reference module, i.e., identify the subject / entity and semantic meaning of the user input. The assistant system may also generate multimodal output based on understanding and send the multimodal output to the user via the client system. As an example and not as a limitation, a user may submit a picture of several people to the assistant system and request the assistant system to initiate a call to one of them (e.g., the user may say "call the guy on the left"). The assistant system may determine the identities of these people, identify the person "on the left", access the contact information of the person on the left (subject to the privacy settings of the person), and initiate a call for the user accordingly. In particular, the assistant system may use a visual recognition agent capable of processing visual input contained in the multimodal user input, wherein the visual recognition agent may access various computer vision models to understand the visual input. The co-reference module can use the understanding of visual input to resolve entities associated with user input. Based on the understanding of visual input and the resolved entities, the assistant system can use the assistant xbot to have a conversation with the user to further enhance the user's experience of the assistant system by performing different tasks corresponding to the visual input. The assistant system can reactively process multimodal user input in response to user queries. The assistant system can also actively process multimodal user input by providing suggestions based on user input. The ability to process multimodal user input and seamlessly switch between different output modes makes the assistant system well adapted to various client systems and software, making the assistant system more useful and attractive to users. Although the present disclosure describes processing a specific multimodal user input via a specific system in a specific manner, the present disclosure contemplates processing any suitable multimodal user input via any suitable system in any suitable manner.

[0010] In a particular embodiment, the assistant system may receive user input based on one or more modalities from a client system associated with a first user. At least one of the modalities of the user input may be a visual modality. In a particular embodiment, the assistant system may identify one or more subjects associated with the user input based on one or more machine learning models and based on a visual modality. In a particular embodiment, the assistant system may determine one or more attributes associated with one or more subjects, respectively, based on one or more machine learning models. The assistant system may then resolve one or more entities corresponding to the one or more subjects based on the determined one or more attributes. In a particular embodiment, the assistant system may perform one or more tasks associated with one or more resolved entities. The assistant system may also send instructions to a client system associated with the first user for presenting communication content in response to user input. The communication content may include information associated with one or more tasks performed.

[0011] There are certain technical challenges in achieving the goal of processing and understanding multimodal user input. One technical challenge includes accurately identifying the subject and its attributes from the visual input. The solution proposed by the embodiments disclosed herein to solve the above challenges is a machine learning model based on facial recognition and object detection, which is effective for identifying people, places, businesses and objects and their attributes. Another technical challenge includes accurately parsing entities from multimodal user input. The solution proposed by the embodiments disclosed herein to solve this challenge is a co-referencing module in the assistant system, which can link the wording information of the sound and / or text input with the visual analysis results of the visual input, so as to accurately parse the entities of the multimodal user input. Another technical challenge includes responding to the multimodal user input with communication content based on the appropriate modality. The solution proposed by the embodiments disclosed herein to solve this challenge is to determine the appropriate modality based on the context information associated with the user and the context information associated with the client system associated with the user, which allows the assistant system to communicate with the user based on the determined modality that may be more suitable for the user's current situation.

[0012] Certain embodiments disclosed herein may provide one or more technical advantages. The technical advantages of these embodiments may include the ability to process multimodal user input and switch between different output modes to effectively adapt to various client systems and software. Another technical advantage of these embodiments may include an improved understanding of multimodal user input when voice and / or text input and visual input are jointly analyzed via a co-referencing module. Another technical advantage of these embodiments may include increasing the degree of user participation in the assistant system by actively suggesting tasks to the user based on the analysis of visual input. Certain embodiments disclosed herein may not provide, provide some or all of the above technical advantages. In view of the drawings, description and claims of the present disclosure, one or more other technical advantages are obvious to those skilled in the art.

[0013] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to them. Specific embodiments may include all, some or none of the components, elements, features, functions, operations or steps of the embodiments disclosed herein. Embodiments according to the present invention are specifically disclosed in the attached claims relating to methods, storage media, assistant systems, systems and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or back references in the attached claims are selected only for formal reasons. However, any subject matter generated by intentional back references (especially multiple references) to any previous claim may also be claimed, so that any combination of claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features as set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any features of the appended claims.

[0014] In an embodiment according to the invention, a method may include, by one or more computing systems:

[0015] receiving, from a client system associated with a first user, user input based on one or more modalities, wherein at least one of the modalities of the user input is a visual modality;

[0016] identifying, based on the one or more machine learning models, one or more subjects associated with the user input based on the visual modality;

[0017] Based on the one or more machine learning models, determining one or more attributes associated with the one or more subjects, respectively;

[0018] resolving one or more entities corresponding to the one or more subjects based on the determined one or more attributes;

[0019] performing one or more tasks associated with the one or more resolved entities; and

[0020] Instructions are sent to a client system associated with the first user to present communications content in response to the user input, wherein the communications content includes information associated with the one or more tasks performed.

[0021] User input may include one or more of the following:

[0022] String;

[0023] Audio editing;

[0024] images; or

[0025] Video editing.

[0026] The one or more subjects associated with the user input may include one or more of a person, a place, a business, or an object.

[0027] Identifying one or more individuals may be based on facial recognition.

[0028] Identifying one or more objects may be based on object detection.

[0029] In an embodiment according to the invention, a method may include generating a feature representation for a user input based on a visual modality.

[0030] The one or more machine learning models may include one or more of the following:

[0031] Support vector machines;

[0032] regression model; or

[0033] Convolutional Neural Networks.

[0034] In an embodiment according to the invention, a method may include identifying one or more intents and one or more slots based on user input.

[0035] One or more tasks associated with the one or more resolved entities may be performed based on the identified intents and slots.

[0036] The communication content may include one or more of the following:

[0037] String;

[0038] Audio editing;

[0039] images; or

[0040] Video editing.

[0041] In an embodiment according to the invention, a method may include determining one or more modalities of communication content.

[0042] Determining one or more modalities of the communication content may include:

[0043] identifying contextual information associated with the first user;

[0044] identifying contextual information associated with the client system; and

[0045] One or more modalities are determined based on context information associated with the first user and context information associated with the client system.

[0046] In an embodiment according to the present invention, a method may include:

[0047] generating multiple tasks based on a visual modality of user input; and

[0048] A user selection of one or more tasks from the plurality of tasks by the first user is received from a client system associated with the first user.

[0049] In an embodiment according to the invention, a method may include storing one or more identified principals in a conversation state.

[0050] User input may include user interaction with a media content object.

[0051] In an embodiment according to the invention, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to:

[0052] receiving, from a client system associated with a first user, user input based on one or more modalities, wherein at least one of the modalities of the user input is a visual modality;

[0053] identifying, based on the one or more machine learning models, one or more subjects associated with the user input based on the visual modality;

[0054] Based on the one or more machine learning models, determining one or more attributes associated with the one or more subjects, respectively;

[0055] resolving one or more entities corresponding to the one or more subjects based on the determined one or more attributes;

[0056] performing one or more tasks associated with the one or more resolved entities; and

[0057] Instructions are sent to a client system associated with the first user to present communications content in response to the user input, wherein the communications content includes information associated with the one or more tasks performed.

[0058] In an embodiment according to the present invention, a system may include: one or more processors; and a non-transitory memory coupled to the processor, the non-transitory memory including instructions executable by the processor, the processor being operable to:

[0059] receiving, from a client system associated with a first user, user input based on one or more modalities, wherein at least one of the modalities of the user input is a visual modality;

[0060] identifying, based on the one or more machine learning models, one or more subjects associated with the user input based on the visual modality;

[0061] Based on the one or more machine learning models, determining one or more attributes associated with the one or more subjects, respectively;

[0062] resolving one or more entities corresponding to the one or more subjects based on the determined one or more attributes;

[0063] performing one or more tasks associated with the one or more resolved entities; and

[0064] Instructions are sent to a client system associated with the first user to present communications content in response to the user input, wherein the communications content includes information associated with the one or more tasks performed.

[0065] In an embodiment according to the present invention, a method, in particular a method used in an assistant system, for helping a user obtain information or services by enabling the user to interact with the assistant system in a conversation using user input to obtain help, wherein the user input includes sound, text, image or video or any combination thereof, the assistant system is particularly implemented by a combination of a computing device, an application programming interface (API) and application proliferation on a user device, the method may include one or more computing systems:

[0066] receiving, from a client system associated with a first user, user input based on one or more modalities, wherein at least one of the modalities of the user input is a visual modality;

[0067] identifying, based on the one or more machine learning models, one or more subjects associated with the user input based on the visual modality;

[0068] Based on the one or more machine learning models, determining one or more attributes associated with the one or more subjects, respectively;

[0069] resolving one or more entities corresponding to the one or more subjects based on the determined one or more attributes;

[0070] performing one or more tasks associated with the one or more resolved entities; and

[0071] Instructions are sent to a client system associated with the first user to present communications content in response to the user input, wherein the communications content includes information associated with the one or more tasks performed.

[0072] In an embodiment according to the present invention, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to perform a method according to the present invention or any of the above-mentioned embodiments.

[0073] In an embodiment according to the present invention, a system may include: one or more processors; and at least one memory, which is coupled to the processor and includes instructions executable by the processor, and the processor is operable to perform a method according to the present invention or any of the above-mentioned embodiments when executing the instructions.

[0074] In an embodiment according to the present invention, a computer program product, preferably comprising a computer-readable non-transitory storage medium, is operable to perform a method according to the present invention or any of the above-mentioned embodiments when executed on a data processing system.

[0075] In an embodiment according to the present invention, an assistant system is provided for helping a user obtain information or services by enabling the user to interact with the assistant system in a conversation using user input to obtain help, wherein the user input includes sound, text, image or video or any combination thereof, and the assistant system is particularly implemented through a combination of a computing device, an application programming interface (API), and a proliferation of applications on a user device, the system comprising: one or more processors; and a non-volatile memory coupled to the processor, the non-volatile memory comprising instructions executable by the processor, and the processor is operable to perform a method according to the present invention or any of the above-mentioned embodiments when executing the instructions.

[0076] In an embodiment according to the present invention, the assistant system may assist the user by performing at least one or more of the following features or steps:

[0077] - Create and store a user profile that includes personal and contextual information associated with the user

[0078] - Use natural language understanding to analyze user input, where the analysis can be based on user profiles for a more personalized and context-aware understanding

[0079] - Resolve entities associated with user input based on analysis

[0080] -Interact with different agents to obtain information or services associated with the resolved entity

[0081] - Generate responses for users regarding information or services by using natural language generation

[0082] - Manage and forward conversation flows with users through interaction with them using conversation management techniques

[0083] -Help users digest the information they have obtained effectively and efficiently by aggregating the information

[0084] - Help users better participate in online social networks by providing tools that help users interact with online social networks (e.g., create posts, comments, messages)

[0085] - Helps users manage different tasks, such as keeping track of events

[0086] - Proactively perform pre-authorized tasks relevant to the user’s interests and preferences based on the user profile at a time relevant to the user and without user input

[0087] - Checking privacy settings whenever necessary to ensure that accessing a user profile and performing different tasks adheres to the user's privacy settings.

[0088] In an embodiment according to the present invention, the assistant system may include at least one or more of the following components:

[0089] a messaging platform for receiving a text-based user input from a client system associated with a user, and / or for receiving an image or video-based user input and processing the image or video-based user input within the messaging platform using optical character recognition techniques to convert the user input into text,

[0090] an audio speech recognition (ASR) module for receiving an audio modality-based user input from a client system associated with a user (e.g., the user may speak or send a video including voice) and converting the audio modality-based user input into text,

[0091] - Assistant xbot, which is used to receive the output of the messaging platform or the ASR module.

[0092] In an embodiment according to the present invention, a system may include:

[0093] at least one client system (130), in particular an electronic device,

[0094] At least one assistant system (140) according to any one of claims 29 to 31,

[0095] The client system and the assistant system are connected to each other in particular via a network (110),

[0096] wherein the client system includes an assistant application (136) for allowing a user at the client system (130) to interact with the assistant system (140),

[0097] wherein the assistant application (136) transmits the user input to the assistant system (140), and based on the user input, the assistant system (140) generates a response and sends the generated response to the assistant application (136), and the assistant application (136) presents the response to the user at the client system (130),

[0098] In particular, the user input is audio or verbal, and the response may be textual or also audio or verbal.

[0099] In an embodiment according to the present invention, the system may include a social networking system (160),

[0100] The client system specifically includes a social networking application (134) for accessing a social networking system (160). BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 An example network environment associated with an assistant system is shown.

[0103] Figure 2 An example architecture for an assistant system is shown.

[0104] Figure 3 An example flow chart of an assistant system responding to a user request is shown.

[0105] Figure 4 Shown based on Figure 2An example flowchart for handling multimodal user input using an example architecture for an assistant system in .

[0106] Figure 5 An example interaction between a user and an assistant system via multimodal user input and system output is shown.

[0107] Figure 6 An example method for processing multimodal user input is shown.

[0108] Figure 7 An example social graph is shown.

[0109] Figure 8 An example view of the embedding space is shown.

[0110] Fig. 9 An example artificial neural network is shown.

[0111] Fig.10 An example computer system is shown.

[0112] Description of Example Embodiments

[0113] Systematic review

[0114] Figure 1 An example network environment 100 associated with an assistant system is shown. The network environment 100 includes a client system 130, an assistant system 140, a social networking system 160, and a third-party system 170 connected to each other via a network 110. Figure 1 A particular arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110 is shown, but the present disclosure contemplates any suitable arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110. By way of example and not limitation, two or more of client system 130, social networking system 160, assistant system 140, and third-party system 170 may be directly connected to each other, bypassing network 110. As another example, two or more of client system 130, assistant system 140, social networking system 160, and third-party system 170 may be physically or logically co-located with each other, in whole or in part. Furthermore, although Figure 1A particular number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110 are shown, but this disclosure contemplates any suitable number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110. By way of example and not limitation, network environment 100 may include a plurality of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110.

[0115] The present disclosure contemplates any suitable network 110. By way of example and not limitation, one or more portions of network 110 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 110 may include one or more networks 110.

[0116] Links 150 may connect client systems 130, assistant systems 140, social networking systems 160, and third-party systems 170 to communication network 110 or to each other. The present disclosure contemplates any suitable links 150. In particular embodiments, one or more links 150 include one or more wired (such as, for example, digital subscriber lines (DSL) or cable-based data service interface specifications (DOCSIS)) links, wireless (such as, for example, Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or optical (such as, for example, synchronous optical network (SONET) or synchronous digital hierarchy (SDH)) links. In particular embodiments, one or more links 150 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 150, or a combination of two or more such links 150. Links 150 need not necessarily be the same throughout network environment 100. The one or more first links 150 may differ from the one or more second links 150 in one or more aspects.

[0117] In certain embodiments, client system 130 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more such components, and is capable of performing appropriate functions implemented or supported by client system 130. By way of example and not limitation, client system 130 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a smart speaker, other suitable electronic devices, or any suitable combination thereof. In certain embodiments, client system 130 may be a smart assistant device. More information about intelligent assistant devices can be found in U.S. Patent Application No. 15 / 949011, filed on April 9, 2018, U.S. Patent Application No. 62 / 655751, filed on April 10, 2018, U.S. Patent Application No. 29 / 631910, filed on January 3, 2018, U.S. Patent Application No. 29 / 631747, filed on January 2, 2018, U.S. Patent Application No. 29 / 631913, filed on January 3, 2018, and U.S. Patent Application No. 29 / 631914, filed on January 3, 2018, which are incorporated by reference. The present disclosure contemplates any suitable client system 130. Client system 130 can enable network users at client system 130 to access network 110. Client system 130 can enable its users to communicate with other users at other client systems 130.

[0118] In a particular embodiment, the client system 130 may include a web browser 132, such as MICROSOFTINTERNET EXPLORER, GOOGLE CHROME or MOZILLA FIREFOX, and may have one or more add-ons, plug-ins or other extensions, such as TOOLBAR or YAHOO TOOLBAR. A user at the client system 130 may enter a uniform resource locator (URL) or other address that directs the web browser 132 to a specific server (such as server 162 or a server associated with a third-party system 170), and the web browser 132 may generate a hypertext transfer protocol (HTTP) request and pass the HTTP request to the server. The server may accept the HTTP request and pass one or more hypertext markup language (HTML) files to the client system 130 in response to the HTTP request. The client system 130 may display a web interface (such as a web page) based on an HTML file from a server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not as a limitation, a web interface may be displayed according to an HTML file, an extensible hypertext markup language (XHTML) file or an extensible markup language (XML) file according to specific needs. Such an interface may also execute scripts, such as, for example and without limitation, scripts written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, a combination of markup language and script (e.g., AJAX (Asynchronous JAVASCRIPT and XML)), etc. Herein, references to a web interface include one or more corresponding source files (which a browser may use to render the web interface), and vice versa, where appropriate.

[0119] In certain embodiments, client system 130 may include social network application 134 installed on client system 130. A user at client system 130 may use social network application 134 to access an online social network. A user at client system 130 may use social network application 134 to communicate with the user's social connections (e.g., friends, followers, followed accounts, contacts, etc.). A user at client system 130 may also use social network application 134 to interact with a plurality of content objects (e.g., posts, news articles, temporary content, etc.) on an online social network. As an example and not by way of limitation, a user may use social network application 134 to browse trending topics and breaking news.

[0120] In certain embodiments, the client system 130 may include an assistant application 136. The user of the client system 130 may use the assistant application 136 to interact with the assistant system 140. In certain embodiments, the assistant application 136 may include an independent application. In certain embodiments, the assistant application 136 may be integrated into the social network application 134 or another suitable application (e.g., a messaging application). In certain embodiments, the assistant application 136 may also be integrated into the client system 130, the assistant hardware device, or any other suitable hardware device. In certain embodiments, the assistant application 136 may be accessed via a web browser 132. In certain embodiments, the user may provide input via different modalities. As an example and not as a limitation, the modality may include audio, text, image, video, etc. The assistant application 136 may transmit the user input to the assistant system 140. Based on the user input, the assistant system 140 may generate a response. The assistant system 140 may send the generated response to the assistant application 136. Then, the assistant application 136 may present the response to the user of the client system 130. The presented response may be based on different modalities, such as audio, text, image, and video. As an example and not by way of limitation, a user may verbally ask assistant application 136 for traffic information (i.e., via an audio modality). Assistant application 136 may then transmit the request to assistant system 140. Assistant system 140 may generate results accordingly and send them back to assistant application 136. Assistant application 136 may also present the results to the user in text.

[0121] In a particular embodiment, the assistant system 140 can help the user retrieve information from different sources. The assistant system 140 can also help the user request services from different service providers. In a particular embodiment, the assistant system 140 can receive a user's request for information or services via the assistant application 136 in the client system 130. The assistant system 140 can use natural language understanding to analyze the user request based on the user profile and other relevant information. The results of the analysis may include different entities associated with online social networks. The assistant system 140 can then retrieve information or request services associated with these entities. In a particular embodiment, when retrieving information or requesting services for the user, the assistant system 140 can interact with the social networking system 160 and / or the third-party system 170. In a particular embodiment, the assistant system 140 can generate personalized communication content for the user using natural language generation technology. The personalized communication content may include, for example, the status of the retrieved information or the requested service. In a particular embodiment, the assistant system 140 can enable the user to interact with it about information or services in a stateful and multi-round conversation by using dialogue management technology. In the following Figure 2 The functionality of assistant system 140 is described in more detail in the discussion of .

[0122] In certain embodiments, social networking system 160 may be a network-addressable computing system that may host an online social network. Social networking system 160 may generate, store, receive, and transmit social networking data (such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to an online social network). Social networking system 160 may be accessed by other components of network environment 100 directly or via network 110. As an example and not by way of limitation, client system 130 may access social networking system 160 directly or via network 110 using a web browser 132 or a native application associated with social networking system 160 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In certain embodiments, social networking system 160 may include one or more servers 162. Each server 162 may be a unitary server or a distributed server across multiple computers or multiple data centers. The server 162 may be of various types, such as, for example and without limitation, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 162 may include hardware, software, or embedded logic components for performing the appropriate functions implemented or supported by the server 162, or a combination of two or more such components. In a particular embodiment, the social networking system 160 may include one or more data storage devices 164. The data storage device 164 may be used to store various types of information. In a particular embodiment, the information stored in the data storage device 164 may be organized according to a particular data structure. In a particular embodiment, each data storage device 164 may be a relational database, a columnar database, a correlation database, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. Particular embodiments may provide an interface that enables client system 130 , social networking system 160 , or third-party system 170 to manage, retrieve, modify, add, or delete information stored in data store 164 .

[0123] In particular embodiments, social networking system 160 may store one or more social graphs in one or more data stores 164. In particular embodiments, a social graph may include a plurality of nodes, which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept), and a plurality of edges connecting the nodes. Social networking system 160 may provide users of an online social network with the ability to communicate and interact with other users. In particular embodiments, a user may join an online social network via social networking system 160 and then add connections (e.g., relationships) with a plurality of other users in social networking system 160 to whom they want to be connected. Herein, the term "friend" may refer to any other user of social networking system 160 with whom the user has formed a connection, association, or relationship via social networking system 160.

[0124] In particular embodiments, social-networking system 160 may provide users with the ability to take actions on various types of items or objects supported by social-networking system 160. By way of example and not limitation, items and objects may include groups or social networks to which users of social-networking system 160 may belong, events or calendar entries that may be of interest to users, computer-based applications that users may use, transactions that allow users to buy or sell goods via a service, interactions with advertisements that users may perform, or other suitable items or objects. Users may interact with anything that can be represented in social-networking system 160 or by an external system such as third-party system 170 that is separate from social-networking system 160 and coupled to social-networking system 160 via network 110.

[0125] In certain embodiments, social networking system 160 can link various entities. As an example and not by way of limitation, social networking system 160 can enable users to interact with each other and receive content from third-party systems 170 or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0126] In certain embodiments, third-party systems 170 may include one or more types of servers, one or more data stores, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable components (e.g., the server may communicate with these components). Third-party systems 170 may be operated by a different entity than the entity that operates social-networking system 160. However, in certain embodiments, social-networking system 160 and third-party systems 170 may operate in conjunction with each other to provide social-networking services to users of social-networking system 160 or third-party systems 170. In this sense, social-networking system 160 may provide a platform or backbone that other systems (e.g., third-party systems 170) may use to provide social-networking services and functionality to users across the Internet.

[0127] In certain embodiments, third-party system 170 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be delivered to client system 130. By way of example and not limitation, content objects may include information about things or activities of interest to the user, such as, for example, movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not limitation, content objects may include incentive content objects (e.g., coupons, discount coupons, gift certificates, or other suitable incentive objects).

[0128] In certain embodiments, social networking system 160 also includes user-generated content objects that can enhance a user's interaction with social networking system 160. User-generated content can include any content that a user can add, upload, send, or "post" to social networking system 160. As an example and not by way of limitation, a user transmits a post from client system 130 to social networking system 160. Posts can include data, such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to social networking system 160 by third parties through "communication channels" such as dynamic messages or streams.

[0129] In certain embodiments, the social networking system 160 may include various servers, subsystems, programs, modules, logs, and data storage. In certain embodiments, the social networking system 160 may include one or more of the following: a web server, an action recorder, an API request server, a correlation and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertisement-targeting module, a user interface module, a user profile storage, a relationship storage, a third-party content storage, or a location storage. The social networking system 160 may also include suitable components, such as a network interface, a security mechanism, a load balancer, a failover server, a management and network operation console, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 160 may include one or more user profile storages for storing user profiles. A user profile may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, affinities, or locations). Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not by way of limitation, if a user "likes" an article about a brand of shoes, the category may be the brand, or the general category of "shoes" or "clothing". The association storage may be used to store association information about users. The association information may indicate users with similar or common work experience, group membership, hobbies, educational history, or who are related or share common attributes in any way. The association information may also include user-defined associations between different users and content (internal and external). The web server may be used to link the social networking system 160 to one or more client systems 130 or one or more third-party systems 170 via the network 110. The web server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 160 and one or more client systems 130. The API request server may allow a third-party system 170 to access information from the social networking system 160 by calling one or more APIs. The action recorder may be used to receive communications from the web server about the user's actions on or outside the social networking system 160. In conjunction with the action log, a third-party content object log of user exposure to third-party content objects can be maintained. The notification controller can provide information about content objects to the client system 130. The information can be pushed to the client system 130 as a notification, or the information can be pulled from the client system 130 in response to a request received from the client system 130.The authorization server may be used to enforce one or more privacy settings of users of the social networking system 160. The privacy settings of the user determine how certain information associated with the user may be shared. The authorization server may allow the user to opt-in or opt-out of having their actions recorded by the social networking system 160 or shared with other systems (e.g., third-party systems 170), such as by setting appropriate privacy settings. A third-party content object store may be used to store content objects received from a third party (e.g., third-party system 170). A location store may be used to store location information associated with the user received from the client system 130. The advertising pricing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0130] Assistant System

[0131] Figure 2 An example architecture of an assistant system 140 is shown. In a particular embodiment, the assistant system 140 can help users obtain information or services. The assistant system 140 can enable users to interact with it in a stateful and multi-turn conversation with multimodal user input (such as sound, text, image, video) to obtain help. The assistant system 140 can create and store a user profile including personal information and contextual information associated with the user. In a particular embodiment, the assistant system 140 can use natural language understanding to analyze user input. The analysis can be based on the user profile to obtain a more personalized and context-aware understanding. The assistant system 140 can parse entities associated with the user input based on the analysis. In a particular embodiment, the assistant system 140 can interact with different agents to obtain information or services associated with the parsed entity. The assistant system 140 can generate responses about information or services for the user by using natural language generation. Through interaction with the user, the assistant system 140 can use dialogue management technology to manage and forward the conversation flow with the user. In a particular embodiment, the assistant system 140 can also help users digest the information obtained effectively and efficiently by summarizing the information. The assistant system 140 may also help users better participate in online social networks by providing tools that help users interact with the online social networks (e.g., create posts, comments, messages). The assistant system 140 may additionally help users manage different tasks, such as keeping track of events. In certain embodiments, the assistant system 140 may proactively perform pre-authorized tasks related to the user's interests and preferences based on the user profile at a time relevant to the user without user input. In certain embodiments, the assistant system 140 may check privacy settings to ensure that access to the user's profile or other user information and performance of different tasks are permitted according to the user's privacy settings.

[0132] In a particular embodiment, the assistant system 140 can receive user input from an assistant application 136 in a client system 130 associated with the user. If the user input is based on a text modality, the assistant system 140 can receive it at the messaging platform 205. If the user input is based on an audio modality (e.g., the user can speak to the assistant application 136 or send a video including voice to the assistant application 136), the assistant system 140 can process it using an audio speech recognition (ASR) module 210 to convert the user input into text. If the user input is based on an image or video modality, the assistant system 140 can process it using optical character recognition technology within the messaging platform 205 to convert the user input into text. The output of the messaging platform 205 or the ASR module 210 can be received at the assistant xbot 215.

[0133] In a particular embodiment, assistant xbot 215 can be a type of chatbot (chat bot). Assistant xbot 215 can include a programmable service channel, which can be a software code, logic or routine used as a user personal assistant. Assistant xbot 215 can serve as a user portal of assistant system 140. Therefore, assistant xbot 215 can be considered as a type of conversational agent. In a particular embodiment, assistant xbot 215 can send text user input to natural language understanding (NLU) module 220 to interpret user input. In a particular embodiment, NLU module 220 can obtain information from user context engine 225 and semantic information aggregator (semantic information aggregator) 230 to accurately understand user input. User context engine 225 can store user profiles of users. User profiles of users can include user profile data, which includes demographic information, social information and context information associated with users. User profile data can also include users' interests and preferences for multiple topics gathered through conversations on dynamic messages, search logs, messaging platforms 205, etc. The use of the user profile can be protected after the privacy check module 245 to ensure that the user's information can only be used for his / her benefit and cannot be shared with anyone else. The semantic information aggregator 230 can provide ontology data associated with multiple predefined domains, intents, and slots to the NLU module 220. In a specific embodiment, the domain can represent the social context of the interaction, for example, education. The intent can indicate the purpose of the user's interaction with the assistant system 140. The slot can represent a basic semantic entity. For example, the slot of "pizza" can be a plate. The semantic information aggregator 230 can also extract information from social graphs, knowledge graphs, and concept graphs, and retrieve user profiles from the user context engine 225. The semantic information aggregator 230 can also process information from these different sources by determining what information to aggregate, annotating n-grams (n-grams) of user input, sorting n-grams with confidence scores based on the aggregated information, and formulating the sorted n-grams into features that can be used by the NLU module 220 to understand the user input. Based on the output of the user context engine 225 and the semantic information aggregator 230, the NLU module 220 can identify domains, intents, and one or more slots from the user input in a personalized and context-aware manner. As an example and not by way of limitation, the user input can include "show me how to get to the Starbucks". The NLU module 220 can identify a specific Starbucks that the user wants to go to based on the user's personal information and associated contextual information.In certain embodiments, the NLU module 220 may include a lexicon of language, a parser, and grammar rules to divide sentences into internal representations. The NLU module 220 may also include one or more programs that use pragmatics to perform naive semantics or random semantic analysis to understand user input. In certain embodiments, the parser may be based on a deep learning architecture including multiple long short-term memory (LSTM) networks. As an example and not as a limitation, the parser may be based on a recurrent neural network grammar (RNNG) model, which is a type of cyclic and recursive LSTM algorithm.

[0134] In a particular embodiment, the identified domain, intent, and one or more slots from the NLU module 220 may be sent to the conversation engine 235. In a particular embodiment, the conversation engine 235 may manage the conversation flow and conversation state between the user and the assistant xbot 215. The conversation engine 235 may also store previous conversations between the user and the assistant xbot 215. In a particular embodiment, the conversation engine 235 may communicate with the entity resolution module 240 to resolve entities associated with one or more slots, which supports the conversation engine 235 to forward the conversation flow between the user and the assistant xbot 215. In a particular embodiment, the entity resolution module 240 may access social graphs, knowledge graphs, and concept graphs when resolving entities. An entity may include, for example, a unique user or concept, each of which may have a unique identifier (ID). As an example and not as a limitation, a knowledge graph may include multiple entities. Each entity may include a single record associated with one or more attribute values. A specific record may be associated with a unique entity identifier. For an attribute of an entity, each record may have a different value. Each attribute value may be associated with a confidence probability. The confidence probability of an attribute value indicates the probability that the value is accurate for a given attribute. Each attribute value may also be associated with a semantic weight. The semantic weight of an attribute value may indicate the degree to which the value is semantically suitable for a given attribute, taking into account all available information. For example, a knowledge graph may include an entity of the movie "The Martian" (2015), which includes information that has been extracted from multiple content sources (e.g., Facebook, Wikipedia, CSM, IMDb, Metacritics, Rovi, TMDb, and Rotten Tomatoes), and then de-duplicated, parsed, and fused to generate a single unique record of the knowledge graph. An entity may be associated with a spatial attribute value indicating the genre of the movie "The Martian" (2015). The entity resolution module 240 may also request a user profile of a user associated with the user input from the user context engine 225. In a particular embodiment, the entity resolution module 240 may communicate with the privacy check module 245 to ensure that the resolution of the entity does not violate a privacy policy. In a particular embodiment, the privacy check module 245 may implement a privacy policy using an authorization / privacy server. As an example and not by way of limitation, the entity to be resolved may be another user who has specified in his / her privacy settings that his / her identity should not be searchable on the online social network, and thus entity resolution module 240 may respond to the request without returning an identifier for that user. Based on information obtained from the social graph, knowledge graph, concept graph, and user profile, and in compliance with applicable privacy policies, entity resolution module 240 may therefore accurately resolve entities associated with user input in a personalized and context-aware manner.In certain embodiments, each resolved entity may be associated with one or more identifiers hosted by social networking system 160. As an example and not by way of limitation, the identifier may include a unique user identifier (ID). In certain embodiments, each resolved entity may also be associated with a confidence score.

[0135] In a particular embodiment, the conversation engine 235 may communicate with different agents based on the identified intent and domain and the resolved entity. In a particular embodiment, the agent may include a first-party agent 250 and a third-party agent 255. In a particular embodiment, the first-party agent 250 may include an internal agent that can be accessed and controlled by the assistant system 140 (e.g., an agent associated with a service provided by an online social network (Messenger, Instagram)). In a particular embodiment, the third-party agent 255 may include an external agent that the assistant system 140 cannot control (e.g., a music streaming agent (Spotify), a ticket sales agent (Ticketmaster)). The first-party agent 250 may be associated with a first-party provider 260 that provides content objects and / or services hosted by the social networking system 160. The third-party agent 255 may be associated with a third-party provider 265 that provides content objects and / or services hosted by a third-party system 170.

[0136] In certain embodiments, communications from the conversation engine 235 to the first party agent 250 may include a request for a specific content object and / or service provided by the first party provider 260. Thus, the first party agent 250 may retrieve the requested content object from the first party provider 260 and / or execute a task that instructs the first party provider 260 to perform the requested service. In certain embodiments, communications from the conversation engine 235 to the third party agent 255 may include a request for a specific content object and / or service provided by the third party provider 265. Thus, the third party agent 255 may retrieve the requested content object from the third party provider 265 and / or execute a task that instructs the third party provider 265 to perform the requested service. The third party agent 255 may access the privacy check module 245 to ensure that there is no privacy violation before interacting with the third party provider 265. As an example and not by way of limitation, a user associated with the user input may specify in his / her privacy settings that his / her profile information is not visible to any third party content provider. Thus, when retrieving a content object associated with a user input from a third party provider 265 , the third party agent 255 may complete the retrieval without revealing to the third party provider 265 which user is requesting the content object.

[0137] In certain embodiments, each of the first party agent 250 or the third party agent 255 may be designated for a specific domain. By way of example and not limitation, the domain may include weather, transportation, music, etc. In certain embodiments, the assistant system 140 may use multiple agents in concert to respond to user input. By way of example and not limitation, the user input may include "direct me to my next meeting". The assistant system 140 may use a calendar agent to retrieve the location of the next meeting. The assistant system 140 may then use a navigation agent to direct the user to the next meeting.

[0138] In certain embodiments, each of the first party agent 250 or the third party agent 255 may retrieve a user profile from the user context engine 225 to perform the task in a personalized and context-aware manner. By way of example and not limitation, the user input may include "book me a ride to the airport". The transportation agent may perform the task of booking a ride. The transportation agent may retrieve the user profile of the user from the user context engine 225 before booking the ride. For example, the user profile may indicate that the user prefers taxis, so the transportation agent may book a taxi for the user. As another example, context information associated with the user profile may indicate that the user is in a hurry, so the transportation agent may book a ride for the user from a ride-sharing service (e.g., Uber, Lyft) because getting a ride from a ride-sharing service may be faster than a taxi company. In certain embodiments, each of the first party agent 250 or the third party agent 255 may consider other factors when performing the task. By way of example and not limitation, the other factors may include price, ratings, efficiency, partnerships with online social networks, etc.

[0139] In certain embodiments, the dialog engine 235 may communicate with a conversation understanding writer (CU writer) 270. The dialog engine 235 may send the requested content object and / or the status of the requested service to the CU writer 270. In certain embodiments, the dialog engine 235 may send the requested content object and / or the status of the requested service to the CU writer 270.<k,c,u,d> Tuples are sent, where k indicates a knowledge source, c indicates a communication target, u indicates a user model, and d indicates a discourse model. In a specific embodiment, the CU writer 270 may include a natural language generator (NLG) 271 and a user interface (UI) payload generator 272. The natural language generator 271 may generate communication content based on the output of the dialogue engine 235. In a specific embodiment, the NLG 271 may include a content determination component, a sentence planner, and a surface realization component. The content determination component may determine the communication content based on the knowledge source, the communication target, and the user's expectations. As an example and not as a limitation, the determination may be based on description logic. The description logic may include, for example, three basic notions, which are individuals (representing objects in a domain), concepts (describing individual sets), and roles (representing binary relationships between individuals or concepts). The description logic may be characterized by a set of constructors that allow the natural language generator 271 to build complex concepts / roles from atomic concepts / roles. In a specific embodiment, the content determination component may perform the following tasks to determine the communication content. The first task may include a translation task, in which the input to the natural language generator 271 can be translated into concepts. The second task may include a selection task, in which relevant concepts can be selected from the concepts generated by the translation task based on the user model. The third task may include a verification task, in which the consistency of the selected concept can be verified. The fourth task may include an instantiation task, in which the verified concept can be instantiated into an executable file that can be processed by the natural language generator 271. The sentence planner can determine the organization of the communication content so that it is understandable to people. The surface implementation component can determine the specific words to be used, the order of sentences, and the style of the communication content. The UI payload generator 272 can determine the preferred modality of the communication content to be presented to the user. In a specific embodiment, the CU writer 270 can communicate with the privacy check module 245 to ensure that the generation of the communication content follows the privacy policy. In a specific embodiment, when generating the communication content and determining the modality of the communication content, the CU writer 270 can retrieve the user profile from the user context engine 225. Therefore, for the user, the communication content can be more natural, personalized, and context-aware. As an example and not by way of limitation, a user profile may indicate that the user prefers short sentences in conversation, and thus the generated communication content may be based on short sentences.As another example and not by way of limitation, context information associated with a user profile may indicate that the user is using a device that only outputs audio signals, so UI payload generator 272 may determine the modality of the communication content as audio.

[0140] In certain embodiments, the CU writer 270 may send the generated communication content to the assistant xbot 215. In certain embodiments, the assistant xbot 215 may send the communication content to the messaging platform 205. The messaging platform 205 may also send the communication content to the client system 130 via the assistant application 136. In an alternative embodiment, the assistant xbot 215 may send the communication content to a text-to-speech (TTS) module 275. The TTS module 275 may convert the communication content into an audio clip. The TTS module 275 may also send the audio clip to the client system 130 via the assistant application 136.

[0141] In certain embodiments, assistant xbot 215 may interact with proactive inference layer 280 without receiving user input. Proactive inference layer 280 may infer user interests and preferences based on a user profile retrieved from user context engine 225. In certain embodiments, proactive inference layer 280 may also communicate with proactive agent 285 regarding inferences. Proactive agent 285 may perform proactive tasks based on inferences. As an example and not as a limitation, proactive tasks may include sending content objects to users or providing services. In certain embodiments, each proactive task may be associated with an agenda item. Agenda items may include recurring items, such as daily summaries. Agenda items may also include one-time items. In certain embodiments, proactive agent 285 may retrieve a user profile from user context engine 225 when performing proactive tasks. Thus, proactive agent 285 may perform proactive tasks in a personalized and context-aware manner. As an example and not as a limitation, the proactive inference layer may infer that a user likes the band Maroon 5, and proactive agent 285 may generate recommendations for new songs / albums of Maroon 5 for the user.

[0142] In a particular embodiment, the active agent 285 can generate candidate entities associated with the active task based on the user profile. The generation can be based on a direct backend query that retrieves the candidate entity from a structured data storage using a deterministic filter. Alternatively, the generation can be based on a machine learning model that is trained based on the user profile, entity attributes, and the correlation between the user and the entity. As an example and not as a limitation, the machine learning model can be based on a support vector machine (SVM). As another example and not as a limitation, the machine learning model can be based on a regression model. As another example and not as a limitation, the machine learning model can be based on a deep convolutional neural network (DCNN). In a particular embodiment, the active agent 285 can also sort the generated candidate entities based on the user profile and the content associated with the candidate entity. The sorting can be based on the similarity between the user's interest and the candidate entity. As an example and not as a limitation, the assistant system 140 can generate a feature vector representing the user's interest and a feature vector representing the candidate entity. The assistant system 140 can then calculate a similarity score (e.g., based on cosine similarity) between the feature vector representing the user's interest and the feature vector representing the candidate entity. Alternatively, the sorting can be based on a sorting model that is trained based on user feedback data.

[0143] In certain embodiments, the active task may include recommending a candidate entity to a user. The active agent 285 may schedule the recommendation, thereby associating the recommendation time with the recommended candidate entity. The recommended candidate entity may also be associated with a priority and an expiration time. In certain embodiments, the recommended candidate entity may be sent to an active scheduler. The active scheduler may determine the actual time to send the recommended candidate entity to the user based on the priority associated with the task and other relevant factors (e.g., clicks and impressions of the recommended candidate entity). In certain embodiments, the active scheduler may then send the recommended candidate entity with the determined actual time to an asynchronous tier. The asynchronous tier may temporarily store the recommended candidate entity as a job. In certain embodiments, the asynchronous tier may send the job to the conversation engine 235 at the determined actual time for execution. In alternative embodiments, the asynchronous tier may execute the job by sending it to other surface layers (e.g., other notification services associated with the social networking system 160). In certain embodiments, the conversation engine 235 may identify the conversation intent, state, and history associated with the user. Based on the conversation intent, the conversation engine 235 may select some candidate entities from the recommended candidate entities to send to the client system 130. In a particular embodiment, the conversation state and history may indicate whether the user is involved in an ongoing conversation with assistant xbot 215. If the user is involved in an ongoing conversation and the priority of the recommended task is low, the conversation engine 235 may communicate with the active scheduler to reschedule the time when the selected candidate entity is sent to the client system 130. If the user is involved in an ongoing conversation and the priority of the recommended task is high, the conversation engine 235 may initiate a new conversation session (dialogsession) with the user, in which the selected candidate entity may be presented. Therefore, the interruption of the ongoing conversation may be prevented. When it is determined that sending the selected candidate entity will not interrupt the user, the conversation engine 235 may send the selected candidate entity to the CU writer 270 to generate personalized and context-aware communication content including the selected candidate entity according to the user's privacy settings. In a particular embodiment, the CU writer 270 may send the communication content to the assistant xbot 215, which may then be sent to the client system 130 via the messaging platform 205 or the TTS module 275.

[0144] In certain embodiments, assistant xbot 215 may communicate with proactive agent 285 in response to user input. As an example and not by way of limitation, a user may ask assistant xbot 215 to set a reminder. Assistant xbot 215 may request proactive agent 285 to set such a reminder, and proactive agent 285 may later proactively perform the task of reminding the user.

[0145] In a particular embodiment, the assistant system 140 may include a summarizer 290. The summarizer 290 may provide a customized dynamic message summary to the user. In a particular embodiment, the summarizer 290 may include multiple metaagents. Multiple metaagents may use first-party agents 250, third-party agents 255, or active agents 285 to generate dynamic message summaries. In a particular embodiment, the summarizer 290 may retrieve user interests and preferences from the active inference layer 280. The summarizer 290 may then retrieve entities associated with user interests and preferences from the entity resolution module 240. The summarizer 290 may also retrieve a user profile from the user context engine 225. Based on information from the active inference layer 280, the entity resolution module 240, and the user context engine 225, the summarizer 290 may generate a personalized and context-aware summary for the user. In a particular embodiment, the summarizer 290 may send the summary to the CU writer 270. The CU writer 270 may process the summary and send the processing result to the assistant xbot 215. Assistant xbot 215 may then send the processed summary to client system 130 via messaging platform 205 or TTS module 275 .

[0146] Figure 3 An example flow chart of the assistant system 140 responding to a user request is shown. In a particular embodiment, the assistant xbot215 can access the request manager 305 when receiving a user request. The request manager 305 may include a context extractor 306 and a conversation understanding object generator (CU object generator) 307. The context extractor 306 can extract context information associated with the user request. The CU object generator 307 can generate a specific content object related to the user request. In a particular embodiment, the request manager 305 can store the context information and the generated content object in a data storage 310, which is a specific data storage implemented in the assistant system 140.

[0147] In a particular embodiment, the request manager 305 may send the generated content object to the NLU module 220. The NLU module 220 may perform a number of steps to process the content object. In step 221, the NLU module 220 may generate a whitelist of content objects. In step 222, the NLU module 220 may perform characterization based on the whitelist. In step 223, the NLU module 220 may perform domain classification / selection based on the features generated from the characterization. The domain classification / selection results may be further processed based on two related processes. In step 224a, the NLU module 220 may use an intent classifier to process the domain classification / selection results. The intent classifier may determine a user intent associated with the user request. As an example and not by way of limitation, the intent classifier may be based on a machine learning model that may take a domain classification / selection result as input and calculate a probability that the input is associated with a specific predefined intent. In step 224b, the NLU module may use a meta-intent classifier to process the domain classification / selection results. The meta-intent classifier may determine a category that describes the user intent. As an example and not as a limitation, the meta-intent classifier can be based on a machine learning model that can take domain classification / selection results as input and calculate the probability that the input is associated with a specific predefined meta-intent. In step 225a, the NLU module 220 can use a slot tagger to annotate one or more slots associated with the user request. In step 225b, the NLU module 220 can use a meta-slot tagger to annotate one or more slots for the classification results from the meta-intent classifier. As an example and not as a limitation, the user request may include "change 500 dollars in my account to Japanese yen". The intent classifier can take the user request as input and formulate it into a vector. The intent classifier can then calculate the probability that the user request is associated with different predefined intents based on a vector comparison between the vector representing the user request and the vector representing different predefined intents. In a similar manner, the slot tagger can take the user request as input and formulate each word into a vector. The intent classifier can then calculate the probability of each word being associated with different predefined slots based on a vector comparison between the vector representing the word and the vector representing the different predefined slots. The user's intent can be classified as "changing money". The slots requested by the user can include "500", "dollars", "account", and "Japanese yen". The user's meta-intent can be classified as "financial service". The meta-slot can include "finance".

[0148] In a particular embodiment, the NLU module 220 may improve the domain classification / selection of content objects by extracting semantic information from the semantic information aggregator 230. In a particular embodiment, the semantic information aggregator 230 may aggregate semantic information in the following manner. The semantic information aggregator 230 may first retrieve information from the user context engine 225. In a particular embodiment, the user context engine 225 may include an offline aggregator 226 and an online inference service 227. The offline aggregator 226 may process multiple data associated with the user collected from a previous time window. As an example and not by way of limitation, the data may include dynamic message posts / comments, interactions with dynamic message posts / comments, Instagram posts / comments, search history, etc. collected from a previous 90-day window. The processing results may be stored in the user context engine 225 as part of the user profile. The online inference service 227 may analyze the session data associated with the user received by the assistant system 140 at the current time. The analysis results may also be stored in the user context engine 225 as part of the user profile. In certain embodiments, the semantic information aggregator 230 may then process the information retrieved from the user context engine 225, i.e., the user profile, in the following steps. In step 231, the semantic information aggregator 230 may process the information retrieved from the user context engine 225 based on natural language processing (NLP). In step 232, the processing result may be annotated with entities by an entity tagger. In step 233, based on the annotations, the semantic information aggregator 230 may generate a dictionary for the retrieved information. In step 234, the semantic information aggregator 230 may rank the entities tagged by the entity tagger. In certain embodiments, the semantic information aggregator 230 may communicate with different graphs 330 including social graphs, knowledge graphs, and concept graphs to extract ontology data related to the information retrieved from the user context engine 225. In certain embodiments, the semantic information aggregator 230 may aggregate the user profile, the ranked entities, and the information from the graphs 330. The semantic information aggregator 230 may then send the aggregated information to the NLU module 220 to facilitate domain classification / selection.

[0149] In certain embodiments, the output of the NLU module 220 may be sent to a coreference module 315 to interpret the references of the content object associated with the user request. The coreference module 315 may include a reference creation 316 and a reference resolution 317. In certain embodiments, the reference creation 316 may create references for the entities determined by the NLU module 220. The reference resolution 317 may accurately resolve these references. As an example and not by way of limitation, the user request may include "find me the nearest Walmart and direct me there". The coreference module 315 may interpret "there" as "the nearest Walmart". In certain embodiments, the coreference module 315 may access the user context engine 225 and the dialog engine 235, if necessary, to interpret the references with improved accuracy.

[0150] In a particular embodiment, the identified domains, intents, meta-intents, slots and meta-slots, and resolved references may be sent to the entity resolution module 240 to resolve related entities. In a particular embodiment, the entity resolution module 240 may include a domain entity resolution 241 and a general entity resolution 242. The domain entity resolution 241 may resolve entities by categorizing slots and meta-slots into different domains. In a particular embodiment, entities may be resolved based on ontology data extracted from the graph 330. The ontology data may include structural relationships between different slots / meta-slots and domains. The ontology may also include information on how slots / meta-slots may be grouped, related, and subdivided according to similarities and differences within a hierarchy that includes domains at a higher level. General entity resolution 242 may resolve entities by categorizing slots and meta-slots into different general topics. In a particular embodiment, resolution may also be based on ontology data extracted from the graph 330. The ontology data may include structural relationships between different slots / meta-slots and general topics. The ontology may also include information on how slots / meta-slots may be grouped, related, and subdivided according to similarities and differences within a hierarchy that includes topics at a higher level. As an example and not by way of limitation, in response to input of a query about the advantages of Tesla cars, general entity resolution 242 may resolve Tesla cars to vehicles and domain entity resolution 241 may resolve Tesla cars to electric cars.

[0151] In a particular embodiment, the output of the entity resolution module 240 may be sent to a conversation engine 235 to forward the conversation flow with the user. The conversation engine 235 may include a conversation intent resolution 236 and a conversation state updater / sequencer 237. In a particular embodiment, the conversation intent resolution 236 may resolve the user intent associated with the current conversation session. In a particular embodiment, the conversation state updater / sequencer 237 may update / sequence the state of the current conversation session. As an example and not as a limitation, if the conversation session ends, the conversation state updater / sequencer 237 may update the conversation state to "completed". As another example and not as a limitation, the conversation state updater / sequencer 237 may sort the conversation state based on the priority associated with the conversation state.

[0152] In certain embodiments, the dialog engine 235 may communicate with the task completion module 335 regarding the dialog intent and the associated content objects. The task completion module 335 may include an action selection component 336. In certain embodiments, the dialog engine 235 may additionally check against a dialog strategy 320 regarding the dialog state. The dialog strategy 320 may include a general strategy 321 and a domain-specific strategy 322, both of which may guide how to select the next system action based on the dialog state. In certain embodiments, the task completion module 335 may communicate with the dialog strategy 320 to obtain guidance for the next system action. In certain embodiments, the action selection component 336 may therefore select an action based on the dialog intent, the associated content objects, and the guidance from the dialog strategy 320.

[0153] In a particular embodiment, the output of the task completion module 335 may be sent to the CU writer 270. In an alternative embodiment, the selected action may require one or more agents 340 to participate. Therefore, the task completion module 335 may notify the agent 340 of the selected action. At the same time, the conversation engine 235 may receive an instruction to update the conversation state. As an example and not as a limitation, the update may include waiting for the response of the agent. In a particular embodiment, the CU writer 270 may generate communication content for the user using NLG 271 based on the output of the task completion module 335. The CU writer 270 may also use the UI payload generator 272 to determine the modality of the generated communication content. Since the generated communication content can be considered as a response to a user request, the CU writer 270 may additionally use the response sorter 273 to sort the generated communication content. As an example and not as a limitation, the sorting may indicate the priority of the response.

[0154] In a particular embodiment, the output of the CU writer 270 may be sent to a response manager 325. The response manager 325 may perform different tasks, including storing / updating the dialog state 326 retrieved from the data storage 310 and generating a response 327. In a particular embodiment, the generated response and the communication content may be sent to the assistant xbot 215. In an alternative embodiment, if the determined modality of the communication content is audio, the output of the CU writer 270 may be additionally sent to the TTS module 275. The speech generated by the TTS module 275 and the response generated by the response manager 325 may then be sent to the assistant xbot 215.

[0155] Handling multimodal user input

[0156] In certain embodiments, the assistant system 140 may receive multimodal user input (e.g., sound, image, video, text) from a client system 130 associated with a user. The assistant system 140 may process the multimodal user input using different modules of the assistant system 140 based on the different modalities of the user input. The assistant system 140 may then understand the user input based on the entity resolution module 240 and the coreference module 315, i.e., identify the subject / entity and semantic meaning of the user input. The assistant system 140 may also generate a multimodal output based on the understanding and send the multimodal output to the user via the client system 130. As an example and not by way of limitation, a user may submit a picture of several people to the assistant system 140 and request the assistant system 140 to initiate a call to one of the people (e.g., the user may say "call the guy on the left"). The assistant system 140 may determine the identities of the people, identify the person "on the left", access the contact information of the person on the left (subject to the privacy settings of the person), and initiate a call for the user accordingly. In particular, the assistant system 140 can use a visual recognition agent capable of processing visual inputs contained in multimodal user inputs, wherein the visual recognition agent can access various computer vision models to understand the visual inputs. The co-referencing module 315 can use the understanding of the visual input to parse the entities associated with the user input. Based on the understanding of the visual input and the parsed entities, the assistant system 140 can use the assistant xbot 215 to have a conversation with the user to further enhance the user's experience of the assistant system 140 by performing different tasks corresponding to the visual input. The assistant system 140 can respond to user queries and reactively process multimodal user inputs. The assistant system 140 can also actively process multimodal user inputs by providing suggestions based on user inputs. The ability to process multimodal user inputs and seamlessly switch between different output modalities makes the assistant system 140 well adapted to various client systems 130 and software, thereby making the assistant system 140 more useful and attractive to users. Although this disclosure describes processing particular multimodal user input in a particular manner via a particular system, this disclosure contemplates processing any suitable multimodal user input in any suitable manner via any suitable system.

[0157] In a particular embodiment, the assistant system 140 may receive user input based on one or more modalities from a client system 130 associated with a first user. At least one of the modalities of the user input may be a visual modality. In a particular embodiment, the assistant system 140 may identify one or more subjects associated with the user input based on a visual modality based on one or more machine learning models. In a particular embodiment, the assistant system 140 may determine one or more attributes associated with one or more subjects, respectively, based on one or more machine learning models. Then, the assistant system 140 may parse one or more entities corresponding to one or more subjects based on the determined one or more attributes. In a particular embodiment, the assistant system 140 may perform one or more tasks associated with one or more parsed entities. The assistant system 140 may also send instructions to the client system 130 associated with the first user for presenting communication content in response to user input. The communication content may include information associated with one or more tasks performed.

[0158] Figure 4 Shown based on Figure 2140 in the example architecture 200 to process multimodal user input. In a particular embodiment, the messaging platform 205 can receive multimodal user input 405 from the client system 130. The multimodal user input 405 can include one or more of a string, an audio clip, an image, or a video clip. In a particular embodiment, the assistant xbot 215 can receive the multimodal user input 405 from the messaging platform 205. The assistant xbot215 can recognize that the multimodal user input 405 includes a visual input 410 and a text input 415. In a particular embodiment, the assistant xbot 215 can send the visual input 410 to one or more visual recognition agents 450 to perform visual analysis on the visual input 410 to identify one or more subjects. In a particular embodiment, the visual recognition agent 450 can be a first-party agent 250. In an alternative embodiment, the visual recognition agent 450 can be a third-party agent 255. As an example and not as a limitation, one or more subjects associated with the user input 405 can include one or more of a person, a place, a business, or an object. In a particular embodiment, the visual recognition agent 450 may identify one or more individuals based on facial recognition. In a particular embodiment, the visual recognition agent 450 may identify one or more objects based on object detection. In a particular embodiment, the visual recognition agent 450 may store the identified one or more subjects in the dialogue state in the dialogue engine 235. Therefore, when necessary in the future, the assistant system 140 can effectively retrieve the identified one or more subjects from the dialogue engine 235. In a particular embodiment, the visual recognition agent 450 may additionally determine one or more attributes associated with one or more subjects respectively. As an example and not as a limitation, one or more attributes may be a position within the image (e.g., on the left), a color associated with the subject, a shape of the subject, etc. In a particular embodiment, the visual recognition agent 450 may access one or more computer vision libraries for visual analysis. As an example and not as a limitation, one or more computer vision libraries may include a library for detecting objects in an image. More information about object detection can be found in U.S. Patent Application No. 15 / 277,938 filed on September 27, 2016, which is incorporated by reference. As another example and not by way of limitation, the one or more computer vision libraries may include a library for recognizing faces in images. More information about facial recognition can be found in U.S. Patent Application No. 12 / 922,984, filed on March 4, 2014, which is incorporated by reference. Using machine learning models based on facial recognition and object detection to recognize people, places, businesses, and objects and their attributes may be an effective solution to the technical challenge of accurately identifying subjects and their attributes from visual input.In a particular embodiment, the visual recognition agent 450 can generate a feature representation for the user input 405 based on the visual modality (i.e., the visual input 410). The generated feature representation can be processed by one or more machine learning models provided by one or more computer vision libraries. As an example and not as a limitation, the one or more machine learning models may include one or more of a support vector machine, a regression model, or a convolutional neural network. In a particular embodiment, the visual recognition agent 450 can communicate with the privacy check module 245 to ensure that the visual analysis complies with the user's privacy settings when accessing the computer vision library. In a particular embodiment, each library can have its own privacy check component, which ensures that object detection or facial recognition complies with the user's privacy settings. Although the present disclosure describes a particular visual analysis in a particular manner, the present disclosure contemplates any suitable visual analysis in any suitable manner.

[0159] In a particular embodiment, the assistant xbot 215 may send the text input 415 to the NLU module 220. The NLU module 220 may identify one or more intents and one or more slots 415 based on the text input 415 of the user input 405. In a particular embodiment, the identified one or more intents and slots 420 and the visual analysis results 425 from the visual recognition agent 450 may be sent to the co-reference module 315. The co-reference module 315 may resolve one or more entities corresponding to one or more subjects identified by the visual recognition agent 450 based on the identified one or more slots. The co-reference module 315 is able to link the wording information of the text input 415 or audio input with the visual analysis results 425 of the visual input 410, which makes it an effective solution to the technical challenge of accurately resolving entities from the multimodal user input 405. On the other hand, the joint analysis of the text input 415 and the visual input 410 via the co-reference module 315 may produce a technical advantage of improved understanding of the multimodal user input 405. In a particular embodiment, the resolved entities and intents 430 may be sent to another or more first-party agents 250 or third-party agents 255, which may perform one or more tasks associated with one or more resolved entities based on the identified intents. As an example and not by way of limitation, the user input 405 may include an image and the sentence "Call the person on the left." The NLU module 220 may identify the intent as [IN: Call (person)] and identify the person slot [SL: The person "on the left"]. The visual recognition agent 450 may perform a visual analysis of the image to identify the person on the left in the image. The co-reference module 315 may then resolve the identity of the person on the left. The assistant system 140 may also perform the task of calling the person accordingly. Although the present disclosure describes resolving particular entities in a particular manner, the present disclosure contemplates resolving any suitable entity in any suitable manner.

[0160] In a particular embodiment, the first party agent 250 or the third party agent 255 may send the status 435 of one or more tasks to the CU writer 270. Then, the CU writer 270 may generate communication content 440 of information associated with the one or more tasks performed. The communication content 440 may include one or more of a string, an audio clip, an image, or a video clip. In a particular embodiment, the assistant system 140 may determine one or more modalities of the communication content 440 in the following manner. First, the assistant system 140 may identify context information associated with the first user. Then, the assistant system 140 may identify context information associated with the client system 130. The assistant system 140 may further determine one or more modalities based on the context information associated with the first user and the context information associated with the client system 130. As an example and not as a limitation, the context information associated with the user may indicate that the user is at home, which means that any modality may be applicable to the communication content 440. On the other hand, the context information associated with the client system 130 may indicate that the client system 130 only supports audio output. Therefore, the determined modality may include only audio. As another example and not by way of limitation, context information associated with a user may indicate that the user is traveling on a high-speed train, and context information associated with client system 130 may indicate that client system 130 is not connected to Wi-Fi but is only connected to cellular data. In this case, although client system 130 supports all modalities, assistant system 140 may still determine text as a modality because it takes longer to download images and / or videos using cellular data than to download text. Determining an appropriate modality based on context information associated with a user and context information associated with a client system 130 associated with a user may be an effective solution to the technical challenge of responding to multimodal user input 405 with communication content 440 based on an appropriate modality, which allows assistant system 140 to communicate with the user based on the determined modality that may be more suitable for the user's current situation. On the other hand, the ability to process multimodal user input 405 and switch between different output modalities can produce technical advantages that effectively adapt to various client systems 130 and software. Although the present disclosure describes determining a particular modality in a particular manner, the present disclosure contemplates determining any suitable modality in any suitable manner.

[0161] In a particular embodiment, user input 405 may include user interaction with a media content object. The user interaction may vary based on which surface the assistant system 140 is integrated with. As an example and not as a limitation, the assistant system 140 may be integrated with a photo library stored in the client system 130 or associated with an online social network. Accordingly, the user interaction may be a long-press photo, which may trigger the assistant system 140 to perform an understanding of the pressed photo. As another example and not as a limitation, the assistant system 140 may be integrated with a messaging application. Accordingly, the user interaction may be sharing a photo in a message thread, which may trigger the assistant system 140 to perform an understanding of the shared photo. As another example and not as a limitation, the assistant system 140 may be integrated with a live streaming surface. During a live broadcast, a user may ask questions to the assistant system 140, which may be used as a user input 405 to trigger an understanding of the live data. For example, a user may be broadcasting his / her trip to the Louvre and asking the assistant system 140 which artist created the painting currently being broadcasted. As another example and not by way of limitation, assistant system 140 may be integrated with a virtual reality (VR) surface. For example, a user may be using a VR surface to explore New York City. The user may see a wallet and ask assistant system 140 about the brand and price of the wallet, which may serve as user input 405. As another example and not by way of limitation, assistant system 140 may be integrated with a social networking application. A user may be browsing his / her friend's posts and ask assistant system 140 about a specific food post, which may serve as user input 405 to trigger assistant system 140 to obtain information about which restaurant provides the food for the user. Although the present disclosure describes a particular user interaction with a particular media content object in a particular manner, the present disclosure contemplates any suitable user interaction with any suitable media content object in any suitable manner.

[0162] In a particular embodiment, the assistant system 140 may proactively generate a plurality of tasks based on the visual modality of the user input 405. The assistant system 140 may then receive a user selection of one or more tasks from the plurality of tasks from the first user from the client system 130 associated with the first user. As an example and not by way of limitation, the user may be browsing pictures of new Nike shoes. The assistant system 140 may proactively generate a plurality of tasks that the user may select to check prices, purchase this shoe, or view similar shoes. As another example and not by way of limitation, the user may be live streaming at home, and the assistant system 140 may detect smoke in the background. The assistant system 140 may then proactively warn the user of smoke to ensure safety. The assistant system 140 may also generate some tasks including calling an emergency number, turning off the electricity at home, and the like. As another example and not by way of limitation, the client system 130 may have a camera that captures real-time visual data of the user's home. The user's child may walk in, and based on detecting the child, the assistant system 140 may proactively generate some tasks for the child. Based on the child's previous gaming history, these tasks may include playing games. As another example and not by way of limitation, a user may have shared a screenshot of a flight reservation with his / her friends (who will be traveling with him / her) in a group messaging surface. Assistant system 140 may recognize the flight information and proactively generate tasks that include setting reminders for the flight, booking a hotel room at the destination, checking the weather forecast at the destination, renting a car at the destination, etc. As another example and not by way of limitation, a user may have shared a trailer of a new movie with his / her friends, and assistant system 140 may proactively generate tasks for the user to select. The generated tasks may include buying tickets for the movie, posting the trailer to the user's dynamic message, and obtaining more information about the movie (e.g., actors and reviews). Proactively suggesting tasks to the user based on analysis of visual input 410 may produce a technical advantage of increasing the degree of user participation in assistant system 140. Although the present disclosure describes proactively generating specific tasks based on specific user input in a specific manner, the present disclosure contemplates generating any suitable tasks based on any suitable user input in any suitable manner.

[0163] Figure 5 An example interaction between a user and assistant system 140 via multimodal user input 405 and system output is shown. Figure 5As shown, the user can submit visual input 410, which is a photo of three people. The assistant system 140 can perform visual analysis on the photo to generate a visual analysis result 425 that identifies the people in the photo, namely, "People in the photo: David Jones, Mark Smith, and Bill Anders". The user can also submit text input 415a, namely, "Where does Mark live". The assistant system 140 can perform the task of determining Mark's residence based on the text input 415a and the visual analysis result 425 (observing Mark's privacy settings). The assistant system 140 can also generate communication content 440a accordingly (i.e., "Your friend currently lives in Hometown USA") and present the communication content 440a to the user. The user can submit another text input 415b (i.e., by entering "What's new about Mark") to query Mark's recent activities. Subsequently, the assistant system 140 can perform the task of summarizing Mark's recent activities based on the text input 415b and the visual analysis result 425. Assistant system 140 may further generate communication content 440b accordingly and present it to the user (i.e., by presenting a horizontal scrolling list of aggregated posts on online social networks that reference Mark Smith). The user may continue with another text input 415c (i.e., by entering "Call him") to attempt to call the person in the photo. In this case, assistant system 140 may generate communication content 440c to interact with the user (i.e., by responding to "Who do you want to call?") ​​to disambiguate which person the user wants to call. Subsequently, the user may clarify with text input 415d (i.e., by entering "David"). Assistant system 140 may perform the task of calling the clarified person and generate communication content 440d (i.e., by responding to "Call") to notify the user of the status 435 of the task. The user may submit another user input 415e (i.e., by entering "Remind me to call Mark on his birthday") to attempt to set a reminder. In a similar manner, assistant system 140 may generate communication content 440e to interact with the user (ie, by responding to “What time do you want to set the reminder for?”) to obtain more details about the reminder from the user.The user may then provide such details with text input 415f (i.e., "Midnight"). At this point, assistant system 140 may perform the task of setting the reminder and generate communication content 440f (i.e., notifying the user of the status of the task 435 by responding "I have set a reminder for midnight, Sept 13 to call Mark on his birthday"). Although this disclosure describes particular interactions between users and particular systems in particular ways, this disclosure contemplates any suitable interactions between users and any suitable systems in any suitable ways.

[0164] Figure 6 An example method 600 for processing multimodal user input 405 is shown. The method may begin at step 610, where assistant system 140 may receive user input 405 based on one or more modalities from client system 130 associated with a first user, wherein at least one of the modalities of user input 405 is a visual modality. At step 620, assistant system 140 may identify one or more subjects associated with user input 405 based on the visual modality based on one or more machine learning models. At step 630, assistant system 140 may determine one or more attributes associated with the one or more subjects, respectively, based on the one or more machine learning models. At step 640, assistant system 140 may resolve one or more entities corresponding to the one or more subjects based on the determined one or more attributes. At step 650, assistant system 140 may perform one or more tasks associated with the one or more resolved entities. At step 660, assistant system 140 may send instructions to client system 130 associated with the first user to present communication content 440 in response to user input 405, wherein communication content 440 includes information associated with one or more tasks performed. Where appropriate, particular embodiments may be repeated. Figure 6 Although the present disclosure will Figure 6 The specific steps of the method are described and shown as occurring in a particular order, but the present disclosure contemplates Figure 6 Any suitable steps of the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates the method including Figure 6 The present disclosure contemplates any suitable method for processing multimodal user input including any suitable steps, which may include, where appropriate, Figure 6In addition, although the present disclosure describes and illustrates the implementation of Figure 6 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Figure 6 Any suitable combination of any suitable components, devices, or systems of any suitable steps of the method.

[0165] Social Graph

[0166] Figure 7 An example social graph 700 is shown. In certain embodiments, social networking system 160 may store one or more social graphs 700 in one or more data stores. In certain embodiments, social graph 700 may include a plurality of nodes, which may include a plurality of user nodes 702 or a plurality of concept nodes 704, and a plurality of edges 706 connecting the nodes. Each node may be associated with a unique entity (i.e., a user or a concept), and each entity may have a unique identifier (ID), such as a unique number or username. For teaching purposes, the two-dimensional visual map representation is shown. Figure 7 Example social graph 700 is shown. In certain embodiments, social networking system 160, client system 130, assistant system 140, or third-party system 170 may access social graph 700 and related social graph information for use in appropriate applications. The nodes and edges of social graph 700 may be stored as data objects, for example, in a data store (e.g., a social graph database). Such a data store may include one or more searchable or queryable indexes of the nodes or edges of social graph 700.

[0167] In certain embodiments, user nodes 702 may correspond to users of social networking system 160 or assistant system 140. As an example and not by way of limitation, a user may be an individual (human user), an entity (e.g., a business, company, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with or through social networking system 160 or assistant system 140. In certain embodiments, when a user registers an account with social networking system 160, social networking system 160 may create a user node 702 corresponding to the user and store the user node 702 in one or more data stores. Users and user nodes 702 described herein may refer to registered users and user nodes 702 associated with registered users, where appropriate. Additionally or alternatively, users and user nodes 702 described herein may refer to users who are not registered with social networking system 160, where appropriate. In certain embodiments, user nodes 702 may be associated with information provided by a user or information collected by various systems, including social networking system 160. As an example and not by way of limitation, a user may provide his or her name, profile picture, contact information, date of birth, gender, marital status, family status, occupation, educational background, preferences, interests, or other demographic information. In certain embodiments, user node 702 may be associated with one or more data objects corresponding to information associated with the user. In certain embodiments, user node 702 may correspond to one or more web interfaces.

[0168] In certain embodiments, concept nodes 704 may correspond to concepts. By way of example and not limitation, a concept may correspond to a place (such as, for example, a movie theater, a restaurant, a landmark, or a city); a website (such as, for example, a website associated with social networking system 160 or a third-party website associated with a web application server); an entity (such as, for example, a person, a business, a group, a sports team, or a celebrity); a resource (such as, for example, an audio file, a video file, a digital photo, a text file, a structured document, or an application), which may be located within social networking system 160 or on an external server (e.g., a web application server); real estate or intellectual property (such as, for example, a sculpture, a painting, a movie, a game, a song, an idea, a photo, or a written work); a game; an activity; an idea or theory; another suitable concept; or two or more such concepts. Concept nodes 704 may be associated with information about the concept provided by a user or information collected by various systems, including social networking system 160 and assistant system 140. By way of example and not limitation, information for a concept may include a name or title; one or more images (e.g., an image of a book cover); a location (e.g., an address or geographic location); a website (which may be associated with a URL); contact information (e.g., a phone number or email address); other suitable concept information; or any suitable combination of such information. In particular embodiments, a concept node 704 may be associated with one or more data objects that correspond to the information associated with the concept node 704. In particular embodiments, a concept node 704 may correspond to one or more web interfaces.

[0169] In certain embodiments, a node in social graph 700 may represent a web interface (which may be referred to as a "profile interface") or be represented by a web interface. The profile interface may be hosted by social networking system 160 or assistant system 140 or accessible to social networking system 160 or assistant system 140. The profile interface may also be hosted on a third-party website associated with third-party system 170. As an example and not as a limitation, a profile interface corresponding to a particular external web interface may be a particular external web interface, and the profile interface may correspond to a particular concept node 704. The profile interface may be viewable by all other users or a selected subset of other users. As an example and not as a limitation, user node 702 may have a corresponding user profile interface, where the corresponding user may add content, make statements, or otherwise express him or herself. As another example and not as a limitation, concept node 704 may have a corresponding concept profile interface, where one or more users may add content, make statements, or express themselves, particularly with respect to the concept corresponding to concept node 704.

[0170] In certain embodiments, concept node 704 may represent a third-party web interface or resource hosted by third-party system 170. The third-party web interface or resource may include content representing an action or activity, selectable icons or other icons or other interactive objects (which may be implemented, for example, with JavaScript, AJAX, or PHP code), and other elements. As an example and not by way of limitation, the third-party web interface may include selectable icons such as “Like,” “Check-in,” “Eat,” “Recommend,” or other suitable actions or activities. A user viewing the third-party web interface may perform an action by selecting one of the icons (e.g., “Check-in”), causing client system 130 to send a message to social-networking system 160 indicating the user's action. In response to the message, social-networking system 160 may create an edge (e.g., a check-in-type edge) between user node 702 corresponding to the user and concept node 704 corresponding to the third-party web interface or resource, and store edge 706 in one or more data stores.

[0171] In particular embodiments, a pair of nodes in the social graph 700 may be connected to each other via one or more edges 706. An edge 706 connecting a pair of nodes may represent a relationship between the pair of nodes. In particular embodiments, an edge 706 may include or represent one or more data objects or attributes corresponding to the relationship between a pair of nodes. As an example and not by way of limitation, a first user may indicate that a second user is a "friend" of the first user. In response to the indication, the social networking system 160 may send a "friend request" to the second user. If the second user confirms the "friend request", the social networking system 160 may create an edge 706 in the social graph 700 connecting the first user's user node 702 to the second user's user node 702, and store the edge 706 as social graph information in one or more data stores 167. Figure 7In the example of , the social graph 700 includes an edge 706 indicating a friend relationship between the user nodes 702 of user "A" and user "B", and an edge indicating a friend relationship between the user nodes 702 of user "C" and user "B". Although the present disclosure describes or illustrates a particular edge 706 with a particular attribute connecting a particular user node 702, the present disclosure contemplates any suitable edge 706 with any suitable attribute connecting the user nodes 702. As an example and not by way of limitation, the edge 706 may represent a friendship, a family relationship, a business or employment relationship, a fan relationship (including, for example, likes, etc.), a follower relationship, a visitor relationship (including, for example, visits, views, check-ins, shares, etc.), a subscriber relationship, a superior / subordinate relationship, a reciprocal relationship, a non-reciprocal relationship, another suitable type of relationship, or two or more such relationships. In addition, although the present disclosure generally describes nodes as being connected, the present disclosure also describes users or concepts as being connected. Herein, references to connected users or concepts may refer to nodes corresponding to those users or concepts connected by one or more edges 706 in social graph 700 , where appropriate.

[0172] In certain embodiments, an edge 706 between a user node 702 and a concept node 704 may represent a particular action or activity performed by a user associated with the user node 702 toward a concept associated with the concept node 704. By way of example and not limitation, Figure 7 As shown, a user can "Like," "Attend," "Play," "Listen to," "Cook," "Work at," or "Watch" a concept, each of which can correspond to an edge type or subtype. The concept profile interface corresponding to concept node 704 may include, for example, a selectable "check-in" icon (such as, for example, a clickable "check-in" icon) or a selectable "add to favorites" icon. Similarly, after the user clicks on these icons, social networking system 160 may create a "Favorites" edge or a "Check-in" edge in response to a user action corresponding to the corresponding action. As another example and not by way of limitation, a user (user "C") may use a particular application (Spotify, which is an online music application) to listen to a particular song ("Imagine"). In this case, social networking system 160 may create a "Listen" edge 706 and a "Use" edge (such as, for example, a clickable "Check-in" icon) between user node 702 corresponding to the user and concept nodes 704 corresponding to the song and application. Figure 7 ), to indicate that the user listened to the song and used the application. In addition, social networking system 160 can create a "play" edge 706 (as shown in FIG. 7 ) between concept nodes 704 corresponding to the song and the application. Figure 7as shown), to indicate that a particular song is played by a particular application. In this case, the "played" edge 706 corresponds to an action performed by an external application (Spotify) on an external audio file (the song "Imagine"). Although this disclosure describes a particular edge 706 with particular attributes connecting the user node 702 and the concept node 704, this disclosure contemplates any suitable edge 706 with any suitable attributes connecting the user node 702 and the concept node 704. Additionally, although this disclosure describes an edge between the user node 702 and the concept node 704 representing a single relationship, this disclosure contemplates an edge between the user node 702 and the concept node 704 representing one or more relationships. By way of example and not limitation, the edge 706 can represent that the user likes and uses a particular concept. Alternatively, another edge 706 can represent each type of relationship (or multiple single relationships) between the user node 702 and the concept node 704 (as Figure 7 shown between the user node 702 of user "E" and the concept node 704 of "Spotify").

[0173] In a particular embodiment, the social networking system 160 can create an edge 706 between a user node 702 and a concept node 704 in the social graph 700. By way of example and not limitation, a user who views a concept profile interface (such as, for example, by using a web browser or a dedicated application hosted by the user's client system 130) can indicate that he or she likes the concept represented by the concept node 704 by clicking or selecting a "like" icon, which can cause the user's client system 130 to send a message to the social networking system 160 indicating that the user likes the concept associated with the concept profile interface. In response to the message, the social networking system 160 can create an edge 706 between the user node 702 associated with the user and the concept node 704, as shown by the "like" edge 706 between the user node and the concept node 704. In a particular embodiment, the social networking system 160 can store the edge 706 in one or more data stores. In a particular embodiment, the edge 706 can be automatically formed by the social networking system 160 in response to a particular user action. By way of example and not limitation, if a first user uploads a picture, watches a movie, or listens to a song, an edge 706 can be formed between the user node 702 corresponding to the first user and the concept nodes 704 corresponding to those concepts. Although this disclosure describes forming a particular edge 706 in a particular manner, this disclosure contemplates forming any suitable edge 706 in any suitable manner.

[0174] Vector Spaces and Embeddings

[0175] Figure 8An example view of vector space 800 is shown. In certain embodiments, objects or n-grams may be represented in a d-dimensional vector space, where d represents any suitable number of dimensions. Although vector space 800 is shown as a three-dimensional space, this is for illustrative purposes only, as vector space 800 may have any suitable dimensions. In certain embodiments, n-grams may be represented in vector space 800 as vectors, which are referred to as term embeddings. Each vector may include coordinates corresponding to a particular point in vector space 800 (i.e., the end point of the vector). By way of example and not limitation, Figure 8 As shown, vectors 810, 820, and 880 can be represented as points in vector space 800. n-grams can be mapped to corresponding vector representations. By way of example and not limitation, by applying a function defined by a dictionary n-gramt1 and n-gramt2 can be mapped to vectors in vector space 800 respectively. and Make and As another example and not by way of limitation, a dictionary trained to map text to a vector representation may be utilized, or such a dictionary itself may be generated by training. As another example and not by way of limitation, a model (e.g., Word2vec) may be used to map n-grams to vector representations in vector space 800. In certain embodiments, n-grams may be mapped to vector representations in vector space 800 using a machine learning model (e.g., a neural network). The machine learning model may have been trained using a sequence of training data (e.g., a corpus of multiple objects each including n-grams).

[0176] In certain embodiments, an object may be represented in vector space 800 as a vector, which is referred to as a feature vector or object embedding. By way of example and not limitation, by applying the function Objects e1 and e2 can be mapped to vectors in vector space 800 respectively. and Make and In certain embodiments, objects may be mapped to vectors based on one or more properties, attributes, or characteristics of the object, the object's relationship to other objects, or any other suitable information associated with the object. By way of example and not limitation, the function Objects can be mapped to vectors by feature extraction, which can start from an initial measurement data set and construct derived values ​​(e.g., features). As an example and not by way of limitation, objects including videos or images can be mapped to vectors by using algorithms to detect or isolate various desired parts or shapes of objects. The features used to calculate the vectors can be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transforms, edge direction, varying intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transforms (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information. As another example and not by way of limitation, objects including audio data can be mapped to vectors based on features (e.g., spectral slope, pitch coefficient, audio spectrum centroid, audio spectrum envelope, Mel-frequency cepstrum, or any other suitable information). In a particular embodiment, when an object has data that is too large to be effectively processed or includes redundant data, the function The transformed reduced feature set (e.g., feature selection) can be used to map objects to vectors. In certain embodiments, the function An object e can be mapped to a vector based on one or more n-grams associated with the object e. Although this disclosure describes representing n-grams or objects in a vector space in a particular manner, this disclosure contemplates representing n-grams or objects in a vector space in any suitable manner.

[0177] In certain embodiments, social networking system 160 may calculate a similarity metric for the vectors in vector space 800. The similarity metric may be cosine similarity, Minkowski distance, Mahalanobis distance, Jaccard similarity coefficient, or any suitable similarity metric. By way of example and not limitation, and The similarity measure can be cosine similarity As another example and not by way of limitation, and The similarity measure can be the Euclidean distance The similarity measure of two vectors may represent how similar two objects or n-grams corresponding to the two vectors, respectively, are to each other, as measured by the distance between the two vectors in vector space 800. As an example and not by way of limitation, vector 810 and vector 820 may correspond to objects that are more similar to each other than objects corresponding to vector 810 and vector 880, based on the distance between the respective vectors. Although this disclosure describes calculating a similarity measure between vectors in a particular manner, this disclosure contemplates calculating a similarity measure between vectors in any suitable manner.

[0178] More information about vector spaces, embeddings, feature vectors, and similarity measures can be found in U.S. patent application Ser. No. 14 / 949,436, filed on Nov. 23, 2015, U.S. patent application Ser. No. 15 / 286,315, filed on Oct. 5, 2016, and U.S. patent application Ser. No. 15 / 365,789, filed on Nov. 30, 2016, each of which is incorporated by reference.

[0179] Artificial Neural Networks

[0180] Fig. 9 An example artificial neural network ("ANN") 900 is shown. In particular embodiments, an ANN may refer to a computational model that includes one or more nodes. The example ANN 900 may include an input layer 910, hidden layers 920, 930, 960, and an output layer 950. Each layer of the ANN 900 may include one or more nodes, such as node 905 or node 915. In particular embodiments, each node of the ANN may be connected to another node of the ANN. As an example and not by way of limitation, each node of the input layer 910 may be connected to one or more nodes of the hidden layer 920. In particular embodiments, one or more nodes may be bias nodes (e.g., a node in a layer that is not connected to any node in the previous layer and does not receive input from it). In particular embodiments, each node in each layer may be connected to one or more nodes of the previous or next layer. Although Fig. 9 A particular ANN having a particular number of layers, a particular number of nodes, and particular connections between nodes is depicted, but the present disclosure contemplates any suitable ANN having any suitable number of layers, any suitable number of nodes, and any suitable connections between nodes. By way of example and not limitation, although Fig. 9 The connection between each node of the input layer 910 and each node of the hidden layer 920 is depicted, but one or more nodes of the input layer 910 may not be connected to one or more nodes of the hidden layer 920.

[0181] In certain embodiments, the ANN may be a feedforward ANN (e.g., an ANN without cycles or loops, where communication between nodes flows in one direction starting from the input layer and proceeding to successive layers). As an example and not by way of limitation, the input to each node of the hidden layer 920 may include the output of one or more nodes of the input layer 910. As another example and not by way of limitation, the input to each node of the output layer 950 may include the output of one or more nodes of the hidden layer 960. In certain embodiments, the ANN may be a deep neural network (e.g., a neural network including at least two hidden layers). In certain embodiments, the ANN may be a deep residual network. The deep residual network may be a feedforward ANN including hidden layers organized into residual blocks. The input to each residual block after the first residual block may be a function of the output of the previous residual block and the input of the previous residual block. As an example and not by way of limitation, the input to residual block N may be F(x)+x, where F(x) may be the output of residual block N-1 and x may be the input to residual block N-1. Although the present disclosure describes a particular ANN, the present disclosure contemplates any suitable ANN.

[0182] In certain embodiments, an activation function may correspond to each node of the ANN. The activation function of a node may define the output of the node for a given input. In certain embodiments, the input of a node may include a set of inputs. As an example and not by way of limitation, the activation function may be an identity function, a binary step function, a logistic function, or any other suitable function. As another example and not by way of limitation, the activation function of node k may be a sigmoid function Hyperbolic tangent function Rectifier F k (s k )=max(0,s k ) or any other suitable function F k (s k ), where s k can be an effective input to node k. In certain embodiments, the inputs to the activation functions corresponding to the nodes can be weighted. Each node can generate an output using a corresponding activation function based on the weighted inputs. In certain embodiments, each connection between nodes can be associated with a weight. As an example and not by way of limitation, a connection 925 between node 905 and node 915 can have a weighting factor of 0.4, which can indicate that the output of node 905 multiplied by 0.4 is used as an input to node 915. As another example and not by way of limitation, the output y of node k is k It can be k =F k (s k ), where F kcan be the activation function corresponding to node k, s k =∑ j (w jk x j ) can be a valid input to node k, x j can be the output of node j connected to node k, and w jk It can be a weighted coefficient between nodes, and nodes k. In a specific embodiment, the input of the nodes of the input layer can be based on a vector representing an object. Although the present disclosure describes specific inputs and outputs of nodes, the present disclosure contemplates any suitable inputs and outputs of nodes. In addition, although the present disclosure can describe specific connections and weights between nodes, the present disclosure contemplates any suitable connections and weights between nodes.

[0183] In a particular embodiment, the ANN may be trained using training data. As an example and not as a limitation, the training data may include the input and expected output of the ANN 900. As another example and not as a limitation, the training data may include vectors, each vector representing a training object and an expected label for each training object. In a particular embodiment, the training ANN may include modifying the weights associated with the connections between the nodes of the ANN by optimizing the objective function. As an example and not as a limitation, a training method (e.g., a conjugate gradient method, a gradient descent method, a stochastic gradient descent) may be used to back propagate the sum of squared errors (e.g., using a cost function that minimizes the sum of squared errors) as distance measurements between each vector representing the training object. In a particular embodiment, the ANN may be trained using a discarding technique. As an example and not as a limitation, one or more nodes may be temporarily ignored during training (e.g., not receiving input and not generating output). For each training object, one or more nodes of the ANN may have a certain probability of being ignored. The nodes ignored for a particular training object may be different from the nodes ignored for other training objects (e.g., nodes may be temporarily ignored on an object-by-object basis). Although this disclosure describes training the ANN in a particular manner, this disclosure contemplates training the ANN in any suitable manner.

[0184] privacy

[0185] In a particular embodiment, one or more objects (e.g., content or other types of objects) of a computing system may be associated with one or more privacy settings. One or more objects may be stored on or otherwise associated with any suitable computing system or application, such as, for example, a social networking system 160, a client system 130, an assistant system 140, a third-party system 170, a social networking application, an assistant application, a messaging application, a photo sharing application, or any other suitable computing system or application. Although the examples discussed herein are in the context of an online social network, these privacy settings may be applied to any other suitable computing system. The privacy settings (or "access settings") of an object may be stored in any suitable manner, such as, for example, associated with an object, indexed on an authorization server, in another suitable manner, or any suitable combination thereof. Privacy settings about an object may specify how the object (or specific information associated with the object) may be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, executed, displayed, or identified) in an online social network. An object may be described as being "visible" to a particular user or other entity when the privacy settings of the object allow that user or other entity to access the object. As an example and not by way of limitation, a user of an online social network may specify privacy settings on a user profile page that identify a set of users that may access work experience information on the user profile page, thereby excluding other users from accessing that information.

[0186] In certain embodiments, the privacy settings of an object may specify a "blocked list" of users or other entities that should not be allowed to access certain information associated with the object. In certain embodiments, the blacklist may include third-party entities. The blacklist may specify one or more users or entities to which the object is invisible. As an example and not by way of limitation, a user may specify a set of users that may not access an album associated with the user, thereby excluding those users from accessing the album (while also possibly allowing certain users not in the specified set of users to access the album). In certain embodiments, privacy settings may be associated with certain social graph elements. The privacy settings of a social graph element (e.g., a node or edge) may specify how the social graph element, information associated with the social graph element, or an object associated with the social graph element may be accessed using an online social network. As an example and not by way of limitation, a specific concept node 704 corresponding to a specific photo may have a privacy setting that specifies that the photo can only be accessed by the user tagged in the photo and the friends of the user tagged in the photo. In certain embodiments, privacy settings may allow users to opt-in or opt-out of having their content, information, or actions stored / recorded by the social networking system 160 or assistant system 140 or shared with other systems (e.g., third-party systems 170). Although this disclosure describes using particular privacy settings in particular manners, this disclosure contemplates using any suitable privacy settings in any suitable manner.

[0187] In certain embodiments, privacy settings may be based on one or more nodes or edges of the social graph 700. Privacy settings may be specified for one or more edges 706 or edge types of the social graph 700, or for one or more nodes 702, 704 or node types of the social graph 700. A privacy setting applied to a particular edge 706 connecting two nodes may control whether the relationship between two entities corresponding to the two nodes is visible to other users of the online social network. Similarly, a privacy setting applied to a particular node may control whether a user or concept corresponding to the node is visible to other users of the online social network. As an example and not by way of limitation, a first user may share an object to the social networking system 160. The object may be associated with a concept node 704 connected to the user node 702 of the first user by an edge 706. The first user may specify a privacy setting that applies to a particular edge 706 of the concept node 704 connected to the object, or may specify a privacy setting that applies to all edges 706 connected to the concept node 704. As another example and not by way of limitation, the first user may share a collection of objects (e.g., a collection of images) of a particular object type. The first user may specify the privacy settings for all objects of that particular object type associated with the first user to have a particular privacy setting (e.g., specifying that all images posted by the first user are visible only to the first user's friends and / or users tagged in the images).

[0188] In certain embodiments, social-networking system 160 may present a "privacy wizard" to the first user (e.g., within a web page, module, one or more dialog boxes, or any other suitable interface) to assist the first user in specifying one or more privacy settings. The privacy wizard may display instructions, appropriate privacy-related information, current privacy settings, one or more input fields for accepting one or more inputs from the first user (which specifies a change or confirmation of a privacy setting), or any suitable combination thereof. In certain embodiments, social-networking system 160 may provide the first user with a "dashboard" function that may display the first user's current privacy settings to the first user. The dashboard function may be displayed to the first user at any appropriate time (e.g., after input from the first user invoking the dashboard function, after a particular event or triggering action occurs). The dashboard function may allow the first user to modify one or more current privacy settings of the first user at any time in any suitable manner (e.g., redirecting the first user to the privacy wizard).

[0189] The privacy settings associated with an object may specify any suitable granularity at which access is allowed or denied. As an example and not by way of limitation, access or denial of access may be specified for specific users (e.g., only me, my roommate, my boss), users within a particular degree of separation (e.g., friends, friends of friends), user groups (e.g., a gaming club, my family), user networks (e.g., employees of a particular employer, students or alumni of a particular university), all users ("public"), no users ("private"), users of third-party systems 170, specific applications (e.g., third-party applications, external websites), other suitable entities, or any suitable combination thereof. Although the present disclosure describes a specific granularity at which access is allowed or denied, the present disclosure contemplates any suitable granularity at which access is allowed or denied.

[0190] In certain embodiments, one or more servers 162 may be authorization / privacy servers for implementing privacy settings. In response to a request from a user (or other entity) for a particular object stored in data storage 164, social networking system 160 may send a request for the object to data storage 164. The request may identify a user associated with the request, and the object may be sent to the user (or the user's client system 130) only if the authorization server determines that the user is authorized to access the object based on the privacy settings associated with the object. If the requesting user is not authorized to access the object, the authorization server may prevent the requested object from being retrieved from data storage 164, or may prevent the requested object from being sent to the user. In a search-query context, an object may be provided as a search result only if the querying user is authorized to access the object, for example, if the privacy settings of the object allow it to be revealed to the querying user, discovered by the querying user, or otherwise visible to the querying user. In certain embodiments, the object may represent content visible to the user through the user's dynamic message. As an example and not by way of limitation, one or more objects may be visible to the user's "Trending" page. In a particular embodiment, an object may correspond to a particular user. An object may be content associated with a particular user, or may be an account of a particular user or information stored on a social networking system 160 or other computing system. As an example and not by way of limitation, a first user may view one or more second users of an online social network through a “People You May Know” feature of the online social network or by viewing a list of friends of the first user. As an example and not by way of limitation, a first user may specify that they do not wish to see objects associated with a particular second user in their dynamic messages or friend lists. If the privacy settings of an object do not allow it to be revealed to the user, discovered by the user, or visible to the user, the object may be excluded from the search results. Although the present disclosure describes implementing privacy settings in a particular manner, the present disclosure contemplates implementing privacy settings in any suitable manner.

[0191] In certain embodiments, different objects of the same type associated with a user may have different privacy settings. Different types of objects associated with a user may have different types of privacy settings. As an example and not as a limitation, a first user may specify that the first user's status update is public, but any images shared by the first user are only visible to the first user's friends on an online social network. As another example and not as a limitation, a user may specify different privacy settings for different types of entities (such as individual users, friends of friends, followers, user groups, or corporate entities). As another example and not as a limitation, a first user may specify a group of users who can view a video posted by the first user while preventing the video from being visible to the first user's employer. In certain embodiments, different privacy settings may be provided for different user groups or user demographics. As an example and not as a limitation, a first user may specify that other users who attend the same university as the first user can view the first user's photos, but other users who are family members of the first user cannot view those same photos.

[0192] In certain embodiments, social networking system 160 may provide one or more default privacy settings for each object of a certain object type. The privacy settings of an object that are set as default may be changed by a user associated with the object. As an example and not by way of limitation, all images posted by a first user may have a default privacy setting of only visible to the first user's friends, and for a certain image, the first user may change the privacy setting of the image to be visible to friends and friends of friends.

[0193] In certain embodiments, privacy settings may allow a first user to specify (e.g., by opting out, by not opting in) whether social-networking system 160 or assistant system 140 may receive, collect, record, or store certain objects or information associated with the user for any purpose. In certain embodiments, privacy settings may allow a first user to specify whether certain applications or processes may access, store, or use certain objects or information associated with the user. Privacy settings may allow a first user to opt in or opt out of having an object or information accessed, stored, or used by a certain application or process. Social-networking system 160 or assistant system 140 may access such information in order to provide certain functionality or services to the first user, while social-networking system 160 or assistant system 140 may not access the information for any other purpose. Prior to accessing, storing, or using such an object or information, social-networking system 160 or assistant system 140 may prompt the user to provide a privacy setting that specifies which applications or processes, if any, may access, store, or use the object or information before any such action is allowed. As an example and not by way of limitation, a first user may transmit a message to a second user via an application associated with an online social network (e.g., a messaging app), and may specify a privacy setting that social networking system 160 or assistant system 140 should not store such messages.

[0194] In certain embodiments, a user may specify whether a particular type of object or information associated with a first user may be accessed, stored, or used by social networking system 160 or assistant system 140. As an example and not by way of limitation, a first user may specify that images sent by the first user via social networking system 160 or assistant system 140 may not be stored by social networking system 160 or assistant system 140. As another example and not by way of limitation, a first user may specify that messages sent from the first user to a particular second user may not be stored by social networking system 160 or assistant system 140. As yet another example and not by way of limitation, a first user may specify that all objects sent via a particular application may be saved by social networking system 160 or assistant system 140.

[0195] In certain embodiments, the privacy settings may allow the first user to specify whether a particular object or information associated with the first user may be accessed from a particular client system 130 or third-party system 170. The privacy settings may allow the first user to opt-in or opt-out of accessing objects or information from a particular device (e.g., a phone book on the user's smartphone), from a particular application (e.g., a messaging app), or from a particular system (e.g., an email server). The social networking system 160 or assistant system 140 may provide a default privacy setting for each device, system, or application, and / or may prompt the first user to specify a specific privacy setting for each context. As an example and not by way of limitation, the first user may utilize a location services feature of the social networking system 160 or assistant system 140 to provide recommendations for restaurants or other places near the user. The first user's default privacy settings may specify that the social networking system 160 or assistant system 140 may use location information provided from the first user's client device 130 to provide location-based services, but the social networking system 160 or assistant system 140 may not store the first user's location information or provide it to any third-party system 170. The first user may then update the privacy settings to allow the third-party image sharing application to use the location information to geotag the photos.

[0196] Location-based privacy settings

[0197] In certain embodiments, privacy settings may allow a user to specify one or more geographic locations from which an object may be accessed. Access or denial of access to an object may depend on the geographic location of the user attempting to access the object. As an example and not as a limitation, a user may share an object and specify that only users in the same city may access or view the object. As another example and not as a limitation, a first user may share an object and specify that the object is visible to a second user only when the first user is in a particular location. If the first user leaves the particular location, the object may no longer be visible to the second user. As another example and not as a limitation, the first user may specify that the object is visible only to a second user within a threshold distance from the first user. If the first user subsequently changes location, the original second user who could access the object may lose access, while a new group of second users may gain access when they enter within a threshold distance of the first user.

[0198] Privacy settings for user authentication and experience personalization information

[0199] In certain embodiments, social networking system 160 or assistant system 140 may have functionality that can use a user's personal or biometric information as input for user authentication or experience personalization purposes. Users may choose to utilize these features to enhance their experience on online social networks. As an example and not by way of limitation, a user may provide personal or biometric information to social networking system 160 or assistant system 140. The user's privacy settings may specify that such information may only be used for specific processes (such as authentication) and may also specify that such information may not be shared with any third-party system 170 or may not be used for other processes or applications associated with social networking system 160 or assistant system 140. As another example and not by way of limitation, social networking system 160 may provide a user with the ability to provide a voiceprint recording to an online social network. As an example and not by way of limitation, if a user wishes to utilize this functionality of an online social network, the user may provide a voice recording of his or her own voice to provide a status update on the online social network. The recording of the voice input may be compared to the user's voiceprint to determine what words the user said. The user's privacy settings may specify that such voice recordings may be used only for voice input purposes (e.g., authenticating the user, sending voice messages, improving voice recognition for use of voice manipulation features of the online social network), and further specify that such voice recordings may not be shared with any third-party system 170, or may not be used by other processes or applications associated with the social networking system 160. As another example and not by way of limitation, the social networking system 160 may provide the user with the ability to provide a reference image (e.g., a facial profile, a retinal scan) to the online social network. The online social network may compare the reference image with an image input received later (e.g., for authenticating the user, tagging the user in a photo). The user's privacy settings may specify that such voice recordings may be used only for limited purposes (e.g., authentication, tagging the user in a photo), and further specify that such voice recordings may not be shared with any third-party system 170, or may not be used by other processes or applications associated with the social networking system 160.

[0200] System and method

[0201] Fig.10An example computer system 1000 is shown. In a particular embodiment, one or more computer systems 1000 perform one or more steps of one or more methods described or shown herein. In a particular embodiment, one or more computer systems 1000 provide functionality described or shown herein. In a particular embodiment, software running on one or more computer systems 1000 performs one or more steps of one or more methods described or shown herein, or provides functionality described or shown herein. A particular embodiment includes one or more portions of one or more computer systems 1000. Herein, references to computer systems may include computing devices, and vice versa, where appropriate. In addition, references to computer systems may include one or more computer systems, where appropriate.

[0202] The present disclosure contemplates any suitable number of computer systems 1000. The present disclosure contemplates computer systems 1000 taking any suitable physical form. By way of example and not limitation, computer system 1000 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 1000 may include one or more computer systems 1000; may be monolithic or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not as a limitation, one or more computer systems 1000 may perform one or more steps of one or more methods described or shown herein in real time or in batch mode. Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or shown herein at different times or at different locations.

[0203] In a particular embodiment, computer system 1000 includes a processor 1002, a memory 1004, a storage device 1006, an input / output (I / O) interface 1008, a communication interface 1010, and a bus 1012. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0204] In a particular embodiment, the processor 1002 includes hardware for executing instructions (e.g., those that constitute a computer program). As an example and not by way of limitation, to execute instructions, the processor 1002 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 1004, or a storage device 1006; decode these instructions and execute them; and then write one or more results to an internal register, an internal cache, a memory 1004, or a storage device 1006. In a particular embodiment, the processor 1002 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1002 that includes any suitable number of any suitable internal caches. As an example and not by way of limitation, the processor 1002 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1004 or the storage device 1006, and the instruction cache may speed up the retrieval of those instructions by the processor 1002. The data in the data cache may be: a copy of data in the memory 1004 or storage device 1006 for operation by instructions executed at the processor 1002; the result of a previous instruction executed at the processor 1002, for access by a subsequent instruction executed at the processor 1002 or for writing to the memory 1004 or storage device 1006; or other suitable data. The data cache may speed up read or write operations performed by the processor 1002. The TLB may speed up virtual address translations for the processor 1002. In a particular embodiment, the processor 1002 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1002 including any suitable number of any suitable internal registers. Where appropriate, the processor 1002 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or include one or more processors 1002. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0205] In a particular embodiment, the memory 1004 includes a main memory for storing instructions for execution by the processor 1002 or data for operation by the processor 1002. As an example and not by way of limitation, the computer system 1000 can load instructions from the storage device 1006 or another source (such as, for example, another computer system 1000) to the memory 1004. The processor 1002 can then load the instructions from the memory 1004 to an internal register or an internal cache. To execute the instructions, the processor 1002 can retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 1002 can write one or more results (which can be intermediate results or final results) to the internal register or the internal cache. The processor 1002 can then write one or more of these results to the memory 1004. In a particular embodiment, the processor 1002 executes only instructions in one or more internal registers or internal caches or in the memory 1004 (rather than the storage device 1006 or elsewhere), and operates only on data in one or more internal registers or internal caches or in the memory 1004 (rather than the storage device 1006 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 1002 to the memory 1004. As described below, the bus 1012 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 1002 and the memory 1004 and facilitate access to the memory 1004 requested by the processor 1002. In a particular embodiment, the memory 1004 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1004 may include one or more memories 1004. Although this disclosure describes and illustrates particular memories, this disclosure contemplates any suitable memories.

[0206] In a particular embodiment, the storage device 1006 includes a large-capacity storage device for data or instructions. As an example and not as a limitation, the storage device 1006 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the storage device 1006 may include a removable or non-removable (or fixed) medium. Where appropriate, the storage device 1006 may be inside or outside the computer system 1000. In a particular embodiment, the storage device 1006 is a non-volatile solid-state memory. In a particular embodiment, the storage device 1006 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM), or a flash memory, or a combination of two or more of these. The present disclosure contemplates a large-capacity storage device 1006 in any suitable physical form. Where appropriate, storage 1006 may include one or more storage control units that facilitate communications between processor 1002 and storage 1006. Where appropriate, storage 1006 may include one or more storage devices 1006. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0207] In a particular embodiment, I / O interface 1008 includes hardware, software, or both that provide one or more interfaces for communication between computer system 1000 and one or more I / O devices. Where appropriate, computer system 1000 may include one or more of these I / O devices. One or more of these I / O devices may enable communication between a person and computer system 1000. As an example and not as a limitation, I / O devices may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a static camera, a stylus, a tablet computer, a touch screen, a tracking ball, a video camera, another suitable I / O device, or a combination of two or more of these. I / O devices may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1008 for them. Where appropriate, I / O interface 1008 may include one or more devices or software drivers that enable processor 1002 to drive one or more of these I / O devices. Where appropriate, I / O interface 1008 may include one or more I / O interfaces 1008. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0208] In a particular embodiment, the communication interface 1010 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1000 and one or more other computer systems 1000 or one or more networks. As an example and not by way of limitation, the communication interface 1010 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (e.g., a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 1010 for it. As an example and not by way of limitation, the computer system 1000 may communicate with one or more parts of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1000 may communicate with a wireless PAN (WPAN) such as, for example, a Bluetooth WPAN, a WI-FI network, a WI-MAX network, a cellular telephone network such as, for example, a Global System for Mobile Communications (GSM) network, or other suitable wireless networks, or a combination of two or more of these. Where appropriate, the computer system 1000 may include any suitable communication interface 1010 for any of these networks. Where appropriate, the communication interface 1010 may include one or more communication interfaces 1010. Although this disclosure describes and illustrates particular communication interfaces, this disclosure contemplates any suitable communication interface.

[0209] In a particular embodiment, bus 1012 includes hardware, software, or both that couples components of computer system 1000 to each other. By way of example and not limitation, bus 1012 may include an accelerated graphics port (AGP) or other graphics bus, an extended industry standard architecture (EISA) bus, a front side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or any other suitable bus, or a combination of two or more of these. Where appropriate, bus 1012 may include one or more buses 1012. Although the present disclosure describes and illustrates a particular bus, the present disclosure contemplates any suitable bus or interconnect.

[0210] Herein, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital (SECURE DIGITAL) cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, the computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0211] Other Miscellaneous

[0212] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated by context. Thus, herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated by context. Furthermore, "and" is both conjunctive and several, unless expressly indicated otherwise or indicated by context. Thus, herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated by context.

[0213] The scope of the present disclosure includes all changes, substitutions, variations, alterations and modifications to the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the corresponding embodiments herein as including specific parts, elements, features, functions, operations or steps, any of these embodiments may include any combination or permutation of any parts, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, references to devices or systems or parts of devices or systems that are suitable for, arranged to, capable of, configured to, implemented to, operable to, or operated to perform specific functions in the appended claims include the device, system, component, whether it or that specific function is activated, turned on or unlocked, as long as the device, system or component is adjusted, arranged, enabled, configured, implemented, operable, or operated in this way. In addition, although the present disclosure describes or illustrates specific embodiments as providing specific advantages, specific embodiments may provide some, all or none of these advantages.

Claims

1. A method comprising: receiving, from a client system associated with a first user, user input based on one or more modalities, wherein at least one of the modalities of the user input is a visual modality; identifying, based on one or more machine learning models, one or more subjects associated with the user input based on the visual modality; Based on the one or more machine learning models, determining one or more attributes associated with the one or more subjects, respectively; resolving one or more entities corresponding to the one or more subjects based on the determined one or more attributes; performing one or more tasks associated with the one or more resolved entities; and Instructions are sent to a client system associated with the first user to present communications content in response to the user input, wherein the communications content includes information associated with one or more tasks performed.

2. The method of claim 1, wherein the user input comprises one or more of the following: String; Audio editing; images; or Video editing. 3 . The method of claim 1 , wherein the one or more subjects associated with the user input include one or more of a person, a place, a business, or an object. The method of claim 3 , wherein identifying the one or more individuals is based on facial recognition. The method of claim 3 , wherein identifying the one or more objects is based on object detection. The method of claim 1 , further comprising generating a feature representation for the user input based on the visual modality.

7. The method of claim 1, wherein the one or more machine learning models include one or more of the following: Support vector machines; Regression models; or Convolutional Neural Networks.

8. The method of claim 1, further comprising identifying one or more intents and one or more slots based on the user input.

9. The method according to claim 8, in, One or more tasks associated with the one or more resolved entities are performed based on the identified intents and slots.

10. The method according to claim 1, in, The communication content includes one or more of the following: String; Audio editing; images; or Video editing.

Images