Automated completion of gesture input in assistant system

Abstract

The invention relates to automatic completion of gesture input in an assistant system. In one embodiment, a method includes receiving a user input from a client system associated with a first user, where the user input includes an incomplete gesture performed by the first user, calculating, by an intent understanding module, one or more confidence scores corresponding to one or more intent of the incomplete gesture, determining that the calculated confidence score associated with each intent is below a threshold score, in response to determining that the calculated confidence score for each intent is below the threshold score, selecting a candidate gesture from a plurality of predefined gestures based on a personalized gesture recognition model, wherein each candidate gesture is associated with a confidence score representing a likelihood that the first user intends to input the respective candidate gesture, and sending an instruction to the client system for presenting one or more suggested inputs corresponding to the one or more candidate gestures.

Description

[0001] This application is a divisional application of an application with an application date of April 19, 2019, application number 201980040312.6, and invention name “Automatic completion of gesture input in an assistant system”. Technical Field

[0002] The present disclosure generally relates to conversation management based on machine learning techniques within a network environment, and specifically to hardware and software for a smart assistant system. background

[0004] The assistant system can provide information or services on behalf of the user based on a combination of user input, location awareness, and the ability to access information from various online sources (such as weather conditions, traffic congestion, news, stock prices, user schedules, retail prices, etc.). User input can include text (e.g., online chat) (especially text in instant messaging applications or other applications), voice, images, motion, orientation, or a combination thereof. The assistant system can perform concierge-type services (e.g., booking dinner, purchasing event tickets, arranging travel) or provide information based on user input. The assistant system can also perform management or data processing tasks based on online information and events without user initiation or interaction. Examples of those tasks that can be performed by the assistant system can include schedule management (e.g., sending a warning message to a dinner appointment that the user is going to be late due to traffic conditions, updating the schedules of both parties, and changing restaurant reservation times). The assistant system can be implemented through a combination of computing devices, application programming interfaces (APIs), and application proliferation on user devices.

[0005] A social networking system, which may include a social networking website, may enable its users (e.g., individuals or organizations) to interact with it and with each other through it. The social networking system may utilize input from a user to create and store a user profile associated with the user in the social networking system. The user profile may include demographic information, communication channel information, and information about personal interests of the user. The social networking system may also utilize input from the user to create and store a record of the user's relationships with other users of the social networking system, as well as provide services (e.g., profile / news feed postings, photo sharing, event organization, messaging, games, or advertising) to facilitate social interaction between or among users.

[0006] The social networking system may send content or messages related to its services to the user's mobile device or other computing device through one or more networks. The user may also install a software application on the user's mobile device or other computing device for accessing the user's user profile and other data within the social networking system. The social networking system may generate a set of personalized content objects to display to the user, such as a dynamic message of aggregated stories of other users connected to the user.

[0007] Overview of Certain Embodiments

[0008] In a particular embodiment, the assistant system can help users obtain information or services. The assistant system can enable users to interact with it through multi-modal user input (such as sound, text, image, video, motion, orientation) in stateful and multi-turn conversations to obtain help. The assistant system can create and store a user profile, which includes personal information and contextual information associated with the user. In a particular embodiment, the assistant system can use natural language understanding to analyze user input. The analysis can be based on the user profile to obtain a more personalized and context-aware understanding. The assistant system can resolve entities associated with user input based on the analysis. In a particular embodiment, the assistant system can interact with different agents to obtain information or services associated with the resolved entity. The assistant system can generate responses about information or services for users by using natural language generation. Through interaction with the user, the assistant system can use dialogue management technology to manage and forward the conversation flow with the user. In a particular embodiment, the assistant system can also assist users to digest the information obtained effectively and efficiently by summarizing the information. The assistant system can also help users better participate in online social networks by providing tools that help users interact with the online social network (e.g., create posts, comments, messages). The assistant system can additionally help users manage different tasks, such as keeping track of events. In certain embodiments, the assistant system can proactively perform tasks related to the user's interests and preferences based on the user profile without user input. In certain embodiments, the assistant system can check privacy settings to ensure that access to the user's profile or other user information and the performance of different tasks are allowed according to the user's privacy settings.

[0009] In a particular embodiment, the assistant system may receive an initial input in a first modality performed by a user (e.g., a gesture input to a virtual reality (VR) headset or augmented reality (AR) smart glasses), the initial input as an interaction with an assistant-based client system, determine candidate continuation-inputs in an auto-complete manner based on the initial input, and recommend these candidate continuation-inputs to the user in one or more second modalities (e.g., text), for which the user can select one to trigger the execution of a specific task. The initial input and the continuation input may be based on any suitable modality, including text, voice, image, video, motion, orientation, etc. In addition, the modality of the initial input and the modality of the continuation input may be different. As an example and not limitation, a user wearing AR glasses to visit Paris may be looking at the Eiffel Tower. Based on the gaze input, the assistant system may suggest a gesture for taking a picture displayed on a virtual screen, or suggest that the user say "take a picture". The user may perform the suggested gesture or say a sentence to allow the assistant system to perform the task of taking a picture of the Eiffel Tower. Thus, an assistant system may have the ability to process user input in one modality and generate suggested input in one or more other modalities, which may be referred to as auto-completion for multimodal user input. Although this disclosure describes generating a particular auto-completion for a particular multimodal user input in a particular manner via a particular system, this disclosure contemplates any suitable auto-completion for generating any suitable multimodal user input in any suitable manner via any suitable system.

[0010] In a particular embodiment, the assistant system may receive an initial input from a first user from a client system associated with the first user. The initial input may be in a first modality. In a particular embodiment, the assistant system may determine one or more intents corresponding to the initial input through an intent understanding module. The assistant system may then generate one or more candidate continuation inputs based on the one or more intents. The one or more candidate continuation inputs may be in one or more candidate modalities, respectively. The candidate modality may be different from the first modality. In a particular embodiment, the assistant system may also send an instruction to the client system for presenting one or more suggested inputs corresponding to the one or more candidate continuation inputs.

[0011] In a particular embodiment, the assistant system may receive an incomplete gesture performed by a user as an interaction with an assistant-based client system (e.g., a VR head-mounted device or AR smart glasses), determine candidate gestures in an automatic completion manner based on the incomplete gesture, recommend these candidate gestures to the user, and for these candidate gestures, the user can select one of them to trigger the execution of a specific task. As an example and not a limitation, a user wearing AR glasses may not know what gesture to perform to trigger a specific function of the assistant system. The user may start to move his / her hand, but pause in the air. Therefore, the assistant system can analyze the user's incomplete gesture and determine possible candidate gestures. The assistant system can visually display these candidate gestures to the user through the AR glasses. After the user selects one of these candidate gestures, the assistant system can also perform the task corresponding to the gesture. In short, the assistant system can use gesture recognition technology to determine the gesture of the user's intention for an incomplete gesture, and suggest the determined gesture to the user as a guide, which can be referred to as automatic completion of gesture input. Although the present disclosure describes a specific automatic completion of generating a specific gesture input via a specific system in a specific manner, the present disclosure contemplates any suitable automatic completion of generating any suitable gesture input via any suitable system in any suitable manner.

[0012] In a particular embodiment, the assistant system may receive user input from the first user from a client system associated with the first user. The user input may include an incomplete gesture performed by the first user. In a particular embodiment, the assistant system may calculate one or more confidence scores corresponding to one or more intents of the incomplete gesture through the intent understanding module. The assistant system may then determine that the calculated confidence score associated with each intent is lower than a threshold score. In a particular embodiment, in response to determining that the calculated confidence score of each intent is lower than a threshold score, the assistant system may select one or more candidate gestures from a plurality of predefined gestures based on a personalized gesture recognition model. Each candidate gesture may be associated with a confidence score representing the likelihood that the first user intends to input the corresponding candidate gesture. In a particular embodiment, the assistant system may also send an instruction to the client system for presenting one or more suggested inputs corresponding to the one or more candidate gestures.

[0013] In order to achieve the goal of suggesting multimodal user input for automatic completion, there are certain technical challenges. One technical challenge may include determining candidate continuation inputs based on an initial input. The solution to the above challenges proposed by the embodiments disclosed herein is to determine candidate continuation inputs based on the intent of the initial input and potential entities related to the intent or objects associated with the initial input, because such information may be necessary to determine what continuation input the user can use to interact with the assistant system to perform tasks based on the intent and the entity or object. Another technical challenge may include determining that the user needs to suggest input. The solution to this challenge proposed by the embodiments disclosed herein is to determine whether to suggest candidate continuation inputs based on different factors, including wake-up input, gaze information, contextual information, and intent, because such information can reveal the user's need for help from different perspectives.

[0014] Certain embodiments disclosed herein may provide one or more technical advantages. A technical advantage of an embodiment may include enriching the user's experience with the assistant system because the user can interact with the assistant system using inputs of various modalities. Another technical advantage of an embodiment may include increasing the degree of user participation in the assistant system by guiding the user to explore various tasks for which the user can request help with suggested input sequences. Certain embodiments disclosed herein may provide none, some, or all of the above technical advantages. In view of the drawings, description, and claims of the present disclosure, one or more other technical advantages may be apparent to those skilled in the art.

[0015] There are certain technical challenges to achieving the goal of suggesting candidate gestures for automatic completion. One technical challenge includes determining the similarity level of candidate gestures relative to incomplete gestures. The solution to the above challenges proposed by the embodiments disclosed herein can determine the similarity based on different factors, including the trajectory, orientation, object, contextual information, and location associated with the incomplete gesture, because such information can provide different informative cues for the assistant system to calculate the similarity. Another technical challenge includes determining that the user needs a suggested input for a gesture. The solution to this challenge proposed by the embodiments disclosed herein is to determine whether to suggest a candidate gesture based on different factors, including the confidence score of the intent, the wake-up gesture, the time information of the incomplete gesture, the speed of the incomplete gesture, and the pattern of the incomplete gesture, because such information can reveal the user's need for help from different angles.

[0016] Certain embodiments disclosed herein may provide one or more technical advantages. A technical advantage of an embodiment may include improving the user's experience with an assistant system because the assistant system may understand the user's intent based on incomplete gestures and teach the user to complete incomplete gestures to interact with the assistant system. Another technical advantage of an embodiment may include increasing the user's involvement in the assistant system by guiding the user to explore various tasks for which the user can request help with suggested gesture sequences. Certain embodiments disclosed herein may provide none, some, or all of the above technical advantages. In view of the drawings, description, and claims of the present disclosure, one or more other technical advantages may be apparent to those skilled in the art.

[0017] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to them. Specific embodiments may include all, some or none of the components, elements, features, functions, operations or steps of the embodiments disclosed herein. Embodiments according to the present invention are specifically disclosed in the attached claims relating to methods, storage media, systems, assistant systems and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or back references in the attached claims are selected only for formal reasons. However, any subject matter generated by intentional back references (especially multiple references) to any previous claim may also be claimed, so that any combination of claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features as set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any features of the appended claims.

[0018] In an embodiment, a method, particularly for use in an assistant system, for assisting a user in obtaining information or services by enabling the user to interact with the assistant system in a conversation using user input to obtain assistance, wherein the user input includes sound, text, image, video, motion or orientation, or any combination thereof, the assistant system being particularly implemented by a combination of a computing device, an application programming interface (API), and application proliferation on a user device, may include, by one or more computing systems:

[0019] receiving, from a client system associated with a first user, user input comprising an incomplete gesture performed by the first user;

[0020] calculating, by the intent understanding module, one or more confidence scores corresponding to the one or more intents of the incomplete gesture;

[0021] determining that the calculated confidence score associated with each intent is below a threshold score;

[0022] In response to determining that the calculated confidence score for each intent is below a threshold score, selecting one or more candidate gestures from a plurality of predefined gestures based on the personalized gesture recognition model, wherein each candidate gesture is associated with a confidence score representing a likelihood that the first user intended to input the corresponding candidate gesture; and

[0023] Instructions are sent to the client system to present one or more suggested inputs corresponding to the one or more candidate gestures.

[0024] In an embodiment, a method may include:

[0025] For each of the one or more candidate gestures, a similarity of the candidate gesture relative to the incomplete gesture is calculated.

[0026] The similarity of each candidate gesture relative to the incomplete gesture can be based on the trajectory of the incomplete gesture relative to the client system.

[0027] The similarity of each candidate gesture relative to the incomplete gesture can be based on the orientation of the incomplete gesture relative to the client system.

[0028] The similarity of each candidate gesture relative to the incomplete gesture may be based on an object associated with the incomplete gesture.

[0029] The similarity of each candidate gesture relative to the incomplete gesture may be based on contextual information associated with the incomplete gesture.

[0030] The similarity of each candidate gesture relative to the incomplete gesture can be based on the location of the incomplete gesture relative to the client system.

[0031] The threshold score may be based on a wake-up gesture performed by the first user.

[0032] Calculating one or more confidence scores corresponding to one or more intents of an incomplete gesture can be based on a speed associated with the incomplete gesture.

[0033] Calculating one or more confidence scores corresponding to one or more intents of the incomplete gesture can be based on time information associated with the incomplete gesture, and where the time information includes pauses in the user input.

[0034] In an embodiment, a method may include:

[0035] receiving, from the client system, a user-selected input from a first user, wherein the user-selected input comprises one of the suggested inputs; and

[0036] One or more tasks are performed based on the user selected input.

[0037] Selection of one or more candidate gestures can be based on one or more intents.

[0038] Each predefined gesture may include one or more of point, poke, tap, wave, or swipe.

[0039] In an embodiment, a method may include:

[0040] receiving, from a client system, a first user-selected input from a first user, wherein the first user-selected input comprises one of the suggested inputs, and wherein the first user-selected input is associated with a first intent;

[0041] generating one or more additional candidate gestures based on the first user-selected input, wherein each of the one or more additional candidate gestures is associated with the first intent;

[0042] sending instructions to the client system to present one or more additional suggested inputs corresponding to the one or more additional candidate gestures;

[0043] receiving, from the client system, a second user-selected input from the first user, wherein the second user-selected input includes one of the additional suggested inputs; and

[0044] One or more tasks are performed based on the second user selected input.

[0045] An incomplete gesture may include one or more of a paused movement of a user's hand, a pause of an open user's hand in the air, an onset of a user's gesture, which may indicate that the user has not completed the entire gesture.

[0046] The client system may be a virtual reality (VR) headset and / or augmented reality (AR) smart glasses.

[0047] The client system may receive user input from an I / O device including one or more sensors.

[0048] The I / O devices may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a video camera, another suitable I / O device, or a combination of two or more of these.

[0049] The intent understanding module may include a natural language understanding module and a gesture classification model.

[0050] The gesture classification model may include a machine learning model that is trained offline to recognize different categories of gestures performed by a user.

[0051] The gesture classification model may be based on one or more of a convolutional neural network, tensor flow, or a hidden Markov model.

[0052] In an embodiment, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to:

[0053] receiving, from a client system associated with a first user, user input comprising an incomplete gesture performed by the first user;

[0054] calculating, by the intent understanding module, one or more confidence scores corresponding to the one or more intents of the incomplete gesture;

[0055] determining that the calculated confidence score associated with each intent is below a threshold score;

[0056] In response to determining that the calculated confidence score for each intent is below a threshold score, selecting one or more candidate gestures from a plurality of predefined gestures based on the personalized gesture recognition model, wherein each candidate gesture is associated with a confidence score representing a likelihood that the first user intended to input the corresponding candidate gesture; and

[0057] Instructions are sent to the client system to present one or more suggested inputs corresponding to the one or more candidate gestures.

[0058] The software, when executed, is operable to:

[0059] For each of the one or more candidate gestures, a similarity of the candidate gesture relative to the incomplete gesture is calculated.

[0060] The similarity of each candidate gesture relative to the incomplete gesture can be based on the trajectory of the incomplete gesture relative to the client system.

[0061] The similarity of each candidate gesture relative to the incomplete gesture can be based on the orientation of the incomplete gesture relative to the client system.

[0062] The similarity of each candidate gesture relative to the incomplete gesture may be based on an object associated with the incomplete gesture.

[0063] In an embodiment, a system may include: one or more processors; and a non-transitory memory coupled to the processor, the non-transitory memory including instructions executable by the processor, the processor being operable when executing the instructions to:

[0064] receiving, from a client system associated with a first user, user input comprising an incomplete gesture performed by the first user;

[0065] calculating, by the intent understanding module, one or more confidence scores corresponding to the one or more intents of the incomplete gesture;

[0066] determining that the calculated confidence score associated with each intent is below a threshold score;

[0067] In response to determining that the calculated confidence score for each intent is below a threshold score, selecting one or more candidate gestures from a plurality of predefined gestures based on the personalized gesture recognition model, wherein each candidate gesture is associated with a confidence score representing a likelihood that the first user intended to input the corresponding candidate gesture; and

[0068] Instructions are sent to the client system to present one or more suggested inputs corresponding to the one or more candidate gestures.

[0069] In an embodiment, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to perform a method according to the present invention or any of the above-mentioned embodiments.

[0070] In an embodiment, a system may include: one or more processors; and at least one memory coupled to the processor and including instructions executable by the processor, the processor being operable to perform a method according to the present invention or any of the above-mentioned embodiments when executing the instructions.

[0071] In an embodiment, a computer program product, preferably comprising a computer readable non-transitory storage medium, is operable to perform a method according to the invention or any of the above mentioned embodiments when executed on a data processing system.

[0072] In an embodiment, an assistant system is used to help a user obtain information or services by enabling the user to interact with the assistant system using user input in a conversation to obtain help, wherein the user input includes sound, text, image, video, motion or orientation, or any combination thereof. The assistant system is specifically implemented through a combination of a computing device, an application programming interface (API), and a proliferation of applications on a user device. The system may include: one or more processors; and a non-volatile memory coupled to the processor, the memory including instructions executable by the processor, and the processor is operable to perform a method according to the present invention or any of the above-mentioned embodiments when executing the instructions.

[0073] In an embodiment, the assistant system may assist the user by performing at least one or more of the following features or steps:

[0074] - Create and store user profiles that include personal and contextual information associated with the user

[0075] - Analyze user input using an intent understanding module, where the intent understanding module may include a natural language understanding module and a gesture classification model

[0076] - Use natural language understanding to analyze user input, where the analysis can be based on user profiles for more personalized and context-aware understanding

[0077] - Resolve entities associated with user input based on analysis

[0078] -Interact with different agents to obtain information or services associated with the resolved entity

[0079] - Generate responses for users regarding information or services by using natural language generation

[0080] - Manage and forward conversation flows with users through interaction with them using conversation management techniques

[0081] - Help users digest the information they have obtained effectively and efficiently by aggregating the information

[0082] - Help users better participate in online social networks by providing tools that help users interact with online social networks (e.g., create posts, comments, messages)

[0083] - Helps users manage different tasks, such as keeping track of events

[0084] - Proactively perform pre-authorized tasks relevant to the user’s interests and preferences, at a time relevant to the user, based on the user’s profile, and without user input

[0085] - Review privacy settings where necessary to ensure that accessing a user profile and performing different tasks complies with the user's privacy settings.

[0086] In an embodiment, the assistant system may include at least one or more of the following components:

[0087] a messaging platform for receiving a text-based user input and / or receiving an image or video-based user input from a client system associated with a user and processing the image or video-based user input within the messaging platform using optical character recognition technology to convert the user input into text,

[0088] an audio speech recognition (ASR) module for receiving audio modality-based user input from a client system associated with a user (e.g., a user may speak into or send a video that includes voice) and converting the audio modality-based user input into text,

[0089] - Assistant xbot, which is used to receive the output of the messaging platform or the ASR module.

[0090] In an embodiment, a system may include:

[0091] at least one client system, in particular an electronic device,

[0092] At least one assistant system according to the present invention or any embodiment herein,

[0093] The client system and the assistant system are connected to each other in particular via a network,

[0094] The client system includes an assistant application for allowing a user of the client system to interact with the assistant system.

[0095] wherein the assistant application transmits the user input to the assistant system, and based on the user input, the assistant system generates a response and sends the generated response to the assistant application, and the assistant application presents the response to the user of the client system,

[0096] In particular, the user input is audio or verbal, and the response may be textual or also audio or verbal.

[0097] In an embodiment, a system may include a social networking system,

[0098] The client system particularly includes a social network application for accessing the social network system. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 An example network environment associated with an assistant system is shown.

[0101] Figure 2 An example architecture for an assistant system is shown.

[0102] Figure 3 An example flow chart of an assistant system responding to a user request is shown.

[0103] Figure 4 An example workflow for processing user input including gesture input is shown.

[0104] Figure 5 An example scenario for automatic completion of multimodal user input is shown.

[0105] Figure 6 An example scenario of automatic completion of gesture input is shown.

[0106] Fig. 7A An example scenario of an incomplete gesture in an AR / VR setting is shown.

[0107] Figure 7B An example scenario of suggesting gesture input in an AR / VR setting is shown.

[0108] Figure 8 An example method for suggesting multimodal user input for auto-completion is shown.

[0109] Fig. 9 An example method for suggesting candidate gestures for auto-completion is shown.

[0110] Fig.10 An example social graph is shown.

[0111] Fig.11 An example view of the embedding space is shown.

[0112] Fig.12 An example artificial neural network is shown.

[0113] Fig.13 An example computer system is shown.

[0114] Description of Example Embodiments

[0115] Systematic review

[0116] Figure 1 An example network environment 100 associated with an assistant system is shown. The network environment 100 includes a client system 130, an assistant system 140, a social networking system 160, and a third-party system 170 connected to each other via a network 110. Figure 1A particular arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110 is shown, but the present disclosure contemplates any suitable arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110. By way of example and not limitation, two or more of client system 130, social networking system 160, assistant system 140, and third-party system 170 may be directly connected to each other, bypassing network 110. As another example, two or more of client system 130, assistant system 140, social networking system 160, and third-party system 170 may be physically or logically co-located with each other, in whole or in part. Furthermore, although Figure 1 A particular number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110 are shown, but this disclosure contemplates any suitable number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110. By way of example and not limitation, network environment 100 may include a plurality of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110.

[0117] The present disclosure contemplates any suitable network 110. By way of example and not limitation, one or more portions of network 110 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 110 may include one or more networks 110.

[0118] Links 150 may connect client systems 130, assistant systems 140, social networking systems 160, and third-party systems 170 to communication network 110 or to each other. The present disclosure contemplates any suitable links 150. In certain embodiments, one or more links 150 include one or more wired links (e.g., digital subscriber lines (DSL) or cable data service interface specifications (DOCSIS)), wireless links (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical links (e.g., synchronous optical networks (SONET) or synchronous digital hierarchies (SDH)). In certain embodiments, one or more links 150 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 150, or a combination of two or more such links 150. Links 150 need not necessarily be the same throughout network environment 100. One or more first links 150 may differ from one or more second links 150 in one or more aspects.

[0119] In certain embodiments, client system 130 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more such components, and is capable of performing appropriate functions implemented or supported by client system 130. By way of example and not limitation, client system 130 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a smart speaker, a virtual reality (VR) headset, an augmented reality (AR) smart glasses, other suitable electronic devices, or any suitable combination thereof. In certain embodiments, client system 130 may be a smart assistant device. More information about smart assistant devices can be found in U.S. Patent Application No. 15 / 949,011, filed April 9, 2018, U.S. Patent Application No. 62 / 655,751, filed April 10, 2018, U.S. Design Patent Application No. 29 / 631,910, filed January 3, 2018, U.S. Design Patent Application No. 29 / 631,747, filed January 2, 2018, U.S. Design Patent Application No. 29 / 631,913, filed January 3, 2018, and U.S. Design Patent Application No. 29 / 631,914, filed January 3, 2018, each of which is incorporated by reference. The present disclosure contemplates any suitable client system 130. Client system 130 can enable network users at client system 130 to access network 110. Client system 130 can enable its users to communicate with other users at other client systems 130.

[0120] In a particular embodiment, the client system 130 may include a web browser 132, such as MICROSOFTINTERNET EXPLORER, GOOGLE CHROME or MOZILLA FIREFOX, and may have one or more add-ons, plug-ins or other extensions, such as TOOLBAR or YAHOO TOOLBAR. A user at the client system 130 may enter a uniform resource locator (URL) or other address that directs the web browser 132 to a specific server (such as server 162 or a server associated with a third-party system 170), and the web browser 132 may generate a hypertext transfer protocol (HTTP) request and pass the HTTP request to the server. The server may accept the HTTP request and pass one or more hypertext markup language (HTML) files to the client system 130 in response to the HTTP request. The client system 130 may display a web interface (such as a web page) based on an HTML file from a server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not limitation, a web interface may be displayed according to an HTML file, an extensible hypertext markup language (XHTML) file or an extensible markup language (XML) file according to specific needs. Such an interface may also execute scripts, such as, for example and without limitation, scripts written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, a combination of markup language and script (e.g., AJAX (Asynchronous JAVASCRIPT and XML)), etc. Herein, references to a web interface include one or more corresponding source files (which a browser may use to render the web interface), and vice versa, where appropriate.

[0121] In certain embodiments, client system 130 may include social network application 134 installed on client system 130. A user at client system 130 may use social network application 134 to access an online social network. A user at client system 130 may use social network application 134 to communicate with the user's social connections (e.g., friends, followers, followed accounts, contacts, etc.). A user at client system 130 may also use social network application 134 to interact with a plurality of content objects (e.g., posts, news articles, temporary content, etc.) on an online social network. By way of example and not limitation, a user may use social network application 134 to browse trending topics and breaking news.

[0122] In certain embodiments, the client system 130 may include an assistant application 136. The user of the client system 130 may use the assistant application 136 to interact with the assistant system 140. In certain embodiments, the assistant application 136 may include an independent application. In certain embodiments, the assistant application 136 may be integrated into the social network application 134 or another suitable application (e.g., a messaging application). In certain embodiments, the assistant application 136 may also be integrated into the client system 130, the assistant hardware device, or any other suitable hardware device. In certain embodiments, the assistant application 136 may be accessed via a web browser 132. In certain embodiments, the user may provide input via different modalities. As an example and not limitation, the modality may include audio, text, image, video, motion, orientation, etc. The assistant application 136 may transmit the user input to the assistant system 140. Based on the user input, the assistant system 140 may generate a response. The assistant system 140 may send the generated response to the assistant application 136. Then, the assistant application 136 may present the response to the user of the client system 130. The presented response may be based on different modalities, such as audio, text, image, and video. As an example and not limitation, a user may verbally ask assistant application 136 for traffic information (i.e., via an audio modality). Assistant application 136 may then transmit the request to assistant system 140. Assistant system 140 may generate results accordingly and send them back to assistant application 136. Assistant application 136 may also present the results to the user in text.

[0123] In a particular embodiment, the assistant system 140 can help the user retrieve information from different sources. The assistant system 140 can also help the user request services from different service providers. In a particular embodiment, the assistant system 140 can receive a user's request for information or services via the assistant application 136 in the client system 130. The assistant system 140 can use natural language understanding to analyze the user request based on the user profile and other relevant information. The results of the analysis may include different entities associated with online social networks. The assistant system 140 can then retrieve information or request services associated with these entities. In a particular embodiment, when retrieving information or requesting services for the user, the assistant system 140 can interact with the social networking system 160 and / or the third-party system 170. In a particular embodiment, the assistant system 140 can generate personalized communication content for the user using natural language generation technology. The personalized communication content may include, for example, the status of the retrieved information or the requested service. In a particular embodiment, the assistant system 140 can enable the user to interact with it about information or services in a stateful and multi-round conversation by using dialogue management technology. In the following Figure 2 The functionality of assistant system 140 is described in more detail in the discussion of .

[0124] In certain embodiments, social networking system 160 may be a network-addressable computing system that may host an online social network. Social networking system 160 may generate, store, receive, and transmit social networking data (such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to an online social network). Social networking system 160 may be accessed by other components of network environment 100 directly or via network 110. By way of example and not limitation, client system 130 may access social networking system 160 directly or via network 110 using a web browser 132 or a native application associated with social networking system 160 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In certain embodiments, social networking system 160 may include one or more servers 162. Each server 162 may be a unitary server or a distributed server across multiple computers or multiple data centers. The server 162 may be of various types, such as, without limitation, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 162 may include a hardware, software, or embedded logic component, or a combination of two or more such components for performing the appropriate functions implemented or supported by the server 162. In a particular embodiment, the social networking system 160 may include one or more data storage devices 164. The data storage device 164 may be used to store various types of information. In a particular embodiment, the information stored in the data storage device 164 may be organized according to a particular data structure. In a particular embodiment, each data storage device 164 may be a relational database, a columnar database, a correlation database, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. Particular embodiments may provide an interface that enables client system 130 , social networking system 160 , assistant system 140 , or third-party system 170 to manage, retrieve, modify, add, or delete information stored in data store 164 .

[0125] In certain embodiments, social networking system 160 may store one or more social graphs in one or more data stores 164. In certain embodiments, a social graph may include a plurality of nodes, which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept), and a plurality of edges connecting the nodes. Social networking system 160 may provide users of an online social network with the ability to communicate and interact with other users. In certain embodiments, a user may join an online social network via social networking system 160 and then add connections (e.g., relationships) with a plurality of other users in social networking system 160 to whom they want to be connected. Herein, the term "friend" may refer to any other user of social networking system 160 with whom the user forms a connection, association, or relationship via social networking system 160.

[0126] In particular embodiments, social-networking system 160 may provide users with the ability to take actions on various types of items or objects supported by social-networking system 160. By way of example and not limitation, items and objects may include groups or social networks to which users of social-networking system 160 may belong, events or calendar entries that may be of interest to users, computer-based applications that users may use, transactions that allow users to buy or sell goods via a service, interactions with advertisements that users may perform, or other suitable items or objects. Users may interact with anything that can be represented in social-networking system 160 or by an external system such as third-party system 170 that is separate from social-networking system 160 and coupled to social-networking system 160 via network 110.

[0127] In certain embodiments, social networking system 160 can link various entities. By way of example and not limitation, social networking system 160 can enable users to interact with each other and receive content from third-party systems 170 or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0128] In certain embodiments, third-party systems 170 may include one or more types of servers, one or more data stores, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable components (e.g., the server may communicate with these components). Third-party systems 170 may be operated by a different entity than the entity that operates social-networking system 160. However, in certain embodiments, social-networking system 160 and third-party systems 170 may operate in conjunction with each other to provide social-networking services to users of social-networking system 160 or third-party systems 170. In this sense, social-networking system 160 may provide a platform or backbone that other systems (e.g., third-party systems 170) may use to provide social-networking services and functionality to users across the Internet.

[0129] In certain embodiments, third-party system 170 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be delivered to client system 130. For example, and not limitation, a content object may include information about things or activities of interest to a user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example and not limitation, a content object may include an incentive content object (e.g., a coupon, discount voucher, gift certificate, or other suitable incentive object).

[0130] In certain embodiments, social networking system 160 also includes user-generated content objects that can enhance a user's interaction with social networking system 160. User-generated content can include any content that a user can add, upload, send, or "post" to social networking system 160. By way of example and not limitation, a user transmits a post from client system 130 to social networking system 160. Posts can include data, such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to social networking system 160 by third parties through "communication channels" such as dynamic messages or streams.

[0131] In certain embodiments, the social networking system 160 may include various servers, subsystems, programs, modules, logs, and data storage. In certain embodiments, the social networking system 160 may include one or more of the following: a web server, an action recorder, an API request server, a correlation and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertisement-targeting module, a user interface module, a user profile storage, a relationship storage, a third-party content storage, or a location storage. The social networking system 160 may also include suitable components, such as a network interface, a security mechanism, a load balancer, a failover server, a management and network operation console, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 160 may include one or more user profile storages for storing user profiles. A user profile may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, affinities, or locations). Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not limitation, if a user "likes" an article about a brand of shoes, the category may be the brand, or the general category of "shoes" or "clothing". The association storage may be used to store association information about users. The association information may indicate users with similar or common work experience, group membership, hobbies, educational history, or who are related or share common attributes in any way. The association information may also include user-defined associations between different users and content (internal and external). The web server may be used to link the social networking system 160 to one or more client systems 130 or one or more third-party systems 170 via the network 110. The web server may include a mail server or other messaging functions for receiving and routing messages between the social networking system 160 and one or more client systems 130. The API request server may allow the assistant system 140 and the third-party system 170 to access information from the social networking system 160 by calling one or more APIs. The action recorder may be used to receive communications about the user's actions on or outside the social networking system 160 from the web server. In conjunction with the action log, a third-party content object log of user exposure to third-party content objects can be maintained. The notification controller can provide information about content objects to the client system 130. The information can be pushed to the client system 130 as a notification, or the information can be pulled from the client system 130 in response to a request received from the client system 130.The authorization server may be used to enforce one or more privacy settings of users of the social networking system 160. The privacy settings of the user determine how certain information associated with the user may be shared. The authorization server may allow the user to opt-in or opt-out of having their actions recorded by the social networking system 160 or shared with other systems (e.g., third-party systems 170), such as by setting appropriate privacy settings. A third-party content object store may be used to store content objects received from a third party (e.g., third-party system 170). A location store may be used to store location information received from a client system 130 associated with the user. The advertising pricing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0132] Assistant System

[0133] Figure 2An example architecture of an assistant system 140 is shown. In a particular embodiment, the assistant system 140 can help users obtain information or services. The assistant system 140 can enable users to interact with it in a stateful and multi-turn conversation with multimodal user input (such as sound, text, image, video, motion, orientation) to obtain help. The assistant system 140 can create and store a user profile including personal information and contextual information associated with the user. In a particular embodiment, the assistant system 140 can use natural language understanding to analyze user input. The analysis can be based on the user profile to obtain a more personalized and context-aware understanding. The assistant system 140 can parse entities associated with the user input based on the analysis. In a particular embodiment, the assistant system 140 can interact with different agents to obtain information or services associated with the parsed entity. The assistant system 140 can generate responses about information or services for the user by using natural language generation. Through interaction with the user, the assistant system 140 can use dialogue management technology to manage and forward the conversation flow with the user. In a particular embodiment, the assistant system 140 can also help users digest the information obtained effectively and efficiently by summarizing the information. The assistant system 140 can also help users better participate in online social networks by providing tools that help users interact with online social networks (e.g., create posts, comments, messages). The assistant system 140 can also help users manage different tasks, such as keeping track of events. In certain embodiments, the assistant system 140 can proactively perform pre-authorized tasks related to user interests and preferences based on the user profile at a time relevant to the user without user input. In certain embodiments, the assistant system 140 can check privacy settings to ensure that access to the user's profile or other user information and the performance of different tasks are allowed according to the user's privacy settings. More information about helping users according to privacy settings can be found in U.S. Patent Application No. 62 / 675,090, filed on May 22, 2018, which is incorporated by reference.

[0134] In certain embodiments, the assistant system 140 may receive user input from an assistant application 136 in a client system 130 associated with a user. In certain embodiments, the user input may be a user-generated input that is sent to the assistant system 140 in a single round. If the user input is based on a text modality, the assistant system 140 may receive it at the messaging platform 205. If the user input is based on an audio modality (e.g., the user may speak to the assistant application 136 or send a video including voice to the assistant application 136), the assistant system 140 may process it using an automatic speech recognition (ASR) module 210 to convert the user input into text. If the user input is based on an image or video modality, the assistant system 140 may process it using optical character recognition technology within the messaging platform 205 to convert the user input into text. The output of the messaging platform 205 or the ASR module 210 may be received at the assistant xbot 215. More information on processing user input based on different modalities may be found in U.S. Patent Application No. 16 / 053600 filed on August 2, 2018, which is incorporated by reference.

[0135] In a particular embodiment, assistant xbot 215 can be a type of chatbot. Assistant xbot 215 can include a programmable service channel, which can be a software code, logic or routine used as a user personal assistant. Assistant xbot 215 can serve as a user portal of assistant system 140. Therefore, assistant xbot 215 can be considered as a type of conversational agent. In a particular embodiment, assistant xbot 215 can send text user input to natural language understanding (NLU) module 220 to interpret user input. In a particular embodiment, NLU module 220 can obtain information from user context engine 225 and semantic information aggregator (semantic information aggregator, SIA) 230 to accurately understand user input. User context engine 225 can store user profiles of users. User profiles of users can include user profile data, which includes demographic information, social information and context information associated with users. User profile data can also include user interests and preferences for multiple topics aggregated by dynamic messages, search logs, conversations on messaging platforms 205, etc. The use of user profiles may be protected by the privacy check module 245 to ensure that the user's information can only be used for his / her benefit and not shared with anyone else. More information about user profiles can be found in U.S. Patent Application No. 15 / 967,239 filed on April 30, 2018, which is incorporated by reference. The semantic information aggregator 230 may provide ontology data associated with a plurality of predefined domains, intents, and slots to the NLU module 220. In a particular embodiment, a domain may represent a social context of an interaction, e.g., education. An intent may be an element in a predefined classification of semantic intents, which may indicate the purpose of a user's interaction with the assistant system 140. In a particular embodiment, if the user input includes text / voice input, the intent may be an output of the NLU module 220. The NLU module 220 may classify the text / voice input as a member of a predefined classification, e.g., for the input "play Beethoven's Fifth Symphony", the NLU module 220 may classify the input as having the intent [IN: play_music]. In certain embodiments, a domain can be conceptually a namespace for a set of intents, e.g., music. A slot can be a named substring with user input that represents a basic semantic entity. For example, a slot for "pizza" can be [SL: plate]. In certain embodiments, the set of valid or expected named slots can be based on the intent of the classification. As an example and not limitation, for [IN: play_music], the slot can be [SL: song_name].The semantic information aggregator 230 can also extract information from social graphs, knowledge graphs, and concept graphs, and retrieve user profiles from the user context engine 225. The semantic information aggregator 230 can also process information from these different sources by determining what information to aggregate, annotating n-grams of user input, sorting n-grams with confidence scores based on the aggregated information, and formulating the sorted n-grams into features that can be used by the NLU module 220 to understand the user input. More information about aggregated semantic information can be found in U.S. Patent Application No. 15 / 967342 filed on April 30, 2018, which is incorporated by reference. Based on the output of the user context engine 225 and the semantic information aggregator 230, the NLU module 220 can identify domains, intents, and one or more slots from the user input in a personalized and context-aware manner. As an example and not limitation, the user input can include "show me how to get to the Starbucks". The NLU module 220 can identify a specific Starbucks that the user wants to go to based on the user's personal information and associated context information. In a particular embodiment, the NLU module 220 may include a lexicon of language, a parser, and grammatical rules to divide sentences into internal representations. The NLU module 220 may also include one or more programs that use pragmatics to perform naive semantics or random semantic analysis to understand user input. In a particular embodiment, the parser may be based on a deep learning architecture including multiple long short-term memory (LSTM) networks. As an example and not limitation, the parser may be based on a recurrent neural network grammar (RNNG) model, which is a type of recursive and cyclic LSTM algorithm. More information about natural language understanding can be found in U.S. Patent Application No. 16 / 011062 filed on June 18, 2018, U.S. Patent Application No. 16 / 025317 filed on July 2, 2018, and U.S. Patent Application No. 16 / 038120 filed on July 17, 2018, each of which is incorporated by reference.

[0136] In a particular embodiment, the identified domain, intent, and one or more slots from the NLU module 220 may be sent to the conversation engine 235. In a particular embodiment, the conversation engine 235 may manage the conversation flow and conversation state between the user and the assistant xbot 215. The conversation engine 235 may also store previous conversations between the user and the assistant xbot 215. In a particular embodiment, the conversation engine 235 may communicate with the entity resolution module 240 to resolve entities associated with one or more slots, which supports the conversation engine 235 to forward the conversation flow between the user and the assistant xbot 215. In a particular embodiment, the entity resolution module 240 may access social graphs, knowledge graphs, and concept graphs when resolving entities. An entity may include, for example, a unique user or concept, each of which may have a unique identifier (ID). As an example and not limitation, a knowledge graph may include multiple entities. Each entity may include a single record associated with one or more attribute values. A specific record may be associated with a unique entity identifier. For an attribute of an entity, each record may have a different value. Each attribute value may be associated with a confidence probability. The confidence probability of an attribute value indicates the probability that the value is accurate for a given attribute. Each attribute value may also be associated with a semantic weight. The semantic weight of an attribute value may indicate the degree to which the value is semantically suitable for a given attribute, taking into account all available information. For example, a knowledge graph may include an entity of the movie "The Martian" (2015), which includes information that has been extracted from multiple content sources (e.g., Facebook, Wikipedia, movie review sources, media databases, and entertainment content sources), and then deduped, parsed, and fused to generate a single unique record of a knowledge graph. An entity may be associated with a spatial attribute value indicating the genre of the movie "The Martian" (2015). More information about the knowledge graph may be found in U.S. Patent Application No. 16 / 048049 filed on July 27, 2018 and U.S. Patent Application No. 16 / 048101 filed on July 27, 2018, each of which is incorporated by reference. The entity resolution module 240 may additionally request a user profile of a user associated with a user input from the user context engine 225. In certain embodiments, entity resolution module 240 may communicate with privacy check module 245 to ensure that the resolution of the entity does not violate the privacy policy. In certain embodiments, privacy check module 245 may use an authorization / privacy server to enforce the privacy policy. By way of example and not limitation, the entity to be resolved may be another user who has specified in his / her privacy settings that his / her identity should not be searchable on an online social network, and therefore entity resolution module 240 may respond to the request without returning an identifier for that user.Based on information obtained from social graphs, knowledge graphs, concept graphs, and user profiles, and in compliance with applicable privacy policies, the entity resolution module 240 can therefore accurately resolve entities associated with user input in a personalized and context-aware manner. In a particular embodiment, each resolved entity can be associated with one or more identifiers hosted by the social networking system 160. As an example and not limitation, the identifier can include a unique user identifier (ID). In a particular embodiment, each resolved entity can also be associated with a confidence score. More information about resolved entities can be found in U.S. Patent Application No. 16 / 048049 filed on July 27, 2018 and U.S. Patent Application No. 16 / 048072 filed on July 27, 2018, each of which is incorporated by reference.

[0137] In a particular embodiment, the conversation engine 235 can communicate with different agents based on the identified intent and domain and the resolved entity. In a particular embodiment, the agent can be an implementation of a broker between multiple content providers acting as a domain. A content provider can be an entity responsible for performing an action associated with an intent or completing a task associated with an intent. As an example and not limitation, multiple device-specific implementations (e.g., real-time calls to a messaging application on or to a client system 130) can be handled internally by a single agent. Alternatively, these device-specific implementations can be handled by multiple agents associated with multiple domains. In a particular embodiment, the agent may include a first-party agent 250 and a third-party agent 255. In a particular embodiment, the first-party agent 250 may include an internal agent (e.g., an agent associated with a service provided by an online social network (Messenger, Instagram)) that can be accessed and controlled by the assistant system 140. In a particular embodiment, the third-party agent 255 may include an external agent (e.g., a music streaming agent (Spotify), a ticket sales agent (Ticketmaster)) that the assistant system 140 cannot control. The first party agent 250 may be associated with a first party provider 260 that provides content objects and / or services hosted by the social networking system 160. The third party agent 255 may be associated with a third party provider 265 that provides content objects and / or services hosted by the third party system 170.

[0138] In certain embodiments, communications from the conversation engine 235 to the first party agent 250 may include a request for a specific content object and / or service provided by the first party provider 260. Thus, the first party agent 250 may retrieve the requested content object from the first party provider 260 and / or execute a task that instructs the first party provider 260 to perform the requested service. In certain embodiments, communications from the conversation engine 235 to the third party agent 255 may include a request for a specific content object and / or service provided by the third party provider 265. Thus, the third party agent 255 may retrieve the requested content object from the third party provider 265 and / or execute a task that instructs the third party provider 265 to perform the requested service. The third party agent 255 may access the privacy check module 245 to ensure that there is no privacy violation before interacting with the third party provider 265. By way of example and not limitation, a user associated with the user input may specify in his / her privacy settings that his / her profile information is not visible to any third party content provider. Thus, when retrieving a content object associated with a user input from a third party provider 265 , the third party agent 255 may complete the retrieval without revealing to the third party provider 265 which user is requesting the content object.

[0139] In certain embodiments, each of the first party agent 250 or the third party agent 255 may be designated for a specific domain. By way of example and not limitation, the domain may include weather, transportation, music, etc. In certain embodiments, the assistant system 140 may use multiple agents in concert to respond to user input. By way of example and not limitation, the user input may include "direct me to my next meeting". The assistant system 140 may use a calendar agent to retrieve the location of the next meeting. The assistant system 140 may then use a navigation agent to direct the user to the next meeting.

[0140] In certain embodiments, each of the first party agent 250 or the third party agent 255 may retrieve a user profile from the user context engine 225 to perform the task in a personalized and context-aware manner. By way of example and not limitation, the user input may include "book me a ride to the airport". The transportation agent may perform the task of booking a ride. The transportation agent may retrieve the user profile of the user from the user context engine 225 before booking the ride. For example, the user profile may indicate that the user prefers taxis, so the transportation agent may book a taxi for the user. As another example, context information associated with the user profile may indicate that the user is in a hurry, so the transportation agent may book a ride for the user from a ride-sharing service (e.g., Uber, Lyft) because getting a ride from a ride-sharing service may be faster than a taxi company. In certain embodiments, each of the first party agent 250 or the third party agent 255 may consider other factors when performing the task. By way of example and not limitation, the other factors may include price, ratings, efficiency, partnerships with online social networks, etc.

[0141] In certain embodiments, the dialog engine 235 may communicate with a conversation understanding writer (CU writer) 270. The dialog engine 235 may send the requested content object and / or the status of the requested service to the CU writer 270. In certain embodiments, the dialog engine 235 may send the requested content object and / or the status of the requested service to the CU writer 270.<k,c,u,d> Tuples are sent, where k indicates a knowledge source, c indicates a communication target, u indicates a user model, and d indicates a discourse model. In a specific embodiment, the CU writer 270 may include a natural language generator (NLG) 271 and a user interface (UI) payload generator 272. The natural language generator 271 may generate communication content based on the output of the dialogue engine 235. In a specific embodiment, the NLG 271 may include a content determination component, a sentence planner, and a surface realization component. The content determination component may determine the communication content based on the knowledge source, the communication target, and the user's expectations. As an example and not limitation, the determination may be based on description logic. The description logic may include, for example, three basic notions, which are individuals (representing objects in a domain), concepts (describing individual sets), and roles (representing binary relationships between individuals or concepts). The description logic may be characterized by a set of constructors that allow the natural language generator 271 to build complex concepts / roles from atomic concepts / roles. In a specific embodiment, the content determination component may perform the following tasks to determine the communication content. The first task may include a translation task, in which the input to the natural language generator 271 can be translated into concepts. The second task may include a selection task, in which relevant concepts can be selected from the concepts generated by the translation task based on the user model. The third task may include a verification task, in which the consistency of the selected concept can be verified. The fourth task may include an instantiation task, in which the verified concept can be instantiated into an executable file that can be processed by the natural language generator 271. The sentence planner can determine the organization of the communication content so that it is understandable to people. The surface implementation component can determine the specific words to be used, the order of sentences, and the style of the communication content. The UI payload generator 272 can determine the preferred modality of the communication content to be presented to the user. In a specific embodiment, the CU writer 270 can communicate with the privacy check module 245 to ensure that the generation of the communication content follows the privacy policy. In a specific embodiment, when generating the communication content and determining the modality of the communication content, the CU writer 270 can retrieve the user profile from the user context engine 225. Therefore, for the user, the communication content can be more natural, personalized, and context-aware. As an example and not limitation, a user profile may indicate that the user prefers short sentences in conversation, and thus the generated communication content may be based on short sentences.As another example and not limitation, context information associated with the user profile may indicate that the user is using a device that only outputs audio signals, so the UI payload generator 272 may determine the modality of the communication content as audio. More information about natural language generation can be found in U.S. Patent Application No. 15 / 967,279, filed on April 30, 2018, and U.S. Patent Application No. 15 / 966,455, filed on April 30, 2018, each of which is incorporated by reference.

[0142] In certain embodiments, the CU writer 270 may send the generated communication content to the assistant xbot 215. In certain embodiments, the assistant xbot 215 may send the communication content to the messaging platform 205. The messaging platform 205 may also send the communication content to the client system 130 via the assistant application 136. In an alternative embodiment, the assistant xbot 215 may send the communication content to a text-to-speech (TTS) module 275. The TTS module 275 may convert the communication content into an audio clip. The TTS module 275 may also send the audio clip to the client system 130 via the assistant application 136.

[0143] In certain embodiments, assistant xbot 215 may interact with proactive inference layer 280 without receiving user input. Proactive inference layer 280 may infer user interests and preferences based on a user profile retrieved from user context engine 225. In certain embodiments, proactive inference layer 280 may also communicate with proactive agent 285 regarding inferences. Proactive agent 285 may perform proactive tasks based on inferences. As an example and not limitation, proactive tasks may include sending content objects or providing services to users. In certain embodiments, each proactive task may be associated with an agenda item. Agenda items may include recurring items, such as daily summaries. Agenda items may also include one-time items. In certain embodiments, proactive agent 285 may retrieve a user profile from user context engine 225 when performing proactive tasks. Thus, proactive agent 285 may perform proactive tasks in a personalized and context-aware manner. As an example and not limitation, the proactive inference layer may infer that a user likes the band Maroon 5, and proactive agent 285 may generate recommendations for new songs / albums of Maroon 5 for the user.

[0144] In a particular embodiment, the active agent 285 can generate candidate entities associated with the active task based on the user profile. The generation can be based on a direct back-end query that retrieves the candidate entity from a structured data storage using a deterministic filter. Alternatively, the generation can be based on a machine learning model that is trained based on the user profile, entity attributes, and the correlation between the user and the entity. As an example and not limitation, the machine learning model can be based on a support vector machine (SVM). As another example and not limitation, the machine learning model can be based on a regression model. As another example and not limitation, the machine learning model can be based on a deep convolutional neural network (DCNN). In a particular embodiment, the active agent 285 can also sort the generated candidate entities based on the user profile and the content associated with the candidate entity. The sorting can be based on the similarity between the user's interest and the candidate entity. As an example and not limitation, the assistant system 140 can generate a feature vector representing the user's interest and a feature vector representing the candidate entity. The assistant system 140 can then calculate a similarity score (e.g., based on cosine similarity) between the feature vector representing the user's interest and the feature vector representing the candidate entity. Alternatively, the sorting can be based on a sorting model that is trained based on user feedback data.

[0145] In certain embodiments, the active task may include recommending a candidate entity to a user. The active agent 285 may schedule the recommendation, thereby associating the recommendation time with the recommended candidate entity. The recommended candidate entity may also be associated with a priority and an expiration time. In certain embodiments, the recommended candidate entity may be sent to an active scheduler. The active scheduler may determine the actual time to send the recommended candidate entity to the user based on the priority associated with the task and other relevant factors (e.g., clicks and impressions of the recommended candidate entity). In certain embodiments, the active scheduler may then send the recommended candidate entity with the determined actual time to an asynchronous tier. The asynchronous tier may temporarily store the recommended candidate entity as a job. In certain embodiments, the asynchronous tier may send the job to the conversation engine 235 at the determined actual time for execution. In alternative embodiments, the asynchronous tier may execute the job by sending it to other surface layers (e.g., other notification services associated with the social networking system 160). In certain embodiments, the conversation engine 235 may identify the conversation intent, state, and history associated with the user. Based on the conversation intent, the conversation engine 235 may select some candidate entities from the recommended candidate entities to send to the client system 130. In a particular embodiment, the conversation state and history may indicate whether the user is involved in an ongoing conversation with assistant xbot 215. If the user is involved in an ongoing conversation and the priority of the recommended task is low, the conversation engine 235 may communicate with the active scheduler to reschedule the time when the selected candidate entity is sent to the client system 130. If the user is involved in an ongoing conversation and the priority of the recommended task is high, the conversation engine 235 may initiate a new conversation session (dialogsession) with the user, in which the selected candidate entity may be presented. Therefore, the interruption of the ongoing conversation may be prevented. When it is determined that sending the selected candidate entity will not interrupt the user, the conversation engine 235 may send the selected candidate entity to the CU writer 270 to generate personalized and context-aware communication content including the selected candidate entity according to the user's privacy settings. In a particular embodiment, the CU writer 270 may send the communication content to the assistant xbot 215, which may then be sent to the client system 130 via the messaging platform 205 or the TTS module 275. More information about proactively helping users can be found in U.S. Patent Application No. 15 / 967,193, filed on April 30, 2018, and U.S. Patent Application No. 16 / 036,827, filed on July 16, 2018, each of which is incorporated by reference.

[0146] In certain embodiments, assistant xbot 215 can communicate with proactive agent 285 in response to user input. As an example and not limitation, a user can ask assistant xbot 215 to set a reminder. Assistant xbot 215 can request proactive agent 285 to set such a reminder, and proactive agent 285 can proactively perform the task of reminding the user later.

[0147] In a particular embodiment, the assistant system 140 may include a summarizer 290. The summarizer 290 may provide a customized dynamic message summary to the user. In a particular embodiment, the summarizer 290 may include multiple metaagents. Multiple metaagents may use first-party agents 250, third-party agents 255, or active agents 285 to generate dynamic message summaries. In a particular embodiment, the summarizer 290 may retrieve user interests and preferences from the active inference layer 280. The summarizer 290 may then retrieve entities associated with user interests and preferences from the entity resolution module 240. The summarizer 290 may also retrieve a user profile from the user context engine 225. Based on information from the active inference layer 280, the entity resolution module 240, and the user context engine 225, the summarizer 290 may generate a personalized and context-aware summary for the user. In a particular embodiment, the summarizer 290 may send the summary to the CU writer 270. The CU writer 270 may process the summary and send the processing result to the assistant xbot 215. Assistant xbot 215 may then send the processed summary to client system 130 via messaging platform 205 or TTS module 275. More information about summarization can be found in U.S. Patent Application No. 15 / 967,290, filed April 30, 2018, which is incorporated by reference.

[0148] Figure 3An example flow chart of the assistant system 140 responding to a user request is shown. In a particular embodiment, the assistant xbot 215 can access the request manager 305 when receiving a user request. The request manager 305 can include a context extractor 306 and a conversation understanding object generator (CU object generator) 307. The context extractor 306 can extract context information associated with the user request. The context extractor 306 can also update the context information based on the assistant application 136 executed on the client system 130. As an example and not limitation, the update of the context information can include displaying a content item on the client system 130. As another example and not limitation, the update of the context information can include whether to set an alarm on the client system 130. As another example and not limitation, the update of the context information can include whether to play a song on the client system 130. The CU object generator 307 can generate a specific content object related to the user request. The content object can include conversation session data and features associated with the user request, which can be shared with all modules of the assistant system 140. In particular embodiments, request manager 305 may store context information and generated content objects in data store 310 , which is a particular data store implemented in assistant system 140 .

[0149] In a particular embodiment, the request manager 305 may send the generated content object to the NLU module 220. The NLU module 220 may perform a number of steps to process the content object. In step 221, the NLU module 220 may generate a whitelist of content objects. In a particular embodiment, the whitelist may include interpretation data that matches the user request. In step 222, the NLU module 220 may perform characterization based on the whitelist. In step 223, the NLU module 220 may perform domain classification / selection on the user request based on the features generated by the characterization to classify the user request into a predefined domain. The domain classification / selection results may also be further processed based on two related processes. In step 224a, the NLU module 220 may use an intent classifier to process the domain classification / selection results. The intent classifier may determine the user intent associated with the user request. In a particular embodiment, each domain may have an intent classifier to determine the most likely intent in a given domain. By way of example and not limitation, the intent classifier may be based on a machine learning model that may take the domain classification / selection results as input and calculate the probability that the input is associated with a particular predefined intent. In step 224b, the NLU module may use a meta-intent classifier to process the domain classification / selection results. The meta-intent classifier may determine a category that describes the user's intent. In a particular embodiment, intents common to multiple domains may be processed by the meta-intent classifier. As an example and not limitation, the meta-intent classifier may be based on a machine learning model that may take the domain classification / selection results as input and calculate the probability that the input is associated with a specific predefined meta-intent. In step 225a, the NLU module 220 may use a slot tagger to annotate one or more slots associated with the user request. In a particular embodiment, the slot tagger may annotate one or more slots for n-grams of the user request. In step 225b, the NLU module 220 may use a meta-slot tagger to annotate one or more slots for the classification results from the meta-intent classifier. In a particular embodiment, the meta-slot tagger may tag a common slot (such as a reference to an item (e.g., the first one)), the type of the slot, the value of the slot, etc. By way of example and not limitation, a user request may include "change 500 dollars in my account to Japanese yen". The intent classifier may take as input the user request and formulate it into a vector. The intent classifier may then calculate the probability that the user request is associated with different predefined intents based on a vector comparison between the vector representing the user request and the vectors representing different predefined intents. In a similar manner, the slot tagger may take as input the user request and formulate each word into a vector.The intent classifier can then calculate the probability of each word being associated with different predefined slots based on a vector comparison between the vector representing the word and the vector representing the different predefined slots. The user's intent can be classified as "changing money". The slots requested by the user can include "500", "dollars", "account", and "Japanese yen". The user's meta-intent can be classified as "financial service". The meta-slot can include "finance".

[0150] In certain embodiments, the NLU module 220 may improve the domain classification / selection of content objects by extracting semantic information from the semantic information aggregator 230. In certain embodiments, the semantic information aggregator 230 may aggregate semantic information in the following manner. The semantic information aggregator 230 may first retrieve information from the user context engine 225. In certain embodiments, the user context engine 225 may include an offline aggregator 226 and an online inference service 227. The offline aggregator 226 may process a plurality of data associated with the user collected from a previous time window. As an example and not limitation, the data may include dynamic message posts / comments, interactions with dynamic message posts / comments, Instagram posts / comments, search history, etc. collected from a previous 90-day window. The processing results may be stored in the user context engine 225 as part of the user profile. The online inference service 227 may analyze the session data associated with the user received by the assistant system 140 at the current time. The analysis results may also be stored in the user context engine 225 as part of the user profile. In certain embodiments, both the offline aggregator 226 and the online inference service 227 may extract personalized features from the plurality of data. The extracted personalized features can be used by other modules of the assistant system 140 to better understand the user input. In a particular embodiment, the semantic information aggregator 230 can then process the information retrieved from the user context engine 225, i.e., the user profile, in the following steps. In step 231, the semantic information aggregator 230 can process the information retrieved from the user context engine 225 based on natural language processing (NLP). In a particular embodiment, the semantic information aggregator 230 can: tokenize the text by text normalization, extract syntax features from the text, and extract semantic features from the text based on NLP. The semantic information aggregator 230 can also extract features from context information, which is accessed from the conversation history between the user and the assistant system 140. The semantic information aggregator 230 can also perform global word embedding, domain-specific embedding, and / or dynamic embedding based on the context information. In step 232, the processing results can be annotated with entities by the entity tagger. In step 233, based on the annotations, the semantic information aggregator 230 can generate a dictionary for the retrieved information. In certain embodiments, the dictionary may include a global dictionary feature that may be dynamically updated offline. At step 234, the semantic information aggregator 230 may rank the entities tagged by the entity tagger. In certain embodiments, the semantic information aggregator 230 may communicate with different graphs 330 including social graphs, knowledge graphs, and concept graphs to extract ontology data related to the information retrieved from the user context engine 225. In certain embodiments, the semantic information aggregator 230 may aggregate the user profile, the ranked entities, and the information from the graphs 330.The semantic information aggregator 230 may then send the aggregated information to the NLU module 220 to facilitate domain classification / selection.

[0151] In certain embodiments, the output of the NLU module 220 may be sent to a co-reference module 315 to interpret the references of the content object associated with the user request. In certain embodiments, the co-reference module 315 may be used to identify the item to which the user request refers. The co-reference module 315 may include reference creation 316 and reference resolution 317. In certain embodiments, the reference creation 316 may create references for the entities determined by the NLU module 220. The reference resolution 317 may accurately resolve these references. As an example and not limitation, the user request may include "find me the nearest Walmart and direct me there". The co-reference module 315 may interpret "there" as "the nearest Walmart". In certain embodiments, the co-reference module 315 may access the user context engine 225 and the dialog engine 235, if necessary, to interpret the references with improved accuracy.

[0152] In a particular embodiment, the identified domains, intents, meta-intents, slots and meta-slots, and resolved references may be sent to entity resolution module 240 to resolve related entities. Entity resolution module 240 may perform general and domain-specific entity resolution. In a particular embodiment, entity resolution module 240 may include domain entity resolution 241 and general entity resolution 242. Domain entity resolution 241 may resolve entities by categorizing slots and meta-slots into different domains. In a particular embodiment, entities may be resolved based on ontology data extracted from graph 330. Ontology data may include structural relationships between different slots / meta-slots and domains. Ontology may also include information on how slots / meta-slots may be grouped, related, and subdivided according to similarities and differences within a hierarchy that includes domains at a higher level. General entity resolution 242 may resolve entities by categorizing slots and meta-slots into different general topics. In a particular embodiment, resolution may also be based on ontology data extracted from graph 330. Ontology data may include structural relationships between different slots / meta-slots and general topics. The ontology may also include information about how slots / meta-slots may be grouped, related, and subdivided according to similarities and differences within a hierarchy that includes topics at a higher level. As an example and not limitation, in response to an input of a query about the advantages of a Tesla car, the general entity resolution 242 may resolve a Tesla car to a vehicle, and the domain entity resolution 241 may resolve a Tesla car to an electric car.

[0153] In a particular embodiment, the output of the entity resolution module 240 may be sent to the dialogue engine 235 to forward the conversation flow with the user. The dialogue engine 235 may include a dialogue intent analysis 236 and a dialogue state updater / sorter 237. In a particular embodiment, the dialogue intent analysis 236 may analyze the user intent associated with the current dialogue session based on the dialogue history between the user and the assistant system 140. The dialogue intent analysis 236 may map the intent determined by the NLU module 220 to different dialogue intents. The dialogue intent analysis 236 may also sort the dialogue intent based on the signals from the NLU module 220, the entity resolution module 240, and the dialogue history between the user and the assistant system 140. In a particular embodiment, the dialogue state updater / sorter 237 may update / sort the dialogue state of the current dialogue session. As an example and not limitation, if the dialogue session ends, the dialogue state updater / sorter 237 may update the dialogue state to "completed". As another example and not limitation, the dialogue state updater / sorter 237 may sort the dialogue state based on the priority associated with the dialogue state.

[0154] In certain embodiments, the dialog engine 235 may communicate with the task completion module 335 regarding the dialog intent and the associated content object. In certain embodiments, the task completion module 335 may rank different dialog hypotheses for different dialog intents. The task completion module 335 may include an action selection component 336. In certain embodiments, the dialog engine 235 may additionally check against the dialog strategy 320 regarding the dialog state. In certain embodiments, the dialog strategy 320 may include a data structure describing the action execution plan of the agent 340. The agent 340 may select from the registered content providers to complete the action. The data structure may be constructed by the dialog engine 235 based on the intent and one or more slots associated with the intent. The dialog strategy 320 may also include multiple goals that are interrelated by logical operators. In certain embodiments, the goal may be the output result of a part of the dialog strategy, and it may be constructed by the dialog engine 235. The goal may be represented by an identifier (e.g., a string) having one or more named parameters, which parameterize the goal. As an example and not limitation, a goal and its associated goal parameter may be represented as {confirm_artist, parameter: {artist: "Madonna"}}. In a particular embodiment, a dialog strategy may be represented based on a tree structure, in which a goal is mapped to a leaf. In a particular embodiment, a dialog engine 235 may execute a dialog strategy 320 to determine the next action to be performed. The dialog strategy 320 may include a general strategy 321 and a domain-specific strategy 322, both of which may guide how to select the next system action based on the dialog state. In a particular embodiment, a task completion module 335 may communicate with the dialog strategy 320 to obtain guidance for the next system action. In a particular embodiment, an action selection component 336 may therefore select an action based on the dialog intent, the associated content object, and guidance from the dialog strategy 320.

[0155] In a particular embodiment, the output of the task completion module 335 may be sent to the CU writer 270. In an alternative embodiment, the selected action may require one or more agents 340 to participate. Therefore, the task completion module 335 may notify the agent 340 of the selected action. At the same time, the conversation engine 235 may receive an instruction to update the conversation state. As an example and not limitation, the update may include waiting for the response of the agent. In a particular embodiment, the CU writer 270 may generate communication content for the user using NLG 271 based on the output of the task completion module 335. In a particular embodiment, NLG 271 may use different language models and / or language templates to generate natural language output. The generation of natural language output may be application-specific. The generation of natural language output may also be personalized for each user. The CU writer 270 may also use the UI payload generator 272 to determine the modality of the generated communication content. Since the generated communication content may be considered as a response to a user request, the CU writer 270 may additionally use a response sorter 273 to sort the generated communication content. As an example and not limitation, the sorting may indicate the priority of the response.

[0156] In a particular embodiment, the output of the CU writer 270 may be sent to a response manager 325. The response manager 325 may perform different tasks, including storing / updating the dialog state 326 retrieved from the data storage 310 and generating a response 327. In a particular embodiment, the output of the CU writer 270 may include one or more of a natural language string, a voice, or an action with parameters. Therefore, the response manager 325 may determine what task to perform based on the output of the CU writer 270. In a particular embodiment, the generated response and communication content may be sent to the assistant xbot 215. In an alternative embodiment, if the modality of the determined communication content is audio, the output of the CU writer 270 may be sent to the TTS module 275 in addition. Then, the voice generated by the TTS module 275 and the response generated by the response manager 325 may be sent to the assistant xbot 215.

[0157] Gesture input workflow

[0158] Figure 4An example workflow for processing user input including gesture input is shown. Gesture input can be input based on the movement of a body part (particularly a hand or head) to express an idea or meaning. Gesture input can be in the form of image information, video information, motion information, or any combination thereof. In a particular embodiment, the assistant system 140 can receive user input 405 of a user from the client system 130. The assistant system 140 can send the user input 405 to the intent understanding module 410. The intent understanding module 410 can include a natural language understanding (NLU) module 220 and a gesture classification model 415. The gesture classification model 415 can be a machine learning model that is trained offline to recognize different categories of gestures performed by a user. As an example and not limitation, the gesture classification model 415 can be based on one or more of a convolutional neural network, a tensor flow, or a hidden Markov model. Based on the user input 405, the intent understanding module 410 can use different components to determine the intent 420 associated with the user input 405. If the user input 405 includes text input or voice input, the intent understanding module 410 can use the NLU module 220 to determine the intent 420. If the user input 405 includes a gesture input, the intent understanding module 410 can use the gesture classification model 415 to determine the intent 420. The determined intent 420 can be associated with a confidence score that indicates how confident the assistant system 140 is in determining the user's intent 420. In a particular embodiment, the confidence score can be determined based on the matching proximity of the user's text input or voice input to the known input for the given intent 420. As an example and not limitation, the proximity can be based on the string similarity between the text input and the known input (text). The assistant system 140 can then send the intent 420 and its confidence score to the dialogue engine 235. If the confidence score is higher than the threshold score, the dialogue engine 235 can determine one or more tasks 425 corresponding to the intent 420 and send them to the first party agent 250 or the third party agent 255 to perform the task 425. The execution result 430 can be sent to the CU writer 270. If the confidence score is lower than the threshold score, the dialog engine 235 may determine one or more suggested inputs 435 for the user, which may help the assistant system 140 determine the user's intent with a higher confidence score. The dialog engine 235 may send the suggested inputs 435 to the CU writer 270. Based on the execution result 430 or the suggested inputs 435, the CU writer 270 may generate a response 440. The response 440 may be in different modalities, including one or more of text, images, videos, or gesture animations. The CU writer 270 may also send the response 440 to the client system 130. Although the present disclosure describes processing user input via a particular system in a particular manner, the present disclosure contemplates describing processing any suitable user input via any suitable system in any suitable manner.

[0159] Auto-completion of multimodal user input in assistant systems

[0160] In a particular embodiment, the assistant system 140 may receive an initial input of a first modality performed by a user (e.g., a gesture input to a virtual reality (VR) head mounted device or augmented reality (AR) smart glasses), which is an interaction with the assistant-based client system 130, determine candidate continuation inputs in an auto-complete manner based on the initial input, and recommend these candidate continuation inputs to the user in one or more second modalities (e.g., text), for which the user can select one of them to trigger the execution of a specific task 425. The initial input and the continuation input may be based on any suitable modality, including text, voice, image, video, motion, orientation, etc. In addition, the modality of the initial input and the modality of the continuation input may be different. As an example and not limitation, a user wearing AR glasses to visit Paris may be looking at the Eiffel Tower. Based on the gaze input, the assistant system 140 may suggest a gesture for taking a photo displayed on a virtual screen, or suggest that the user say "take a photo". The user may perform the suggested gesture or say a sentence to allow the assistant system 140 to perform the task of taking a photo of the Eiffel Tower. Thus, assistant system 140 may have the ability to process user input 405 in one modality and generate suggested inputs in one or more other modalities, which may be referred to as auto-completion for multimodal user input. Although this disclosure describes generating particular auto-completion for particular multimodal user input in a particular manner via a particular system, this disclosure contemplates any suitable auto-completion for generating any suitable multimodal user input in any suitable manner via any suitable system.

[0161] In a particular embodiment, assistant system 140 may receive an initial input from a first user from a client system 130 associated with the first user. The initial input may be in a first modality. In a particular embodiment, assistant system 140 may determine one or more intents 420 corresponding to the initial input through an intent understanding module. Assistant system 140 may then generate one or more candidate continuation inputs based on the one or more intents 420. The one or more candidate continuation inputs may be in one or more candidate modalities, respectively. The candidate modality may be different from the first modality. In a particular embodiment, assistant system 140 may also send an instruction to client system 130 for presenting one or more suggested inputs corresponding to the one or more candidate continuation inputs.

[0162] In a particular embodiment, the first modality may include one of audio, text, image, video, motion, or orientation. As an example and not limitation, the first modality may include motion, and accordingly, the initial input may include a gesture. As another example and not limitation, the first modality may include an orientation, and accordingly, the initial input may include a gaze at an object. In a particular embodiment, the candidate modalities of the first modality and the candidate continuation input may be different. As an example and not limitation, the user may be wearing AR glasses, and his / her initial input may be a framing gesture of two hands for locating an object in front of him / her. The assistant system 140 may suggest that the user say "click" as a candidate continuation input, which is displayed as text on the screen of the AR glasses. Once the user says "click", the assistant system 140 may further perform the task of taking a photo of the object in the frame. Therefore, the assistant system 140 may have a technical advantage of enriching the user experience of the assistant system 140 because the user can interact with the assistant system 140 with inputs of various modalities. Although the present disclosure describes a particular modality in a particular manner, the present disclosure contemplates any suitable modality in any suitable manner.

[0163] In a particular embodiment, the assistant system 140 may determine candidate continuation inputs based on the intent of the initial input and potential entities associated with the intent 420. The assistant system 140 may first identify one or more entities associated with one or more intents 420. The assistant system 140 may then also generate one or more candidate continuation inputs based on the one or more entities. As an example and not limitation, the first user may perform a call gesture as an initial input. The assistant system 140 may determine a call intent and possible entities (e.g., contacts) associated with the call intent. The entity may be suggested to the user as text, and the assistant system 140 may additionally ask the user to say the name of the entity that the user intends to call. Determining candidate continuation inputs based on the intent 420 of the initial input and potential entities associated with the intent 420 may be an effective solution to the technical challenge of determining candidate continuation inputs based on the initial input, because the intent 420 and potential entities may be necessary to determine what continuation inputs the user can use to interact with the assistant system 140 to perform tasks based on them. Although the present disclosure describes determining a particular input in a particular manner, the present disclosure contemplates determining any suitable input in any suitable manner.

[0164] In certain embodiments, assistant system 140 may determine that the first user needs one or more suggested inputs. Assistant system 140 may determine whether to suggest candidate continuation inputs based on different factors. In certain embodiments, determining that the first user needs one or more suggested inputs may be based on a wake-up input from the first user. The wake-up input may include one or more of a voice utterance, a string, an image, a video clip, a gesture, or a gaze. As an example and not limitation, the wake-up input may be a voice command saying "assistant!" or a gesture like snapping fingers. Once assistant system 140 receives the wake-up input, it may start a process for suggesting candidate continuation inputs. Assistant system 140 may be continuously active for suggesting candidate continuation inputs. In addition, whether to suggest a continuation input may be based on user attention detected by assistant system 140, i.e., gaze information. In certain embodiments, the initial input may include a gaze on an object, and determining that the first user needs one or more suggested inputs may also be based on a gaze on an object. On the other hand, generating one or more candidate continuation inputs may also be based on an object. Determining candidate continuation inputs based on objects associated with the initial input can be an effective solution to the technical challenge of determining candidate continuation inputs based on the initial input, because the object may be necessary to determine what continuation input the user can use to interact with the assistant system to perform a task based on it. As an example and not limitation, if a user wearing AR glasses is staring at a pair of shoes in a store, the assistant system 140 can determine that the user needs a suggestion input and generate a suggestion input, such as "take a photo" and "check the reviews", etc. The suggestion input can be based on any suitable modality. For example, the suggestion input can be presented as a gesture displayed in the screen of the AR glasses, and the user can follow the gesture to perform. As another example, the suggestion input can be audio-based, and the user can listen to the audio-based suggestion input. As another example and not limitation, the user makes a video call on the AR glasses, and then the user turns around and watches the TV for a threshold time. The assistant system 140 can then determine that the user needs a suggestion input. If the user wants the video call to be mirrored to the TV, the assistant system 140 can present a suggestion input indicating that the user is pointing to the TV. In a particular embodiment, determining that the first user needs one or more suggestion inputs can also be based on contextual information associated with the initial input. In certain embodiments, the context information may include one or more of time information, location information, presence information, or social information. As an example and not limitation, the user may enter only a few letters and then stop entering text for a certain period of time. Assistant system 140 may then determine that the user needs suggested input because the user may have difficulty entering the desired text.As another example and not limitation, a user may be at a cafeteria and sends an image of a dish to assistant system 140. Assistant system 140 may then determine that the user needs a suggestion input, such as the name of the dish or a request to order the dish. In particular embodiments, determining that the first user needs one or more suggestion inputs may also be based on one or more intents 420. As an example and not limitation, the user's initial input may indicate that the user wants to buy movie tickets, but the user did not provide any additional information. Assistant system 140 may then determine that the user needs a suggestion input, such as the name of a recently popular movie. Although the present disclosure describes determining whether to suggest a particular input in a particular manner, the present disclosure contemplates determining whether to suggest any suitable input in any suitable manner.

[0165] In certain embodiments, assistant system 140 may also receive user-selected input from a first user from client system 130. The user-selected input may include one of the suggested inputs. Assistant system 140 may then perform one or more tasks 425 based on the user-selected input. In certain embodiments, assistant system 140 may suggest candidate continuation inputs more than once. In other words, assistant system 140 may guide the user through a sequence of candidate continuation inputs, each of which may be mapped to a specific intent 420. More specifically, assistant system 140 may receive a first user-selected input from a first user from client system 130. The first user-selected input may include one of the suggested inputs, and the first user-selected input may be associated with first intent 420. Assistant system 140 may then generate one or more additional candidate continuation inputs based on the first user-selected input. Each of the one or more additional candidate continuation inputs may be associated with first intent 420. Assistant system 140 may then send instructions to client system 130 for presenting one or more additional suggested inputs corresponding to one or more additional candidate continuation inputs. The assistant system 140 may then receive a second user-selected input from the first user from the client system 130. The second user-selected input may include one of the additional suggested inputs. The assistant system 140 may also perform one or more tasks 425 based on the second user-selected input. One or more tasks 425 may correspond to the first intention 420. In a particular embodiment, the assistant system 140 may continue the foregoing process until one or more tasks 425 that require the entire input sequence are executed. As an example and not a limitation, a user wearing AR glasses may want to set a reminder by performing a "reminder" gesture. The assistant system 140 may recognize such a gesture and generate a suggestion for what event the reminder is for. The assistant system 140 may present three suggested events in text, for example, "mowing the lawn", "feeding the dog", and "buying a gift for mom". The user may click "feeding the dog". The assistant system 140 may also generate suggested times (e.g., 10 a.m., 3 p.m., and 7 p.m.) and ask the user to select the time by tapping or speaking. After the user selects the time (e.g., by saying "7 p.m."), the assistant system 140 may set the reminder accordingly. Thus, assistant system 140 may have a technical advantage of increasing the degree of user engagement with assistant system 140 by guiding the user with suggested input sequences to explore various tasks for which the user may request assistance. Although this disclosure describes generating a particular input sequence in a particular manner, this disclosure contemplates generating any suitable input sequence in any suitable manner.

[0166] Auto-completion of gesture input in the Assistant system

[0167] In a particular embodiment, the assistant system 140 may receive an incomplete gesture performed by a user as an interaction with an assistant-based client system 130 (e.g., a VR head-mounted device or AR smart glasses), determine candidate gestures in an automatic completion manner based on the incomplete gesture, recommend these candidate gestures to the user, and for these candidate gestures, the user can select one of them to trigger the execution of a specific task 425. As an example and not a limitation, a user wearing AR glasses may not know what gesture to perform to trigger a specific function of the assistant system 140. The user may start to move his / her hand, but pause in the air. Therefore, the assistant system 140 can analyze the user's incomplete gesture and determine possible candidate gestures. The assistant system 140 can visually display these candidate gestures to the user through the AR glasses. After the user selects one of the candidate gestures, the assistant system 140 can also perform the task 425 corresponding to the gesture. In summary, the assistant system 140 can use gesture recognition technology for incomplete gestures to determine the gesture intended by the user, and suggest the determined gesture to the user as a guide, which can be referred to as automatic completion of gesture input. Although this disclosure describes generating a particular auto-completion for a particular gesture input in a particular manner via a particular system, this disclosure contemplates generating any suitable auto-completion for any suitable gesture input in any suitable manner via any suitable system.

[0168] In a particular embodiment, the assistant system 140 may receive user input 405 from the first user from a client system associated with the first user. The user input 405 may include an incomplete gesture performed by the first user. In a particular embodiment, the assistant system 140 may calculate one or more confidence scores corresponding to one or more intents 420 of the incomplete gesture through the intent understanding module. The assistant system 140 may then determine that the calculated confidence score associated with each intent 420 is lower than a threshold score. In a particular embodiment, in response to determining that the calculated confidence score of each intent 420 is lower than a threshold score, the assistant system 140 may select one or more candidate gestures from a plurality of predefined gestures based on a personalized gesture recognition model. Each candidate gesture may be associated with a confidence score representing the possibility that the first user intends to input the corresponding candidate gesture. In a particular embodiment, the assistant system 140 may also send an instruction to the client system 130 for presenting one or more suggested inputs corresponding to one or more candidate gestures.

[0169] In certain embodiments, assistant system 140 may identify an incomplete gesture of user input 405. An incomplete gesture may include the beginning of a gesture, which may indicate that the user has not yet completed the entire gesture. As an example and not limitation, an open hand suspended in the air may be an incomplete gesture of a waving gesture. In certain embodiments, assistant system 140 may suggest a list of predefined gestures, one of which may be the gesture that the user intends to perform. As an example and not limitation, each predefined gesture may include one or more of pointing, poking, tapping, waving, or sliding. Therefore, assistant system 140 may have a technical advantage of improving the user's experience of assistant system 140, because assistant system 140 can understand the user's intent 420 based on the incomplete gesture, and teach the user to complete the incomplete gesture to interact with assistant system 140. Although the present disclosure describes specific gestures in a specific manner, the present disclosure contemplates any suitable gesture in any suitable manner.

[0170] In a particular embodiment, the assistant system 140 may determine the gesture intended by the user based on a similarity evaluation relative to a list of predefined gestures. The assistant system 140 may calculate the similarity of the candidate gesture relative to the incomplete gesture for each of one or more candidate gestures. As an example and not limitation, if the user's hand is in a particular position, the assistant system 140 may determine how similar the incomplete gesture is to the predefined gesture. After determining which predefined gestures the unfinished gesture is most similar to, the assistant system 140 may select them as candidate gestures. In a particular embodiment, the assistant system 140 may determine whether to present a candidate gesture to the user based on the similarity. As an example and not limitation, if the similarity of all candidate gestures relative to the incomplete gesture is below a predefined threshold level, the assistant system 140 may present them to the user for selection. As another example and not limitation, if two or more candidate gestures have a similar similarity greater than a predefined threshold level, the assistant system 140 may still present them to the user for ambiguity elimination. In a particular embodiment, selecting one or more candidate gestures may also be based on one or more intentions 420. As an example and not limitation, the intention 420 may include locating an object on social media or interacting with another user. Accordingly, the candidate gesture may include pointing or poking. In certain embodiments, assistant system 140 may also suggest inputs that may guide the user to complete the candidate gesture. Although this disclosure describes determining a particular candidate gesture in a particular manner, this disclosure contemplates determining any suitable candidate gesture in any suitable manner.

[0171] In certain embodiments, assistant system 140 may determine similarity in different ways. In certain embodiments, the similarity of each candidate gesture relative to an incomplete gesture may be based on the trajectory of the incomplete gesture relative to the client system. As an example and not limitation, if the user moves his / her hand from left to right and pauses, it is likely that assistant system 140 can determine some gestures (e.g., sliding) that continue the motion with a relatively similar trajectory. But if the user moves his / her hand from the bottom to the top, the trajectory is different, for which assistant system 140 may determine a different gesture (e.g., pointing). In certain embodiments, the similarity of each candidate gesture relative to an incomplete gesture may be based on the orientation of the incomplete gesture relative to client system 130. As an example and not limitation, if the orientation is parallel to client system 130, a waving or sliding gesture may have a higher similarity relative to an incomplete gesture. As another example and not limitation, if the orientation is perpendicular to client system 130, a pointing or poking gesture may have a higher similarity relative to an incomplete gesture. In certain embodiments, the similarity of each candidate gesture relative to an incomplete gesture may be based on an object associated with the incomplete gesture. As an example and not limitation, the gesture determined to correspond to a phone may be different from the gesture corresponding to a television. In a particular embodiment, the similarity of each candidate gesture relative to the incomplete gesture may be based on contextual information associated with the incomplete gesture. As an example and not limitation, if the contextual information indicates that the user is browsing another user's social media profile, the candidate gesture may be more likely to be a poke rather than a point. In a particular embodiment, the similarity of each candidate gesture relative to the incomplete gesture may be based on the location of the incomplete gesture relative to the client system 130. As an example and not limitation, if the incomplete gesture is very close to the client system 130, the candidate gesture may be more likely to be a tap. A technical challenge includes determining the similarity of the candidate gesture relative to the incomplete gesture. Determining similarity based on different factors including the trajectory, orientation, object, contextual information, and location associated with the incomplete gesture can be an effective solution to the technical challenge of determining the similarity of the candidate gesture relative to the incomplete gesture, because such information can provide different information cues for the assistant system 140 to calculate the similarity. Although the present disclosure describes determining a specific similarity in a particular manner, the present disclosure contemplates determining any suitable similarity in any suitable manner.

[0172] In a particular embodiment, the assistant system 140 may determine whether the first user needs advice to complete the incomplete gesture based on the calculated confidence score of the intent 420 corresponding to the incomplete gesture. If the confidence score is lower than the threshold score, it may indicate that the assistant system 140 cannot determine the user's intent 420. This difficulty in determining the user's intent 420 may be caused by the fact that the incomplete gesture information is insufficient, which also indicates that the user does not know how to perform the desired gesture and may need advice. In a particular embodiment, the threshold score may be based on a wake-up gesture performed by the first user. The wake-up gesture may indicate that the user obviously needs advice. Therefore, the assistant system 140 may set a high threshold score, which causes the calculated confidence score to be lower than the threshold score. The assistant system 140 can then generate candidate gestures accordingly. In short, once the user makes a wake-up gesture, the assistant system 140 can start the process of gesture suggestions. The assistant system can continue to be active to make gesture suggestions. In a particular embodiment, calculating one or more confidence scores corresponding to one or more intents 420 of an incomplete gesture may be based on time information associated with the incomplete gesture. The time information may include a pause in the user input 405. As an example and not limitation, when there is a pause in the user input 405, the calculated confidence score may be low. Therefore, the assistant system 140 can determine that the user needs advice to complete the incomplete gesture. In a specific embodiment, one or more confidence scores corresponding to one or more intentions 420 of the incomplete gesture can be calculated based on the speed associated with the incomplete gesture. As an example and not limitation, the assistant system 140 can monitor how fast the user moves. If the user is quite confident and moves quickly, the user may already know the gesture he / she wants to perform, for which the assistant system 140 can calculate a high confidence score and do not suggest any gesture. If the assistant system 140 detects that the user is hesitant to perform a gesture (e.g., moving slowly), the assistant system 140 can infer that the user is trying to figure out what gesture he / she can do, calculate a low confidence score, and suggest gestures accordingly. In a specific embodiment, determining that the first user needs advice to complete the incomplete gesture can also be based on the pattern of the incomplete gesture. As an example and not limitation, if the user is repeating the same incomplete gesture, the assistant system 140 can determine that the gesture is suggested to complete the gesture of the user's intention. Determining whether to suggest a candidate gesture based on different factors (including the confidence score of the intent, the wake-up gesture, the time information of the incomplete gesture, the speed of the incomplete gesture, and the pattern of the incomplete gesture) can be an effective solution to the technical challenge of determining that the user needs a suggested input for a gesture, because such information can reveal the user's need for help from different angles. Although the present disclosure describes determining whether a user needs a particular suggested gesture in a particular manner, the present disclosure contemplates determining whether a user needs any suitable suggested gesture in any suitable manner.

[0173] In a particular embodiment, assistant system 140 may receive user-selected input from a first user from client system 130. The user-selected input may include one of the suggested inputs. Assistant system 140 may then perform one or more tasks based on the user-selected input. In a particular embodiment, assistant system 140 may suggest candidate gestures more than once. In other words, assistant system 140 may guide the user through a sequence of candidate gestures, each of which may be mapped to a specific intent 420. More specifically, assistant system 140 may receive a first user-selected input from a first user from client system 130. The first user-selected input may include one of the suggested inputs, and the first user-selected input may be associated with first intent 420. Assistant system 140 may then generate one or more additional candidate gestures based on the first user-selected input. Each of the one or more additional candidate gestures may be associated with first intent 420. Assistant system 140 may then send instructions to client system 130 for presenting one or more additional suggested inputs corresponding to one or more additional candidate gestures. Assistant system 140 may then receive a second user-selected input from the first user from client system 130. The input selected by the second user may include one of the additional suggested inputs. The assistant system 140 may also perform one or more tasks 425 based on the input selected by the second user. One or more tasks 425 may correspond to the first intention 420. In a particular embodiment, the assistant system 140 may continue the foregoing process until one or more tasks 425 requiring the entire sequence of gestures are performed. As an example and not limitation, a user wearing AR glasses may want to make a call and has performed an incomplete gesture. The assistant system 140 may suggest some candidate gestures, including a "call" gesture. The user can then complete the "call" gesture based on the suggestion. The assistant system 140 may then display three different people (e.g., names or photos) on the virtual screen of the VR glasses and suggest additional gestures (e.g., pointing or tapping) to the user. Accordingly, the user may perform one of the additional suggested gestures on the person he / she wants to call. Finally, the assistant system 140 may perform the task 425 of calling the intended person. As another example and not limitation, the user may want to take a picture of a friend and has performed an incomplete gesture. The assistant system 140 may suggest several candidate gestures to the user, including a "photograph" gesture. The user can then complete the "take a photo" gesture based on the suggestion. After the photo is taken, the assistant system 140 can also suggest some additional gestures that the user can perform on the photo. For example, one gesture can be a gesture to send the photo to a friend. The user can then perform such a gesture to complete the gesture sequence, and the assistant system 140 can then send the photo to the friend.Thus, assistant system 140 may have a technical advantage of increasing the level of user engagement with assistant system 140 by guiding the user with suggested gesture sequences to explore various tasks that the user may request assistance with. Although this disclosure describes suggesting particular gesture sequences in particular ways, this disclosure contemplates suggesting any suitable gesture sequences in any suitable way.

[0174] In a particular embodiment, determining candidate gestures in response to an incomplete gesture of a user can be applied to user education in an assistant-based AR / VR system. As an example and not a limitation, when a user initially puts on AR glasses, he may not know what gestures he can use to interact with the assistant system 140. The user may raise his / her hand and wonder what to do. In this case, the assistant system 140 can analyze the user's current incomplete gesture and determine which gestures may be similar gestures that the user can perform. Therefore, the assistant system 140 can suggest the determined gestures by virtually displaying them on the screen of the AR glasses. In a particular embodiment, when only gestures can be used to interact with the assistant system 140, it may be appropriate to determine candidate gestures in response to an incomplete gesture of the user. As an example and not a limitation, the user cannot speak and does not have a keyboard for typing. In this case, if the user starts his / her interaction with the assistant system 140 with an incomplete gesture, the assistant system 140 may suggest that the user perform one or more gestures to complete a specific task 425. Although the present disclosure describes specific applications of specific methods of the embodiments disclosed herein, the present disclosure contemplates any suitable application of any suitable method of the embodiments disclosed herein.

[0175] Figure 5 An example scenario of automatic completion of multimodal user input 405 is shown. Figure 5 As shown, the user 500 may be wearing a pair of AR glasses 505. The AR glasses 505 can be used as a client system 130 for the user 500 to interact with the assistant system 140. The user 500 can provide an initial input, which can be a "call" gesture 510. The assistant system 140 can determine the user's intent 420 to call the contact corresponding to the "call" gesture 510. The assistant system 140 can identify two contacts that the user 500 may want to call, such as his mother and his wife Jane. Based on the identified contacts, the assistant system 140 can generate two candidate continuation inputs, including selecting a contact by voice and selecting a contact by gesture. The assistant system 140 can also generate suggested inputs that can be displayed on the screen 515 of the VR glasses 505. In Figure 5, the suggested input is presented as “Who do you want to contact? You can say “Mom” or “Jane”, or you can tap on a photo” 520. In addition, a photo 525 of “Mom” and a photo 530 of “Jane” may be presented in association with the suggested input. Although the present disclosure describes a particular scenario in which a particular multimodal user input is automatically completed in a particular manner, the present disclosure contemplates any suitable scenario in which any suitable multimodal user input is automatically completed in any suitable manner.

[0176] Figure 6 An example scenario of automatic completion of gesture input is shown. Figure 6 As shown, user 500 can interact with assistant system 140 via client system 130. User 500 may want to perform gestures to interact with assistant system 140, but does not really know what gestures he can perform. Therefore, user 500 may raise his hand in the air, which may have formed an incomplete gesture 605. Based on incomplete gesture 605, assistant system 140 may use a personalized gesture recognition model to determine one or more candidate gestures. As an example and not limitation, one or more candidate gestures may include two gestures, namely, waving and tapping. Assistant system 140 may also generate suggested inputs that can be displayed on screen 610 of client system 130. In Figure 6 In the example, the suggested input is presented as “Please perform one of the following gestures to let me know how I can help you” 615. In addition, an animation 620 of a waving gesture and an animation 625 of a tapping gesture may be presented in association with the suggested input. Although the present disclosure describes a specific scenario in which a specific gesture input is automatically completed in a specific manner, the present disclosure contemplates any suitable scenario in which any suitable gesture input is automatically completed in any suitable manner.

[0177] Figure 7A-7B Another example scenario of automatic completion of gesture input is shown. Fig. 7A An example scenario of an incomplete gesture in an AR / VR setting is shown. Fig. 7A As shown, user 500 may be wearing a pair of AR glasses 505. AR glasses 505 can be used as a client system 130 for user 500 to interact with assistant system 140. User 500 can see the dog on the screen 705 of AR glasses 505. User 500 may want to perform gestures to interact with assistant system 140, but does not really know what gestures he can perform. Therefore, user 500 may raise his hands in the air, which may have formed an incomplete gesture 710. Based on the incomplete gesture 710 and the focus of the user's attention (i.e., the dog), assistant system 140 can use a personalized gesture recognition model to determine one or more candidate gestures. Assistant system 140 can also generate suggested inputs that can be displayed on the screen 705 of AR glasses 505. Figure 7BAn example scenario of suggesting gesture input in an AR / VR setting is shown. Figure 7B , the suggested input is presented as “Please perform the displayed gesture to take a photo” 715. In addition, a “take a photo” gesture 720 (i.e., a dotted line gesture) may be presented in association with the suggested input. Although the present disclosure describes a specific scenario in which a specific gesture input is automatically completed in a specific manner, the present disclosure contemplates any suitable scenario in which any suitable gesture input is automatically completed in any suitable manner.

[0178] Figure 8 An example method 800 for suggesting multimodal user input 405 for auto-completion is shown. The method may begin at step 810, where assistant system 140 may receive an initial input from a first user from a client system 130 associated with the first user, where the initial input is in a first modality. At step 820, assistant system 140 may determine, via intent understanding module 410, one or more intents 420 corresponding to the initial input. At step 830, assistant system 140 may generate one or more candidate continuation inputs based on the one or more intents 420, where the one or more candidate continuation inputs are respectively in one or more candidate modalities, and where the candidate modalities are different from the first modality. At step 840, assistant system 140 may send instructions to client system 130 for presenting one or more suggested inputs corresponding to the one or more candidate continuation inputs. Particular embodiments may be repeated where appropriate. Figure 8 One or more steps of the method. Although the present disclosure describes and illustrates Figure 8 The specific steps of the method are preferably performed in a specific order, but the present disclosure contemplates Figure 8 Any suitable steps of the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates an example method for suggesting multimodal user input for auto-completion including Figure 8 method, but the present disclosure contemplates any suitable method for suggesting multimodal user input for auto-completion including any suitable steps, which, where appropriate, may include Figure 8 All steps, some steps, or none of the steps Figure 8 In addition, although the present disclosure describes and illustrates specific components, devices, or systems that perform specific steps of the method, the present disclosure contemplates the implementation of Figure 8 Any suitable combination of any suitable components, devices or systems of any suitable steps of a method.

[0179] Fig. 9An example method 900 for suggesting candidate gestures for auto-completion is shown. The method may begin at step 910, where assistant system 140 may receive user input 405 from a first user from a client system 130 associated with the first user, wherein user input 405 includes an incomplete gesture performed by the first user. At step 920, assistant system 140 may calculate, via intent understanding module 410, one or more confidence scores for one or more intents 420 corresponding to the incomplete gestures. At step 930, assistant system 140 may determine that the calculated confidence score associated with each intent 420 is below a threshold score. At step 940, in response to determining that the calculated confidence score for each intent 420 is below the threshold score, assistant system 140 may select one or more candidate gestures from a plurality of predefined gestures based on a personalized gesture recognition model, wherein each candidate gesture is associated with a confidence score representing a likelihood that the first user intended to enter the corresponding candidate gesture. At step 950, assistant system 140 may send instructions to client system 130 for presenting one or more suggested inputs corresponding to one or more candidate gestures. Particular embodiments may repeat the steps described above where appropriate. Fig. 9 Although this disclosure describes and illustrates specific steps of a method as occurring in a particular order, this disclosure contemplates that Fig. 9 Any suitable steps of the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates an example method for suggesting candidate gestures for automatic completion including Fig. 9 method, but the present disclosure contemplates any suitable method for suggesting candidate gestures for automatic completion including any suitable steps, which, where appropriate, may include Fig. 9 All steps, some steps, or none of the steps Fig. 9 In addition, although the present disclosure describes and illustrates specific components, devices, or systems that perform specific steps of the method, the present disclosure contemplates the implementation of Fig. 9 Any suitable combination of any suitable components, devices or systems of any suitable steps of a method.

[0180] Social Graph

[0181] Fig.10An example social graph 1000 is shown. In certain embodiments, social networking system 160 may store one or more social graphs 1000 in one or more data stores. In certain embodiments, social graph 1000 may include a plurality of nodes, which may include a plurality of user nodes 1002 or a plurality of concept nodes 1004, and a plurality of edges 1006 connecting the nodes. Each node may be associated with a unique entity (i.e., a user or a concept), and each entity may have a unique identifier (ID), such as a unique number or username. For teaching purposes, Fig.10 An example social graph 1000 is shown in a two-dimensional visual map representation in FIG. 1000. In certain embodiments, social networking system 160, client system 130, assistant system 140, or third-party system 170 may access social graph 1000 and related social graph information for use in appropriate applications. The nodes and edges of social graph 1000 may be stored as data objects, for example, in a data store (e.g., a social graph database). Such a data store may include one or more searchable or queryable indexes of the nodes or edges of social graph 1000.

[0182] In certain embodiments, user nodes 1002 may correspond to users of social networking system 160 or assistant system 140. By way of example and not limitation, a user may be an individual (human user), an entity (e.g., a business, company, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with or through social networking system 160 or assistant system 140. In certain embodiments, when a user registers an account with social networking system 160, social networking system 160 may create a user node 1002 corresponding to the user and store the user node 1002 in one or more data stores. Users and user nodes 1002 described herein may refer to registered users and user nodes 1002 associated with registered users, where appropriate. Additionally or alternatively, users and user nodes 1002 described herein may refer to users who are not registered with social networking system 160, where appropriate. In certain embodiments, user nodes 1002 may be associated with information provided by a user or information collected by various systems, including social networking system 160. As an example and not limitation, a user may provide his or her name, profile picture, contact information, date of birth, gender, marital status, family status, occupation, educational background, preferences, interests, or other demographic information. In certain embodiments, user node 1002 may be associated with one or more data objects corresponding to information associated with the user. In certain embodiments, user node 1002 may correspond to one or more web interfaces.

[0183] In certain embodiments, concept nodes 1004 may correspond to concepts. By way of example and not limitation, a concept may correspond to a place (such as, for example, a movie theater, a restaurant, a landmark, or a city); a website (such as, for example, a website associated with social networking system 160 or a third-party website associated with a web application server); an entity (such as, for example, a person, a business, a group, a sports team, or a celebrity); a resource (such as, for example, an audio file, a video file, a digital photo, a text file, a structured document, or an application), which may be located within social networking system 160 or on an external server (e.g., a web application server); real estate or intellectual property (such as, for example, a sculpture, a painting, a movie, a game, a song, an idea, a photo, or a written work); a game; an activity; an idea or theory; another suitable concept; or two or more such concepts. Concept nodes 1004 may be associated with information about the concept provided by a user or information collected by various systems, including social networking system 160 and assistant system 140. By way of example and not limitation, information about a concept may include a name or title; one or more images (e.g., an image of a book cover); a location (e.g., an address or geographic location); a website (which may be associated with a URL); contact information (e.g., a phone number or email address); other suitable concept information; or any suitable combination of such information. In particular embodiments, a concept node 1004 may be associated with one or more data objects that correspond to the information associated with the concept node 1004. In particular embodiments, a concept node 1004 may correspond to one or more web interfaces.

[0184] In certain embodiments, a node in social graph 1000 may represent or be represented by a web interface (which may be referred to as a "profile interface"). The profile interface may be hosted by social networking system 160 or assistant system 140 or accessible to social networking system 160 or assistant system 140. The profile interface may also be hosted on a third-party website associated with third-party system 170. As an example and not limitation, a profile interface corresponding to a particular external web interface may be a particular external web interface, and the profile interface may correspond to a particular concept node 1004. The profile interface may be viewable by all or a selected subset of other users. As an example and not limitation, user node 1002 may have a corresponding user profile interface, where the corresponding user may add content, make statements, or otherwise express him or herself. As another example and not limitation, concept node 1004 may have a corresponding concept profile interface, where one or more users may add content, make statements, or express themselves, particularly with respect to the concept corresponding to concept node 1004.

[0185] In certain embodiments, concept node 1004 may represent a third-party web interface or resource hosted by third-party system 170. The third-party web interface or resource may include content representing an action or activity, selectable icons or other icons or other interactable objects (which may be implemented, for example, with JavaScript, AJAX, or PHP code), and other elements. By way of example and not limitation, the third-party web interface may include selectable icons such as “Like,” “Check-in,” “Eat,” “Recommend,” or other suitable actions or activities. A user viewing the third-party web interface may perform an action by selecting one of the icons (e.g., “Check-in”), causing client system 130 to send a message to social-networking system 160 indicating the user's action. In response to the message, social-networking system 160 may create an edge (e.g., a check-in-type edge) between user node 1002 corresponding to the user and concept node 1004 corresponding to the third-party web interface or resource, and store edge 1006 in one or more data stores.

[0186] In particular embodiments, a pair of nodes in the social graph 1000 may be connected to each other via one or more edges 1006. An edge 1006 connecting a pair of nodes may represent a relationship between the pair of nodes. In particular embodiments, an edge 1006 may include or represent one or more data objects or attributes corresponding to the relationship between a pair of nodes. As an example and not limitation, a first user may indicate that a second user is a "friend" of the first user. In response to the indication, the social networking system 160 may send a "friend request" to the second user. If the second user confirms the "friend request", the social networking system 160 may create an edge 1006 in the social graph 1000 connecting the first user's user node 1002 to the second user's user node 1002, and store the edge 1006 as social graph information in one or more data stores 164. Fig.10In the example of , the social graph 1000 includes an edge 1006 indicating a friend relationship between the user nodes 1002 of user "A" and user "B", and an edge indicating a friend relationship between the user nodes 1002 of user "C" and user "B". Although the present disclosure describes or illustrates a particular edge 1006 with a particular attribute connecting a particular user node 1002, the present disclosure contemplates any suitable edge 1006 with any suitable attribute connecting the user nodes 1002. By way of example and not limitation, the edge 1006 may represent a friendship, a family relationship, a business or employment relationship, a fan relationship (including, for example, likes, etc.), a follower relationship, a visitor relationship (including, for example, visits, views, check-ins, shares, etc.), a subscriber relationship, a superior / subordinate relationship, a reciprocal relationship, a non-reciprocal relationship, another suitable type of relationship, or two or more such relationships. In addition, although the present disclosure generally describes nodes as being connected, the present disclosure also describes users or concepts as being connected. Herein, references to connected users or concepts may refer to nodes corresponding to those users or concepts connected by one or more edges 1006 in the social graph 1000 , where appropriate.

[0187] In certain embodiments, edges 1006 between user nodes 1002 and concept nodes 1004 may represent certain actions or activities performed by a user associated with user node 1002 toward a concept associated with concept node 1004. By way of example and not limitation, Fig.10 As shown, a user can "Like," "Attend," "Play," "Listen to," "Cook," "Work at," or "Watch" a concept, each of which can correspond to an edge type or subtype. The concept profile interface corresponding to concept node 1004 may include, for example, a selectable "Check-in" icon (such as, for example, a clickable "Check-in" icon) or a selectable "Add to Favorites" icon. Similarly, after the user clicks on these icons, social networking system 160 may create a "Favorites" edge or a "Check-in" edge in response to a user action corresponding to the corresponding action. As another example and not limitation, a user (user "C") may use a particular application (SPOTIFY, which is an online music application) to listen to a particular song ("Imagine"). In this case, social networking system 160 may create a "Listen" edge 1006 and a "Use" edge (such as, for example, a clickable "Check-in" icon) between user node 1002 corresponding to the user and concept nodes 1004 corresponding to the song and application. Fig.10 ) to indicate that the user listened to the song and used the application. In addition, social networking system 160 can create a "play" edge 1006 (as shown in FIG. Fig.10as shown), to indicate that a particular song is played by a particular application. In this case, the "played" edge 1006 corresponds to an action performed by an external application (Spotify) on an external audio file (the song "Imagine"). Although this disclosure describes a particular edge 1006 having particular properties that connects the user node 1002 and the concept node 1004, this disclosure contemplates any suitable edge 1006 having any suitable properties that connects the user node 1002 and the concept node 1004. Additionally, although this disclosure describes an edge between the user node 1002 and the concept node 1004 that represents a single relationship, this disclosure contemplates an edge between the user node 1002 and the concept node 1004 that represents one or more relationships. By way of example and not limitation, the edge 1006 can represent that the user likes and uses a particular concept. Alternatively, another edge 1006 can represent each type of relationship (or multiple single relationships) between the user node 1002 and the concept node 1004 (as Fig.10 shown, between the user node 1002 of user "E" and the concept node 1004 of "Spotify").

[0188] In a particular embodiment, the social networking system 160 can create an edge 1006 between a user node 1002 and a concept node 1004 in the social graph 1000. By way of example and not limitation, a user who views a concept profile interface (such as, for example, by using a web browser or a dedicated application hosted by the user's client system 130) can indicate that he or she likes the concept represented by the concept node 1004 by clicking or selecting a "like" icon, which can cause the user's client system 130 to send a message to the social networking system 160 indicating that the user likes the concept associated with the concept profile interface. In response to the message, the social networking system 160 can create an edge 1006 between the user node 1002 associated with the user and the concept node 1004, as shown by the "like" edge 1006 between the user node and the concept node 1004. In a particular embodiment, the social networking system 160 can store the edge 1006 in one or more data stores. In a particular embodiment, the edge 1006 can be automatically formed by the social networking system 160 in response to a particular user action. By way of example and not limitation, if a first user uploads a picture, watches a movie, or listens to a song, an edge 1006 can be formed between the user node 1002 corresponding to the first user and the concept nodes 1004 corresponding to those concepts. Although this disclosure describes forming a particular edge 1006 in a particular manner, this disclosure contemplates forming any suitable edge 1006 in any suitable manner.

[0189] Vector Spaces and Embeddings

[0190] Fig.11An example view of vector space 1100 is shown. In certain embodiments, objects or n-grams may be represented in a d-dimensional vector space, where d represents any suitable number of dimensions. Although vector space 1100 is shown as a three-dimensional space, this is for illustrative purposes only, as vector space 1100 may have any suitable dimensions. In certain embodiments, n-grams may be represented in vector space 1100 as vectors, which are referred to as term embeddings. Each vector may include coordinates corresponding to a particular point in vector space 1100 (i.e., the end point of the vector). By way of example and not limitation, Fig.11 As shown, vectors 1110, 1120, and 1130 can be represented as points in vector space 1100. n-grams can be mapped to corresponding vector representations. By way of example and not limitation, by applying a function defined by a dictionary n-grams t1 and t2 can be mapped to vectors in vector space 1100, respectively. and Make and As another example and not limitation, a dictionary trained to map text to vector representations may be utilized, or such a dictionary itself may be generated by training. As another example and not limitation, a model (e.g., Word2vec) may be used to map n-grams to vector representations in vector space 1100. In certain embodiments, n-grams may be mapped to vector representations in vector space 1100 using a machine learning model (e.g., a neural network). The machine learning model may have been trained using a sequence of training data (e.g., a corpus of multiple objects each including n-grams).

[0191] In certain embodiments, an object may be represented as a vector in vector space 1100, which is referred to as a feature vector or object embedding. By way of example and not limitation, by applying the function Objects e1 and e2 can be mapped to vectors in vector space 1100 respectively. and Make and In certain embodiments, objects may be mapped to vectors based on one or more properties, attributes, or characteristics of the object, the object's relationship to other objects, or any other suitable information associated with the object. By way of example and not limitation, the function Objects can be mapped to vectors by feature extraction, which can start from an initial measurement data set and construct derived values ​​(e.g., features). By way of example and not limitation, objects including videos or images can be mapped to vectors by using algorithms to detect or isolate various desired parts or shapes of objects. The features used to calculate the vectors can be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transforms, edge direction, varying intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transforms (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information. As another example and not limitation, objects including audio data can be mapped to vectors based on features (e.g., spectral slope, pitch coefficient, audio spectrum centroid, audio spectrum envelope, Mel-frequency cepstrum, or any other suitable information). In a particular embodiment, when an object has data that is too large to be effectively processed or includes redundant data, the function The transformed reduced feature set (e.g., feature selection) can be used to map objects to vectors. In certain embodiments, the function Objects can be mapped to vectors based on one or more n-grams associated with the object e. Although this disclosure describes representing n-grams or objects in a vector space in a particular manner, this disclosure contemplates representing n-grams or objects in a vector space in any suitable manner.

[0192] In certain embodiments, social networking system 160 may calculate a similarity metric for the vectors in vector space 1100. The similarity metric may be cosine similarity, Minkowski distance, Mahalanobis distance, Jaccard similarity coefficient, or any suitable similarity metric. By way of example and not limitation, and The similarity measure can be cosine similarity As another example and not by way of limitation, and The similarity measure can be the Euclidean distance The similarity measure of two vectors may represent how similar two objects or n-grams corresponding to the two vectors, respectively, are to each other, as measured by the distance between the two vectors in vector space 1100. As an example and not limitation, based on the distance between the corresponding vectors, vector 1110 and vector 1120 may correspond to objects that are more similar to each other than objects corresponding to vector 1110 and vector 1130. Although this disclosure describes calculating similarity measures between vectors in a particular manner, this disclosure contemplates calculating similarity measures between vectors in any suitable manner.

[0193] More information about vector spaces, embeddings, feature vectors, and similarity measures can be found in U.S. patent application Ser. No. 14 / 949,436, filed on Nov. 23, 2015, U.S. patent application Ser. No. 15 / 286,315, filed on Oct. 5, 2016, and U.S. patent application Ser. No. 15 / 365,789, filed on Nov. 30, 2016, each of which is incorporated by reference.

[0194] Artificial Neural Networks

[0195] Fig.12 An example artificial neural network ("ANN") 1200 is shown. In particular embodiments, an ANN may refer to a computational model that includes one or more nodes. The example ANN 1200 may include an input layer 1210, hidden layers 1220, 1230, 1240, and an output layer 1250. Each layer of the ANN 1200 may include one or more nodes, such as node 1205 or node 1215. In particular embodiments, each node of the ANN may be connected to another node of the ANN. As an example and not limitation, each node of the input layer 1210 may be connected to one or more nodes of the hidden layer 1220. In particular embodiments, one or more nodes may be bias nodes (e.g., nodes in a layer that are not connected to any nodes in the previous layer and do not receive input from it). In particular embodiments, each node in each layer may be connected to one or more nodes of the previous or next layer. Although Fig.12 A particular ANN having a particular number of layers, a particular number of nodes, and particular connections between nodes is depicted, but the present disclosure contemplates any suitable ANN having any suitable number of layers, any suitable number of nodes, and any suitable connections between nodes. By way of example and not limitation, although Fig.12 A connection between each node of the input layer 1210 and each node of the hidden layer 1220 is depicted, but one or more nodes of the input layer 1210 may not be connected to one or more nodes of the hidden layer 1220 .

[0196] In certain embodiments, the ANN may be a feedforward ANN (e.g., an ANN without cycles or loops, where communication between nodes flows in one direction starting from the input layer and proceeding to successive layers). As an example and not limitation, the input of each node of the hidden layer 1220 may include the output of one or more nodes of the input layer 1210. As another example and not limitation, the input of each node of the output layer 1250 may include the output of one or more nodes of the hidden layer 1240. In certain embodiments, the ANN may be a deep neural network (e.g., a neural network including at least two hidden layers). In certain embodiments, the ANN may be a deep residual network. The deep residual network may be a feedforward ANN that includes hidden layers organized into residual blocks. The input of each residual block after the first residual block may be a function of the output of the previous residual block and the input of the previous residual block. As an example and not limitation, the input to residual block N may be F(x)+x, where F(x) may be the output of residual block N-1, and x may be the input to residual block N-1. Although the present disclosure describes a particular ANN, the present disclosure contemplates any suitable ANN.

[0197] In certain embodiments, an activation function may correspond to each node of the ANN. The activation function of a node may define the output of the node for a given input. In certain embodiments, the input of a node may include a set of inputs. By way of example and not limitation, the activation function may be an identity function, a binary step function, a logic function, or any other suitable function. As another example and not limitation, the activation function of node k may be a sigmoid function Hyperbolic tangent function Rectifier F k (s k )=max(0,s k ) or any other suitable function F k (s k ), where s k can be an effective input to node k. In certain embodiments, the inputs to the activation functions corresponding to the nodes can be weighted. Each node can generate an output using a corresponding activation function based on the weighted inputs. In certain embodiments, each connection between nodes can be associated with a weight. As an example and not limitation, a connection 1225 between node 1205 and node 1215 can have a weighting coefficient of 0.4, which can indicate that the output of node 1205 multiplied by 0.4 is used as the input to node 1215. As another example and not limitation, the output y of node k is k It can be k =F k (s k ), where F k can be the activation function corresponding to node k, Sk =∑ j (w jk x j ) can be a valid input to node k, x j can be the output of node j connected to node k, and w jk It can be a weighted coefficient between node j and node k. In a specific embodiment, the input of the node of the input layer can be based on a vector representing an object. Although the present disclosure describes specific inputs and outputs of nodes, the present disclosure contemplates any suitable inputs and outputs of nodes. In addition, although the present disclosure can describe specific connections and weights between nodes, the present disclosure contemplates any suitable connections and weights between nodes.

[0198] In certain embodiments, the ANN may be trained using training data. As an example and not limitation, the training data may include the input and expected output of the ANN 1200. As another example and not limitation, the training data may include vectors, each vector representing a training object and an expected label for each training object. In certain embodiments, training the ANN may include modifying the weights associated with the connections between the nodes of the ANN by optimizing an objective function. As an example and not limitation, a training method (e.g., a conjugate gradient method, a gradient descent method, a stochastic gradient descent) may be used to back propagate the sum of squared errors (e.g., using a cost function that minimizes the sum of squared errors) as a distance measurement between each vector representing the training object. In certain embodiments, the ANN may be trained using a discarding technique. As an example and not limitation, one or more nodes may be temporarily ignored during training (e.g., not receiving input and not generating output). For each training object, one or more nodes of the ANN may have a certain probability of being ignored. Nodes ignored for a particular training object may be different from nodes ignored for other training objects (e.g., nodes may be temporarily ignored on an object-by-object basis). Although the present disclosure describes training the ANN in a particular manner, the present disclosure contemplates training the ANN in any suitable manner.

[0199] privacy

[0200] In a particular embodiment, one or more objects (e.g., content or other types of objects) of a computing system may be associated with one or more privacy settings. One or more objects may be stored on or otherwise associated with any suitable computing system or application, such as, for example, a social networking system 160, a client system 130, an assistant system 140, a third-party system 170, a social networking application, an assistant application, a messaging application, a photo sharing application, or any other suitable computing system or application. Although the examples discussed herein are in the context of an online social network, these privacy settings may be applied to any other suitable computing system. The privacy settings (or "access settings") of an object may be stored in any suitable manner, such as, for example, associated with an object, indexed on an authorization server, in another suitable manner, or any suitable combination thereof. Privacy settings about an object may specify how the object (or specific information associated with the object) may be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, executed, displayed, or identified) in an online social network. When the privacy settings of an object allow a particular user or other entity to access the object, the object may be described as being "visible" to that user or other entity. By way of example and not limitation, a user of an online social network may specify a privacy setting on a user profile page that identifies a set of users that may access work experience information on the user profile page, thereby excluding other users from accessing that information.

[0201] In certain embodiments, the privacy settings of an object may specify a "blocked list" of users or other entities that should not be allowed to access certain information associated with the object. In certain embodiments, the blacklist may include third-party entities. The blacklist may specify one or more users or entities to which the object is invisible. As an example and not limitation, a user may specify a set of users that may not access an album associated with the user, thereby excluding those users from accessing the album (while also possibly allowing certain users not in the specified set of users to access the album). In certain embodiments, privacy settings may be associated with certain social graph elements. The privacy settings of a social graph element (e.g., a node or edge) may specify how an online social network may be used to access the social graph element, information associated with the social graph element, or an object associated with the social graph element. As an example and not limitation, a specific concept node 1004 corresponding to a specific photo may have a privacy setting that specifies that the photo can only be accessed by the user tagged in the photo and the friends of the user tagged in the photo. In certain embodiments, privacy settings may allow users to opt in or opt out of having their content, information, or actions stored / recorded by the social networking system 160 or assistant system 140 or shared with other systems (e.g., third-party systems 170). Although this disclosure describes using particular privacy settings in particular manners, this disclosure contemplates using any suitable privacy settings in any suitable manner.

[0202] In certain embodiments, privacy settings may be based on one or more nodes or edges of the social graph 1000. Privacy settings may be specified for one or more edges 1006 or edge types of the social graph 1000, or for one or more nodes 1002, 1004 or node types of the social graph 1000. A privacy setting applied to a particular edge 1006 connecting two nodes may control whether the relationship between two entities corresponding to the two nodes is visible to other users of the online social network. Similarly, a privacy setting applied to a particular node may control whether a user or concept corresponding to the node is visible to other users of the online social network. By way of example and not limitation, a first user may share an object to the social networking system 160. The object may be associated with a concept node 1004 connected to the user node 1002 of the first user by an edge 1006. The first user may specify a privacy setting that applies to a particular edge 1006 of a concept node 1004 connected to an object, or may specify a privacy setting that applies to all edges 1006 connected to a concept node 1004. By way of another example and not limitation, the first user may share a collection of objects (e.g., a collection of images) of a particular object type. The first user may specify privacy settings for all objects of that particular object type associated with the first user to have a particular privacy setting (e.g., specifying that all images posted by the first user are visible only to the first user's friends and / or users tagged in the images).

[0203] In certain embodiments, social-networking system 160 may present a "privacy wizard" to the first user (e.g., within a web page, module, one or more dialog boxes, or any other suitable interface) to assist the first user in specifying one or more privacy settings. The privacy wizard may display instructions, appropriate privacy-related information, current privacy settings, one or more input fields for accepting one or more inputs from the first user (which specifies a change or confirmation of a privacy setting), or any suitable combination thereof. In certain embodiments, social-networking system 160 may provide the first user with a "dashboard" function that may display the first user's current privacy settings to the first user. The dashboard function may be displayed to the first user at any appropriate time (e.g., after input from the first user invoking the dashboard function, after a particular event or triggering action occurs). The dashboard function may allow the first user to modify one or more current privacy settings of the first user at any time in any suitable manner (e.g., redirecting the first user to the privacy wizard).

[0204] The privacy settings associated with an object may specify any suitable granularity at which access is allowed or denied. By way of example and not limitation, access or denial of access may be specified for specific users (e.g., only me, my roommate, my boss), users within a particular degree of separation (e.g., friends, friends of friends), user groups (e.g., a gaming club, my family), user networks (e.g., employees of a particular employer, students or alumni of a particular university), all users ("public"), no users ("private"), users of third-party systems 170, specific applications (e.g., third-party applications, external websites), other suitable entities, or any suitable combination thereof. Although the present disclosure describes a specific granularity at which access is allowed or denied, the present disclosure contemplates any suitable granularity at which access is allowed or denied.

[0205] In certain embodiments, one or more servers 162 may be authorization / privacy servers for implementing privacy settings. In response to a request from a user (or other entity) for a particular object stored in data storage 164, social networking system 160 may send a request for the object to data storage 164. The request may identify a user associated with the request, and the object may be sent to the user (or the user's client system 130) only if the authorization server determines that the user is authorized to access the object based on the privacy settings associated with the object. If the requesting user is not authorized to access the object, the authorization server may prevent the requested object from being retrieved from data storage 164, or may prevent the requested object from being sent to the user. In a search-query context, an object may be provided as a search result only if the querying user is authorized to access the object, for example, if the privacy settings of the object allow it to be revealed to the querying user, discovered by the querying user, or otherwise visible to the querying user. In certain embodiments, the object may represent content visible to the user through the user's dynamic message. As an example and not limitation, one or more objects may be visible to the user's "Trending" page. In a particular embodiment, an object may correspond to a particular user. An object may be content associated with a particular user, or may be an account of a particular user or information stored on a social networking system 160 or other computing system. By way of example and not limitation, a first user may view one or more second users of an online social network through a “People You May Know” feature of an online social network or by viewing a list of friends of the first user. By way of example and not limitation, a first user may specify that they do not wish to see objects associated with a particular second user in their dynamic messages or friend lists. If the privacy settings of an object do not allow it to be revealed to the user, discovered by the user, or visible to the user, the object may be excluded from the search results. Although the present disclosure describes implementing privacy settings in a particular manner, the present disclosure contemplates implementing privacy settings in any suitable manner.

[0206] In certain embodiments, different objects of the same type associated with a user may have different privacy settings. Different types of objects associated with a user may have different types of privacy settings. As an example and not limitation, a first user may specify that the first user's status update is public, but any images shared by the first user are only visible to the first user's friends on an online social network. As another example and not limitation, a user may specify different privacy settings for different types of entities (such as individual users, friends of friends, followers, user groups, or corporate entities). As another example and not limitation, a first user may specify a group of users who may view a video posted by the first user while preventing the video from being visible to the first user's employer. In certain embodiments, different privacy settings may be provided for different user groups or user demographics. As an example and not limitation, a first user may specify that other users who attend the same university as the first user may view the first user's photos, but other users who are family members of the first user may not view those same photos.

[0207] In certain embodiments, social networking system 160 may provide one or more default privacy settings for each object of a certain object type. The privacy settings of an object that are set as default may be changed by a user associated with the object. As an example and not limitation, all images posted by a first user may have a default privacy setting that is visible only to the first user's friends, and for a particular image, the first user may change the privacy setting of the image to be visible to friends and friends of friends.

[0208] In certain embodiments, privacy settings may allow a first user to specify (e.g., by opting out, by not opting in) whether social-networking system 160 or assistant system 140 may receive, collect, record, or store certain objects or information associated with the user for any purpose. In certain embodiments, privacy settings may allow a first user to specify whether certain applications or processes may access, store, or use certain objects or information associated with the user. Privacy settings may allow a first user to opt in or opt out of having an object or information accessed, stored, or used by a certain application or process. Social-networking system 160 or assistant system 140 may access such information in order to provide certain functionality or services to the first user, while social-networking system 160 or assistant system 140 may not access the information for any other purpose. Prior to accessing, storing, or using such an object or information, social-networking system 160 or assistant system 140 may prompt the user to provide a privacy setting that specifies which applications or processes, if any, may access, store, or use the object or information before any such action is allowed. As an example and not limitation, a first user may transmit a message to a second user via an application associated with an online social network (e.g., a messaging app), and may specify a privacy setting that social networking system 160 or assistant system 140 should not store such messages.

[0209] In certain embodiments, a user may specify whether a particular type of object or information associated with a first user may be accessed, stored, or used by social networking system 160 or assistant system 140. By way of example and not limitation, a first user may specify that images sent by the first user via social networking system 160 or assistant system 140 may not be stored by social networking system 160 or assistant system 140. By way of another example and not limitation, a first user may specify that messages sent from the first user to a particular second user may not be stored by social networking system 160 or assistant system 140. By way of yet another example and not limitation, a first user may specify that all objects sent via a particular application may be saved by social networking system 160 or assistant system 140.

[0210] In certain embodiments, the privacy settings may allow the first user to specify whether a particular object or information associated with the first user may be accessed from a particular client system 130 or third-party system 170. The privacy settings may allow the first user to opt-in or opt-out of accessing objects or information from a particular device (e.g., a phone book on the user's smartphone), from a particular application (e.g., a messaging app), or from a particular system (e.g., an email server). The social networking system 160 or assistant system 140 may provide a default privacy setting for each device, system, or application, and / or may prompt the first user to specify a specific privacy setting for each context. As an example and not limitation, the first user may utilize the location service feature of the social networking system 160 or assistant system 140 to provide recommendations for restaurants or other places near the user. The first user's default privacy settings may specify that the social networking system 160 or assistant system 140 may use the location information provided from the first user's client device 130 to provide location-based services, but the social networking system 160 or assistant system 140 may not store the first user's location information or provide it to any third-party system 170. The first user may then update the privacy settings to allow a third-party image sharing application to use the location information to geotag photos.

[0211] In certain embodiments, privacy settings may allow a user to specify one or more geographic locations from which an object may be accessed. Access or denial of access to an object may depend on the geographic location of the user attempting to access the object. As an example and not limitation, a user may share an object and specify that only users in the same city may access or view the object. As another example and not limitation, a first user may share an object and specify that the object is visible to a second user only when the first user is in a particular location. If the first user leaves the particular location, the object may no longer be visible to the second user. As another example and not limitation, the first user may specify that the object is visible only to a second user within a threshold distance from the first user. If the first user subsequently changes location, the original second user who could access the object may lose access, and a new group of second users may gain access when they enter within a threshold distance of the first user.

[0212] In certain embodiments, social networking system 160 or assistant system 140 may have features that can use a user's personal or biometric information as input for user authentication or experience personalization purposes. Users may choose to utilize these features to enhance their experience on online social networks. By way of example and not limitation, a user may provide personal or biometric information to social networking system 160 or assistant system 140. The user's privacy settings may specify that such information may only be used for specific processes (such as authentication), and may also specify that such information may not be shared with any third-party system 170, or may not be used for other processes or applications associated with social networking system 160 or assistant system 140. As another example and not limitation, social networking system 160 may provide a user with the ability to provide a voiceprint recording to an online social network. By way of example and not limitation, if a user wishes to utilize this feature of an online social network, the user may provide a voice recording of his or her own voice to provide a status update on the online social network. The recording of the voice input may be compared to the user's voiceprint to determine what words the user said. The user's privacy settings may specify that such voice recordings may be used only for voice input purposes (e.g., authenticating the user, sending voice messages, improving voice recognition for use of voice manipulation features of the online social network), and further specify that such voice recordings may not be shared with any third-party system 170, or may not be used by other processes or applications associated with the social networking system 160. As another example and not limitation, the social networking system 160 may provide the user with the ability to provide a reference image (e.g., a facial profile, a retinal scan) to the online social network. The online social network may compare the reference image with an image input received later (e.g., for authenticating the user, tagging the user in a photo). The user's privacy settings may specify that such images may only be used for limited purposes (e.g., authentication, tagging the user in a photo), and further specify that such images may not be shared with any third-party system 170, or may not be used by other processes or applications associated with the social networking system 160.

[0213] System and method

[0214] Fig.13An example computer system 1300 is shown. In certain embodiments, one or more computer systems 1300 perform one or more steps of one or more methods described or shown herein. In certain embodiments, one or more computer systems 1300 provide functionality described or shown herein. In certain embodiments, software running on one or more computer systems 1300 performs one or more steps of one or more methods described or shown herein, or provides functionality described or shown herein. Certain embodiments include one or more portions of one or more computer systems 1300. Herein, references to computer systems may include computing devices, and vice versa, where appropriate. Furthermore, references to computer systems may include one or more computer systems, where appropriate.

[0215] The present disclosure contemplates any suitable number of computer systems 1300. The present disclosure contemplates that computer system 1300 takes any suitable physical form. By way of example and not limitation, computer system 1300 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 1300 may include one or more computer systems 1300; be monolithic or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1300 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not limitation, one or more computer systems 1300 may perform one or more steps of one or more methods described or shown herein in real time or in batch mode. Where appropriate, one or more computer systems 1300 may perform one or more steps of one or more methods described or shown herein at different times or at different locations.

[0216] In a particular embodiment, computer system 1300 includes a processor 1302, memory 1304, storage 1306, input / output (I / O) interface 1308, communication interface 1310, and bus 1312. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0217] In a particular embodiment, the processor 1302 includes hardware for executing instructions (e.g., those that constitute a computer program). By way of example and not limitation, to execute instructions, the processor 1302 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 1304, or a storage device 1306; decode these instructions and execute them; and then write one or more results to an internal register, an internal cache, a memory 1304, or a storage device 1306. In a particular embodiment, the processor 1302 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates that the processor 1302 includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 1302 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1304 or the storage device 1306, and the instruction cache may speed up the retrieval of those instructions by the processor 1302. The data in the data cache may be: a copy of data in the memory 1304 or storage 1306 for operating the instructions executed at the processor 1302; the result of a previous instruction executed at the processor 1302 for access by a subsequent instruction executed at the processor 1302 or for writing to the memory 1304 or storage 1306; or other suitable data. The data cache may speed up read or write operations performed by the processor 1302. The TLB may speed up virtual address translations for the processor 1302. In a particular embodiment, the processor 1302 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates that the processor 1302 includes any suitable number of any suitable internal registers. Where appropriate, the processor 1302 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1302. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0218] In a particular embodiment, the memory 1304 includes a main memory for storing instructions for the processor 1302 to execute or data for the processor 1302 to operate. As an example and not limitation, the computer system 1300 can load instructions from the storage device 1306 or another source (e.g., another computer system 1300) to the memory 1304. The processor 1302 can then load the instructions from the memory 1304 to an internal register or an internal cache. In order to execute the instructions, the processor 1302 can retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 1302 can write one or more results (which can be intermediate results or final results) to the internal register or the internal cache. The processor 1302 can then write one or more of these results to the memory 1304. In a particular embodiment, the processor 1302 executes only instructions in one or more internal registers or internal caches or in the memory 1304 (rather than elsewhere in the storage device 1306), and operates only on data in one or more internal registers or internal caches or in the memory 1304 (rather than in the storage device 1306 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 1302 to the memory 1304. As described below, the bus 1312 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 1302 and the memory 1304 and facilitate access to the memory 1304 requested by the processor 1302. In a particular embodiment, the memory 1304 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1304 may include one or more memories 1304. Although this disclosure describes and illustrates particular memories, this disclosure contemplates any suitable memories.

[0219] In a particular embodiment, storage device 1306 includes a large-capacity storage device for data or instructions. As an example and not limitation, storage device 1306 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, storage device 1306 may include a removable or non-removable (or fixed) medium. Where appropriate, storage device 1306 may be inside or outside computer system 1300. In a particular embodiment, storage device 1306 is a non-volatile solid-state memory. In a particular embodiment, storage device 1306 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM), or a flash memory, or a combination of two or more of these. The present disclosure contemplates a large-capacity storage device 1306 in any suitable physical form. Where appropriate, storage 1306 may include one or more storage control units that facilitate communications between processor 1302 and storage 1306. Where appropriate, storage 1306 may include one or more storage devices 1306. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0220] In a particular embodiment, I / O interface 1308 includes hardware, software, or both that provide one or more interfaces for communication between computer system 1300 and one or more I / O devices. Where appropriate, computer system 1300 may include one or more of these I / O devices. One or more of these I / O devices may enable communication between a person and computer system 1300. As an example and not limitation, an I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a static camera, a stylus, a tablet computer, a touch screen, a tracking ball, a video camera, another suitable I / O device, or a combination of two or more of these. An I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1308 for them. Where appropriate, an I / O interface 1308 may include one or more devices or software drivers that enable processor 1302 to drive one or more of these I / O devices. Where appropriate, an I / O interface 1308 may include one or more I / O interfaces 1308. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0221] In a particular embodiment, the communication interface 1310 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1300 and one or more other computer systems 1300 or one or more networks. By way of example and not limitation, the communication interface 1310 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (e.g., a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 1310 for it. By way of example and not limitation, the computer system 1300 may communicate with one or more parts of a self-organizing network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more parts of one or more of these networks may be wired or wireless. As an example, computer system 1300 may communicate with a wireless PAN (WPAN) (e.g., a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Where appropriate, computer system 1300 may include any suitable communication interface 1310 for any of these networks. Where appropriate, communication interface 1310 may include one or more communication interfaces 1310. Although this disclosure describes and illustrates particular communication interfaces, this disclosure contemplates any suitable communication interface.

[0222] In a particular embodiment, bus 1312 includes hardware, software, or both that couples components of computer system 1300 to each other. By way of example and not limitation, bus 1312 may include an accelerated graphics port (AGP) or other graphics bus, an extended industry standard architecture (EISA) bus, a front side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (Extended) (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or any other suitable bus, or a combination of two or more of these. Where appropriate, bus 1312 may include one or more buses 1312. Although the present disclosure describes and illustrates a particular bus, the present disclosure contemplates any suitable bus or interconnect.

[0223] Herein, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital (SECURE DIGITAL) cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, the computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0224] Other Miscellaneous

[0225] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated by context. Furthermore, "and" is both conjunctive and several, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated by context.

[0226] The scope of the present disclosure includes all changes, substitutions, variations, alterations and modifications to the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the corresponding embodiments herein as including specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or replacement of any components, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, references to devices or systems or components of devices or systems that are suitable for, arranged to, capable of, configured to, implemented to, operable to, or operated to perform specific functions in the appended claims include the device, system, component, whether it or that specific function is activated, turned on or unlocked, as long as the device, system or component is adjusted, arranged, enabled, configured, implemented, operable, or operated in this way. In addition, although the present disclosure describes or illustrates specific embodiments as providing specific advantages, specific embodiments may provide some, all or none of these advantages.

Claims

1. A method comprising: receiving, from a client system associated with a first user, user input comprising an incomplete gesture performed by the first user; calculating, by an intent understanding module, one or more confidence scores corresponding to one or more intents of the incomplete gesture; determining that the calculated confidence score associated with each intent is below a threshold score; In response to determining that the calculated confidence score for each intent is below the threshold score, selecting one or more candidate gestures from a plurality of predefined gestures based on the personalized gesture recognition model, wherein each candidate gesture is associated with a confidence score representing a likelihood that the first user intended to input the corresponding candidate gesture; as well as Instructions are sent to the client system to present one or more suggested inputs corresponding to one or more of the candidate gestures.

2. The method according to claim 1, further comprising: For each of the one or more candidate gestures, a similarity of the candidate gesture relative to the incomplete gesture is calculated.

3. The method according to claim 2, wherein: The similarity of each candidate gesture relative to the incomplete gesture is based on a trajectory of the incomplete gesture relative to the client system.

4. The method according to claim 2, wherein: The similarity of each candidate gesture relative to the incomplete gesture is based on an orientation of the incomplete gesture relative to the client system.

5. The method according to claim 2, wherein: The similarity of each candidate gesture relative to the incomplete gesture is based on an object associated with the incomplete gesture.

6. The method according to claim 2, wherein: The similarity of each candidate gesture relative to the incomplete gesture is based on contextual information associated with the incomplete gesture.

7. The method according to claim 2, wherein: The similarity of each candidate gesture relative to the incomplete gesture is based on a location of the incomplete gesture relative to the client system.

8. The method according to claim 1, wherein: The threshold score is based on a wake-up gesture performed by the first user.

9. The method according to claim 1, wherein: Calculating one or more confidence scores corresponding to one or more intents of the incomplete gesture is based on a speed associated with the incomplete gesture.

10. The method according to claim 1, wherein: Calculating one or more confidence scores corresponding to one or more intents of the incomplete gesture is based on time information associated with the incomplete gesture, and wherein the time information includes pauses in the user input.

Images