Personalized gesture recognition for user interaction with assistant system

Abstract

The invention relates to personalized gesture recognition for user interaction with an assistant system. In one embodiment, a method includes accessing, from a data store, a plurality of input tuples associated with a first user, where each input tuple includes a gesture input and a corresponding voice input, determining, by a natural language understanding (NLU) module, a plurality of intent corresponding to the plurality of voice inputs, respectively, and transmitting the plurality of intent to the first user. Generating a plurality of feature representations for the plurality of gesture inputs based on one or more machine learning models, determining a plurality of gesture identifiers for the plurality of gesture inputs, respectively, based on respective feature representations of the plurality of gesture inputs, associating a plurality of intent with the plurality of gesture identifiers, respectively, and training a personalized gesture classification model for the first user based on the plurality of feature representations of their respective gesture inputs and associations between the plurality of intent and their respective gesture identifiers.

Description

[0001] This application is a divisional application of an application with a filing date of April 18, 2019, application number 201980040760.6, and invention name “Personalized gesture recognition for user interaction with assistant system”. Technical Field

[0002] The present disclosure generally relates to conversation management based on machine learning techniques within a network environment, and specifically to hardware and software for a smart assistant system. Background Art

[0003] The assistant system can provide information or services to the user based on a combination of user input, location awareness, and the ability to access information from various online sources (such as weather conditions, traffic congestion, news, stock prices, user schedules, retail prices, etc.). User input can include text (e.g., online chat) (especially text in instant messaging applications or other applications), voice, images, motion, orientation, or a combination thereof. The assistant system can perform concierge-type services (e.g., making dinner reservations, purchasing event tickets, arranging travel) or provide information based on user input. The assistant system can also perform management or data processing tasks based on online information and events without user initiation or interaction. Examples of those tasks that can be performed by the assistant system can include schedule management (e.g., sending a warning message to a dinner appointment that the user is going to be late due to traffic conditions, updating the schedules of both parties, and changing restaurant reservation times). The assistant system can be implemented through a combination of computing devices, application programming interfaces (APIs), and application proliferation on user devices.

[0004] A social networking system, which may include a social networking website, may enable its users (e.g., individuals or organizations) to interact with it and with each other through it. The social networking system may utilize input from a user to create and store a user profile associated with the user in the social networking system. The user profile may include demographic information, communication channel information, and information about personal interests of the user. The social networking system may also utilize input from the user to create and store a record of the user's relationships with other users of the social networking system, as well as provide services (e.g., profile / news feed postings, photo sharing, event organization, messaging, games, or advertising) to facilitate social interaction between or among users.

[0005] The social networking system may send content or messages related to its services to the user's mobile device or other computing device through one or more networks. The user may also install a software application on the user's mobile device or other computing device for accessing the user's user profile and other data within the social networking system. The social networking system may generate a set of personalized content objects to display to the user, such as a dynamic message of aggregated stories of other users connected to the user. Summary of the invention

[0006] In a particular embodiment, the assistant system can help users obtain information or services. The assistant system can enable users to interact with it in a stateful and multi-turn conversation using multimodal user input (e.g., sound, text, image, video, motion, orientation) to obtain help. The assistant system can create and store a user profile, which includes personal information and contextual information associated with the user. In a particular embodiment, the assistant system can use natural language understanding to analyze user input. The analysis can be based on the user profile to obtain a more personalized and context-aware understanding. The assistant system can resolve entities associated with user input based on the analysis. In a particular embodiment, the assistant system can interact with different agents to obtain information or services associated with the resolved entity. The assistant system can generate responses about information or services for users by using natural language generation. Through interaction with the user, the assistant system can use dialogue management technology to manage and forward the conversation flow with the user. In a particular embodiment, the assistant system can also assist users to digest the information obtained effectively and efficiently by summarizing the information. The assistant system can also help users better participate in online social networks by providing tools that help users interact with the online social network (e.g., create posts, comments, messages). The assistant system can additionally help users manage different tasks, such as keeping track of events. In certain embodiments, the assistant system can proactively perform tasks related to the user's interests and preferences based on the user profile without user input. In certain embodiments, the assistant system can check privacy settings to ensure that access to the user's profile or other user information and the performance of different tasks are allowed according to the user's privacy settings.

[0007] In a particular embodiment, the assistant system can train a personalized gesture classification model. The training can include receiving multiple input tuples, each input tuple including gesture input and voice input, determining multiple intents based on the voice input through a natural language understanding (NLU) module, accessing a general gesture classification model, associating intents with their respective gesture inputs, and training the personalized gesture classification model based on the association between intent and gesture input and the general gesture classification model. In a particular embodiment, the general gesture classification model can be generated by crowdsourcing about a large number of standard gestures. However, the general gesture classification model may not be able to determine the correct intent of a specific user gesture, especially if the user is using non-standard gestures or gestures that are commonly used for different intents. Therefore, the training of the personalized gesture classification model can additionally utilize information prompts from the user's own input in addition to the general gesture classification model, and the user's own input includes voice and gestures. In addition to using voice input alone to assist the training of the personalized gesture classification model, the assistant system can alternatively train the personalized gesture classification model by jointly using voice input and gesture input. Once the personalized gesture classification model is trained, the assistant system can use it in the future to determine the user's intent corresponding to his / her own gesture. Although this disclosure describes training a particular gesture classification model in a particular manner via a particular system, this disclosure contemplates training any suitable gesture classification model in any suitable manner via any suitable system.

[0008] In a particular embodiment, the assistant system may access multiple input tuples associated with the first user from a data storage device. Each input tuple may include a gesture input and a corresponding voice input. In a particular embodiment, the assistant system may then determine multiple intentions corresponding to the multiple voice inputs respectively through a natural language understanding (NLU) module. The assistant system may then generate multiple feature representations for the multiple gesture inputs based on one or more machine learning models. In a particular embodiment, the assistant system may then determine multiple gesture identifiers for the multiple gesture inputs respectively based on the feature representations of the multiple gesture inputs. The assistant system may then associate the multiple intentions with the multiple gesture identifiers respectively. In a particular embodiment, the assistant system may further train a personalized gesture classification model for the first user based on the multiple feature representations of their respective gesture inputs and the association between the multiple intentions and their respective gesture identifiers.

[0009] There may be certain technical challenges in achieving the goal of training a personalized gesture classification model. One technical challenge may include enabling the personalized gesture classification model to recognize the user's intention from the user's personal gestures. The solution proposed by the embodiments disclosed herein to solve the above challenges is to train the personalized gesture classification model based on both the user's voice input and gesture input, because the user's intention can be learned from the voice input, and the association between these intentions and gesture inputs can be generated for further use during the training of the personalized gesture classification model. Another technical challenge may include generating a reliable feature representation for the gesture input. The solution proposed by the embodiments disclosed herein to solve this challenge is to model the components of the gesture input and the temporal information associated with the gesture input into the feature representation, because dividing the gesture input into components can help distinguish the gesture from another gesture, and the temporal information can carry contextual information indicating the user's intention, which is useful for gesture classification.

[0010] Certain embodiments disclosed herein may provide one or more technical advantages. The technical advantages of the embodiments may include enriching the user's experience of the assistant system by enabling the user to interact with the assistant system with gestures in addition to traditional inputs (such as text and voice), to which the assistant system can accurately recognize the user's gestures and perform tasks corresponding to the recognized gestures. Certain embodiments disclosed herein may not provide, provide some or all of the above technical advantages. In view of the drawings, description, and claims of the present disclosure, one or more other technical advantages may be apparent to those skilled in the art.

[0011] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to them. Specific embodiments may include all, some or none of the components, elements, features, functions, operations or steps of the embodiments disclosed herein. Embodiments according to the present invention are specifically disclosed in the attached claims relating to methods, storage media, systems, assistant systems and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or back references in the attached claims are selected only for formal reasons. However, any subject matter generated by intentional back references (especially multiple references) to any previous claim may also be claimed, so that any combination of claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features as set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any features of the appended claims.

[0012] In an embodiment, a method, particularly for use in an assistant system, for assisting a user in obtaining information or services by enabling the user to interact with the assistant system in a conversation using user input to obtain assistance, wherein the user input includes sound, text, image, video, motion or orientation or any combination thereof, the assistant system being particularly implemented by a combination of a computing device, an application programming interface (API), and application proliferation on a user device, may include, by one or more computing systems:

[0013] accessing from a data store a plurality of input tuples associated with a first user, wherein each input tuple comprises a gesture input and a corresponding voice input;

[0014] Determining, by a natural language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech inputs, respectively;

[0015] generating a plurality of feature representations for the plurality of gesture inputs based on one or more machine learning models;

[0016] Determining a plurality of gesture identifiers for the plurality of gesture inputs based on respective feature representations of the plurality of gesture inputs;

[0017] Associating the plurality of intents with the plurality of gesture identifiers, respectively; and

[0018] A personalized gesture classification model is trained for the first user based on the plurality of feature representations of their respective gesture inputs and the associations between the plurality of intents and their respective gesture identifiers.

[0019] In an embodiment, a method may include:

[0020] A general gesture classification model corresponding to a general user population is accessed from a data storage, wherein the training of the personalized gesture classification model is also based on the general gesture classification model.

[0021] A general gesture classification model can be trained based on multiple gesture inputs from a general user population.

[0022] In an embodiment, a method may include:

[0023] A plurality of text inputs are generated for the plurality of speech inputs respectively by one or more automatic speech recognition (ASR) modules.

[0024] Respective determination of multiple intents corresponding to multiple voice inputs may be based on multiple text inputs of the corresponding voice inputs.

[0025] The one or more machine learning models may be based on one or more of a neural network model or a long short-term memory (LSTM) model.

[0026] The personalized gesture classification model can be based on a convolutional neural network.

[0027] Generating each feature representation for each gesture input may include:

[0028] Breaking the gesture input into one or more component parts; and

[0029] The one or more components are modeled into a feature representation of the gesture input.

[0030] Generating each feature representation for each gesture input may include:

[0031] determining temporal information associated with the gesture input; and

[0032] Modeling temporal information into feature representations of gesture inputs.

[0033] In an embodiment, a method may include:

[0034] receiving, from a client system associated with the first user, new gesture input from the first user; and

[0035] Based on the personalized gesture classification model, an intent corresponding to the new gesture input is determined for the new gesture input.

[0036] In an embodiment, a method may include:

[0037] One or more tasks are performed based on the determined intent.

[0038] The input tuple or gesture input and / or voice input may be captured by a client system associated with the assistant system.

[0039] The client system may be a virtual reality (VR) headset and / or augmented reality (AR) smart glasses.

[0040] The client system may receive user input from an I / O device including one or more sensors.

[0041] The I / O devices may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a video camera, another suitable I / O device, or a combination of two or more of these devices.

[0042] In an embodiment, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to:

[0043] accessing from a data store a plurality of input tuples associated with a first user, wherein each input tuple comprises a gesture input and a corresponding voice input;

[0044] Determining, by a natural language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech inputs, respectively;

[0045] generating a plurality of feature representations for the plurality of gesture inputs based on one or more machine learning models;

[0046] Determining a plurality of gesture identifiers for the plurality of gesture inputs based on respective feature representations of the plurality of gesture inputs;

[0047] Associating the plurality of intents with the plurality of gesture identifiers, respectively; and

[0048] A personalized gesture classification model is trained for the first user based on the plurality of feature representations of their respective gesture inputs and the associations between the plurality of intents and their respective gesture identifiers.

[0049] The software, when executed, is operable to:

[0050] A general gesture classification model corresponding to a general user group is accessed from a data storage, wherein the training of the personalized gesture classification model is also based on the general gesture classification model.

[0051] A general gesture classification model can be trained based on multiple gesture inputs from a general user population.

[0052] The software, when executed, is operable to:

[0053] A plurality of text inputs are generated for the plurality of speech inputs respectively by one or more automatic speech recognition (ASR) modules.

[0054] Respective determination of multiple intents corresponding to multiple voice inputs may be based on multiple text inputs of the corresponding voice inputs.

[0055] The one or more machine learning models may be based on one or more of a neural network model or a long short-term memory (LSTM) model.

[0056] The personalized gesture classification model can be based on a convolutional neural network.

[0057] Generating each feature representation for each gesture input may include:

[0058] Breaking the gesture input into one or more component parts; and

[0059] The one or more components are modeled into a feature representation of the gesture input.

[0060] In an embodiment, a system may include: one or more processors; and a non-transitory memory coupled to the processor, the non-transitory memory including instructions executable by the processor, the processor being operable when executing the instructions to:

[0061] accessing from a data store a plurality of input tuples associated with a first user, wherein each input tuple comprises a gesture input and a corresponding voice input;

[0062] Determining, by a natural language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech inputs, respectively;

[0063] generating a plurality of feature representations for the plurality of gesture inputs based on one or more machine learning models;

[0064] Determining a plurality of gesture identifiers for the plurality of gesture inputs based on respective feature representations of the plurality of gesture inputs;

[0065] Associating the plurality of intents with the plurality of gesture identifiers, respectively; and

[0066] A personalized gesture classification model is trained for the first user based on the plurality of feature representations of their respective gesture inputs and the associations between the plurality of intents and their respective gesture identifiers.

[0067] In an embodiment, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to perform a method according to the present invention or any of the above-mentioned embodiments.

[0068] In an embodiment, a system may include: one or more processors; and at least one memory, which is coupled to the processor and includes instructions executable by the processor, and the processor is operable to perform a method according to the present invention or any of the above-mentioned embodiments when executing the instructions.

[0069] In an embodiment, a computer program product, preferably comprising a computer readable non-transitory storage medium, when executed on a data processing system, may be operable to perform a method according to the present invention or any of the above mentioned embodiments.

[0070] In an embodiment, an assistant system is used to help a user obtain information or services by enabling the user to interact with the assistant system using user input in a conversation to obtain help, wherein the user input includes sound, text, image, video, motion or orientation or any combination thereof. The assistant system is specifically implemented through a combination of a computing device, an application programming interface (API), and a proliferation of applications on a user device. The system may include: one or more processors; and a non-volatile memory coupled to the processor, the non-volatile memory including instructions executable by the processor, and the processor is operable to perform a method according to the present invention or any of the above-mentioned embodiments when executing the instructions.

[0071] In an embodiment, the assistant system may assist the user by performing at least one or more of the following features or steps:

[0072] - Create and store a user profile that includes personal and contextual information associated with the user

[0073] - Use an intent understanding module to analyze user input, where the intent understanding module can include a natural language understanding module and a gesture classification model

[0074] -Analyze user input using natural language understanding, where the analysis can be based on user profiles for a more personalized and context-aware understanding

[0075] - Resolve entities associated with user input based on analysis

[0076] -Interact with different agents to obtain information or services associated with the resolved entity

[0077] - Generate responses for users regarding that information or service by using natural language generation

[0078] - Manage and forward conversation flows with users through interaction with them using conversation management techniques

[0079] - Help users digest the information they have obtained effectively and efficiently by aggregating the information

[0080] - Help users better participate in online social networks by providing tools that help users interact with online social networks (e.g., create posts, comments, messages)

[0081] - Helps users manage different tasks, such as keeping track of events

[0082] - Proactively perform pre-authorized tasks related to the user's interests and preferences based on the user profile at a time relevant to the user and without user input

[0083] - Checking privacy settings whenever necessary to ensure that accessing a user profile and performing different tasks adheres to the user's privacy settings.

[0084] In an embodiment, an assistant system may include at least one or more of the following components:

[0085] a messaging platform for receiving a text-based user input from a client system associated with a user, and / or for receiving an image or video-based user input and processing the image or video-based user input within the messaging platform using optical character recognition technology to convert the user input into text,

[0086] an audio speech recognition (ASR) module for receiving audio modality-based user input from a client system associated with a user (e.g., the user may speak or send a video including voice) and converting the audio modality-based user input into text,

[0087] - Assistant xbot, which is used to receive the output of the messaging platform or the ASR module.

[0088] In an embodiment, a system may include:

[0089] at least one client system, in particular an electronic device,

[0090] At least one assistant system according to the present invention or any embodiment herein,

[0091] The client system and the assistant system are connected to each other in particular via a network,

[0092] The client system includes an assistant application that allows a user of the client system to interact with the assistant system.

[0093] wherein the assistant application transmits the user input to the assistant system, and the assistant system generates a response based on the user input and sends the generated response to the assistant application, and the assistant application presents the response to the user of the client system,

[0094] In particular, the user input is audio or verbal, and the response may be textual or also audio or verbal.

[0095] In an embodiment, a system may include a social networking system,

[0096] The client system particularly includes a social network application for accessing the social network system. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 An example network environment associated with an assistant system is shown.

[0098] Figure 2 An example architecture for an assistant system is shown.

[0099] Figure 3 An example flow chart of an assistant system responding to a user request is shown.

[0100] Figure 4A-4C Example gestures are shown.

[0101] Figure 5 An example workflow for processing user input including gesture input is shown.

[0102] Figure 6 An example method for training a personalized gesture classification model is shown.

[0103] Figure 7 An example social graph is shown.

[0104] Figure 8 An example view of the embedding space is shown.

[0105] Fig. 9 An example artificial neural network is shown.

[0106] Fig.10 An example computer system is shown. DETAILED DESCRIPTION

[0107] Systematic review

[0108] Figure 1 An example network environment 100 associated with an assistant system is shown. The network environment 100 includes a client system 130, an assistant system 140, a social networking system 160, and a third-party system 170 connected to each other via a network 110. Figure 1A particular arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110 is shown, but the present disclosure contemplates any suitable arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110. By way of example and not limitation, two or more of client system 130, social networking system 160, assistant system 140, and third-party system 170 may be directly connected to each other, bypassing network 110. As another example, two or more of client system 130, assistant system 140, social networking system 160, and third-party system 170 may be physically or logically co-located with each other, in whole or in part. Furthermore, although Figure 1 A particular number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110 are shown, but this disclosure contemplates any suitable number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110. By way of example and not limitation, network environment 100 may include a plurality of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110.

[0109] The present disclosure contemplates any suitable network 110. By way of example and not limitation, one or more portions of network 110 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 110 may include one or more networks 110.

[0110] Links 150 may connect client systems 130, assistant systems 140, social networking systems 160, and third-party systems 170 to communication network 110 or to each other. The present disclosure contemplates any suitable links 150. In particular embodiments, one or more links 150 include one or more wired (such as, for example, digital subscriber lines (DSL) or cable-based data service interface specifications (DOCSIS)) links, wireless (such as, for example, Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or optical (such as, for example, synchronous optical network (SONET) or synchronous digital hierarchy (SDH)) links. In particular embodiments, one or more links 150 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 150, or a combination of two or more such links 150. Links 150 need not necessarily be the same throughout network environment 100. The one or more first links 150 may differ from the one or more second links 150 in one or more aspects.

[0111] In certain embodiments, client system 130 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more such components, and is capable of performing appropriate functions implemented or supported by client system 130. By way of example and not limitation, client system 130 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a smart speaker, a virtual reality (VR) headset, an augmented reality (AR) smart glasses, other suitable electronic devices, or any suitable combination thereof. In certain embodiments, client system 130 may be a smart assistant device. More information about smart assistant devices can be found in U.S. Patent Application No. 15 / 949,011, filed on April 9, 2018, U.S. Patent Application No. 62 / 655,751, filed on April 10, 2018, U.S. Design Patent Application No. 29 / 631,910, filed on January 3, 2018, U.S. Design Patent Application No. 29 / 631,747, filed on January 2, 2018, U.S. Design Patent Application No. 29 / 631,913, filed on January 3, 2018, and U.S. Design Patent Application No. 29 / 631,914, filed on January 3, 2018, each of which is incorporated by reference. The present disclosure contemplates any suitable client system 130. Client system 130 can enable network users at client system 130 to access network 110. Client system 130 can enable its users to communicate with other users at other client systems 130.

[0112] In a particular embodiment, the client system 130 may include a web browser 132, such as MICROSOFTINTERNET EXPLORER, GOOGLE CHROME or MOZILLA FIREFOX, and may have one or more add-ons, plug-ins or other extensions, such as TOOLBAR or YAHOO TOOLBAR. A user at the client system 130 may enter a uniform resource locator (URL) or other address that directs the web browser 132 to a specific server (such as server 162 or a server associated with a third-party system 170), and the web browser 132 may generate a hypertext transfer protocol (HTTP) request and pass the HTTP request to the server. The server may accept the HTTP request and pass one or more hypertext markup language (HTML) files to the client system 130 in response to the HTTP request. The client system 130 may display a web interface (such as a web page) based on an HTML file from a server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not as a limitation, a web interface may be displayed according to an HTML file, an extensible hypertext markup language (XHTML) file or an extensible markup language (XML) file according to specific needs. Such an interface may also execute scripts, such as, for example and without limitation, scripts written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, a combination of markup language and script (e.g., AJAX (Asynchronous JAVASCRIPT and XML)), etc. Herein, references to a web interface include one or more corresponding source files (which a browser may use to render the web interface), and vice versa, where appropriate.

[0113] In certain embodiments, client system 130 may include social network application 134 installed on client system 130. A user at client system 130 may use social network application 134 to access an online social network. A user at client system 130 may use social network application 134 to communicate with the user's social connections (e.g., friends, followers, followed accounts, contacts, etc.). A user at client system 130 may also use social network application 134 to interact with a plurality of content objects (e.g., posts, news articles, temporary content, etc.) on an online social network. As an example and not by way of limitation, a user may use social network application 134 to browse trending topics and breaking news.

[0114] In certain embodiments, the client system 130 may include an assistant application 136. The user of the client system 130 may use the assistant application 136 to interact with the assistant system 140. In certain embodiments, the assistant application 136 may include an independent application. In certain embodiments, the assistant application 136 may be integrated into the social network application 134 or another suitable application (e.g., a messaging application). In certain embodiments, the assistant application 136 may also be integrated into the client system 130, the assistant hardware device, or any other suitable hardware device. In certain embodiments, the assistant application 136 may be accessed via a web browser 132. In certain embodiments, the user may provide input via different modalities. As an example and not as a limitation, the modality may include audio, text, image, video, motion, orientation, etc. The assistant application 136 may transmit the user input to the assistant system 140. Based on the user input, the assistant system 140 may generate a response. The assistant system 140 may send the generated response to the assistant application 136. Then, the assistant application 136 may present the response to the user of the client system 130. The presented response may be based on different modalities, such as audio, text, image, and video. As an example and not by way of limitation, a user may verbally ask assistant application 136 for traffic information (i.e., via an audio modality). Assistant application 136 may then transmit the request to assistant system 140. Assistant system 140 may generate results accordingly and send them back to assistant application 136. Assistant application 136 may also present the results to the user in text.

[0115] In a particular embodiment, the assistant system 140 can help the user retrieve information from different sources. The assistant system 140 can also help the user request services from different service providers. In a particular embodiment, the assistant system 140 can receive a user's request for information or services via the assistant application 136 in the client system 130. The assistant system 140 can use natural language understanding to analyze the user request based on the user profile and other relevant information. The results of the analysis may include different entities associated with online social networks. The assistant system 140 can then retrieve information or request services associated with these entities. In a particular embodiment, when retrieving information or requesting services for the user, the assistant system 140 can interact with the social networking system 160 and / or the third-party system 170. In a particular embodiment, the assistant system 140 can generate personalized communication content for the user using natural language generation technology. The personalized communication content may include, for example, the status of the retrieved information or the requested service. In a particular embodiment, the assistant system 140 can enable the user to interact with it about information or services in a stateful and multi-round conversation by using dialogue management technology. In the following Figure 2 The functionality of assistant system 140 is described in more detail in the discussion of .

[0116] In certain embodiments, social networking system 160 may be a network-addressable computing system that may host an online social network. Social networking system 160 may generate, store, receive, and transmit social networking data (such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to an online social network). Social networking system 160 may be accessed by other components of network environment 100 directly or via network 110. As an example and not by way of limitation, client system 130 may access social networking system 160 directly or via network 110 using a web browser 132 or a native application associated with social networking system 160 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In certain embodiments, social networking system 160 may include one or more servers 162. Each server 162 may be a unitary server or a distributed server across multiple computers or multiple data centers. The server 162 may be of various types, such as, for example and without limitation, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 162 may include hardware, software, or embedded logic components for performing the appropriate functions implemented or supported by the server 162, or a combination of two or more such components. In a particular embodiment, the social networking system 160 may include one or more data storage devices 164. The data storage device 164 may be used to store various types of information. In a particular embodiment, the information stored in the data storage device 164 may be organized according to a particular data structure. In a particular embodiment, each data storage device 164 may be a relational database, a columnar database, a correlation database, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. Particular embodiments may provide an interface that enables client system 130 , social networking system 160 , assistant system 140 , or third-party system 170 to manage, retrieve, modify, add, or delete information stored in data store 164 .

[0117] In certain embodiments, social networking system 160 may store one or more social graphs in one or more data stores 164. In certain embodiments, a social graph may include a plurality of nodes, which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept), and a plurality of edges connecting the nodes. Social networking system 160 may provide users of an online social network with the ability to communicate and interact with other users. In certain embodiments, a user may join an online social network via social networking system 160 and then add connections (e.g., relationships) with a plurality of other users in social networking system 160 to whom they want to be connected. Herein, the term "friend" may refer to any other user of social networking system 160 with whom the user has formed a connection, association, or relationship via social networking system 160.

[0118] In particular embodiments, social-networking system 160 may provide users with the ability to take actions on various types of items or objects supported by social-networking system 160. By way of example and not limitation, items and objects may include groups or social networks to which users of social-networking system 160 may belong, events or calendar entries that may be of interest to users, computer-based applications that users may use, transactions that allow users to buy or sell goods via a service, interactions with advertisements that users may perform, or other suitable items or objects. Users may interact with anything that can be represented in social-networking system 160 or by an external system such as third-party system 170 that is separate from social-networking system 160 and coupled to social-networking system 160 via network 110.

[0119] In certain embodiments, social networking system 160 can link various entities. As an example and not by way of limitation, social networking system 160 can enable users to interact with each other and receive content from third-party systems 170 or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0120] In certain embodiments, third-party systems 170 may include one or more types of servers, one or more data stores, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable components (e.g., the server may communicate with these components). Third-party systems 170 may be operated by a different entity than the entity that operates social-networking system 160. However, in certain embodiments, social-networking system 160 and third-party systems 170 may operate in conjunction with each other to provide social-networking services to users of social-networking system 160 or third-party systems 170. In this sense, social-networking system 160 may provide a platform or backbone that other systems (e.g., third-party systems 170) may use to provide social-networking services and functionality to users across the Internet.

[0121] In certain embodiments, third-party system 170 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be delivered to client system 130. By way of example and not limitation, content objects may include information about things or activities of interest to the user, such as, for example, movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not limitation, content objects may include incentive content objects (e.g., coupons, discount coupons, gift certificates, or other suitable incentive objects).

[0122] In certain embodiments, social networking system 160 also includes user-generated content objects that can enhance a user's interaction with social networking system 160. User-generated content can include any content that a user can add, upload, send, or "post" to social networking system 160. As an example and not by way of limitation, a user transmits a post from client system 130 to social networking system 160. Posts can include data, such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to social networking system 160 by third parties through "communication channels" such as dynamic messages or streams.

[0123] In certain embodiments, the social networking system 160 may include various servers, subsystems, programs, modules, logs, and data storage. In certain embodiments, the social networking system 160 may include one or more of the following: a web server, an action recorder, an API request server, a correlation and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertisement-targeting module, a user interface module, a user profile storage, a relationship storage, a third-party content storage, or a location storage. The social networking system 160 may also include suitable components, such as a network interface, a security mechanism, a load balancer, a failover server, a management and network operation console, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 160 may include one or more user profile storages for storing user profiles. A user profile may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, affinities, or locations). Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not by way of limitation, if a user "likes" an article about a brand of shoes, the category may be the brand, or the general category of "shoes" or "clothing". The association storage may be used to store association information about users. The association information may indicate users with similar or common work experience, group membership, hobbies, educational history, or who are related or share common attributes in any way. The association information may also include user-defined associations between different users and content (internal and external). The web server may be used to link the social networking system 160 to one or more client systems 130 or one or more third-party systems 170 via the network 110. The web server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 160 and one or more client systems 130. The API request server may allow the assistant system 140 and the third-party system 170 to access information from the social networking system 160 by calling one or more APIs. The action recorder may be used to receive communications from the web server about the user's actions on or outside the social networking system 160. In conjunction with the action log, a third-party content object log of user exposure to third-party content objects can be maintained. The notification controller can provide information about content objects to the client system 130. The information can be pushed to the client system 130 as a notification, or the information can be pulled from the client system 130 in response to a request received from the client system 130.The authorization server may be used to enforce one or more privacy settings of users of the social networking system 160. The privacy settings of the user determine how certain information associated with the user may be shared. The authorization server may allow the user to opt-in or opt-out of having their actions recorded by the social networking system 160 or shared with other systems (e.g., third-party systems 170), such as by setting appropriate privacy settings. A third-party content object store may be used to store content objects received from a third party (e.g., third-party system 170). A location store may be used to store location information associated with the user received from the client system 130. The advertising pricing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0124] Assistant System

[0125] Figure 2An example architecture of an assistant system 140 is shown. In a particular embodiment, the assistant system 140 can help users obtain information or services. The assistant system 140 can enable users to interact with it in a stateful and multi-turn conversation with multimodal user input (such as sound, text, image, video, motion, orientation) to obtain help. The assistant system 140 can create and store a user profile including personal information and contextual information associated with the user. In a particular embodiment, the assistant system 140 can use natural language understanding to analyze user input. The analysis can be based on the user profile to obtain a more personalized and context-aware understanding. The assistant system 140 can parse entities associated with the user input based on the analysis. In a particular embodiment, the assistant system 140 can interact with different agents to obtain information or services associated with the parsed entity. The assistant system 140 can generate responses about information or services for the user by using natural language generation. Through interaction with the user, the assistant system 140 can use dialogue management technology to manage and forward the conversation flow with the user. In a particular embodiment, the assistant system 140 can also help users digest the information obtained effectively and efficiently by summarizing the information. The assistant system 140 can also help users become more engaged in online social networks by providing tools that help users interact with online social networks (e.g., create posts, comments, messages). The assistant system 140 can also help users manage different tasks, such as keeping track of events. In certain embodiments, the assistant system 140 can proactively perform pre-authorized tasks related to user interests and preferences based on the user profile at a time relevant to the user without user input. In certain embodiments, the assistant system 140 can check privacy settings to ensure that access to the user's profile or other user information and the performance of different tasks are allowed according to the user's privacy settings. More information about helping users based on privacy settings can be found in U.S. Patent Application No. 62 / 675,090, filed on May 22, 2018, which is incorporated by reference.

[0126] In certain embodiments, the assistant system 140 may receive user input from an assistant application 136 in a client system 130 associated with a user. In certain embodiments, the user input may be a user-generated input that is sent to the assistant system 140 in a single round. If the user input is based on a text modality, the assistant system 140 may receive it at the messaging platform 205. If the user input is based on an audio modality (e.g., the user may speak to the assistant application 136 or send a video including voice to the assistant application 136), the assistant system 140 may process it using an automatic speech recognition (ASR) module 210 to convert the user input into text. If the user input is based on an image or video modality, the assistant system 140 may process it using optical character recognition technology within the messaging platform 205 to convert the user input into text. The output of the messaging platform 205 or the ASR module 210 may be received at the assistant xbot 215. More information on processing user input based on different modalities may be found in U.S. Patent Application No. 16 / 053600 filed on August 2, 2018, which is incorporated by reference.

[0127] In a particular embodiment, assistant xbot 215 can be a type of chatbot. Assistant xbot 215 can include a programmable service channel, which can be a software code, logic or routine used as a user personal assistant. Assistant xbot 215 can serve as a user portal of assistant system 140. Therefore, assistant xbot 215 can be considered as a type of conversational agent. In a particular embodiment, assistant xbot 215 can send text user input to natural language understanding (NLU) module 220 to interpret user input. In a particular embodiment, NLU module 220 can obtain information from user context engine 225 and semantic information aggregator (SIA) 230 to accurately understand user input. User context engine 225 can store user profiles of users. User profiles of users can include user profile data, which includes demographic information, social information and context information associated with users. User profile data can also include user interests and preferences for multiple topics aggregated by dynamic messages, search logs, messaging platforms 205, etc. The use of user profiles may be protected by the privacy check module 245 to ensure that the user's information can only be used for his / her benefit and not shared with anyone else. More information about user profiles can be found in U.S. Patent Application No. 15 / 967,239 filed on April 30, 2018, which is incorporated by reference. The semantic information aggregator 230 may provide ontology data associated with a plurality of predefined domains, intents, and slots to the NLU module 220. In a particular embodiment, a domain may represent a social context of an interaction, e.g., education. An intent may be an element in a predefined classification of semantic intents, which may indicate the purpose of a user's interaction with the assistant system 140. In a particular embodiment, if the user input includes text / voice input, the intent may be an output of the NLU module 220. The NLU module 220 may classify the text / voice input as a member of a predefined classification, e.g., for the input "play Beethoven's Fifth Symphony", the NLU module 220 may classify the input as having the intent [IN: play_music]. In certain embodiments, a domain can be conceptually a namespace for a set of intents, e.g., music. A slot can be a named substring with user input that represents a basic semantic entity. For example, the slot for "pizza" can be [SL:dish]. In certain embodiments, the set of valid or expected named slots can be based on the intent of the classification. As an example and not by way of limitation, for [IN:play_music], the slot can be [SL:song_name].The semantic information aggregator 230 can also extract information from social graphs, knowledge graphs, and concept graphs, and retrieve user profiles from the user context engine 225. The semantic information aggregator 230 can also process information from these different sources by determining what information to aggregate, annotating n-grams of user input, sorting n-grams with confidence scores based on the aggregated information, and formulating the sorted n-grams into features that can be used by the NLU module 220 to understand the user input. More information about aggregated semantic information can be found in U.S. Patent Application No. 15 / 967,342 filed on April 30, 2018, which is incorporated by reference. Based on the output of the user context engine 225 and the semantic information aggregator 230, the NLU module 220 can identify domains, intents, and one or more slots from the user input in a personalized and context-aware manner. As an example and not by way of limitation, the user input can include "show me how to get to the Starbucks". The NLU module 220 can identify a specific Starbucks that the user wants to go to based on the user's personal information and associated context information. In a particular embodiment, the NLU module 220 may include a lexicon of language, a parser, and grammar rules to divide sentences into internal representations. The NLU module 220 may also include one or more programs that use pragmatics to perform naive semantics or random semantic analysis to understand user input. In a particular embodiment, the parser may be based on a deep learning architecture including multiple long short-term memory (LSTM) networks. As an example and not as a limitation, the parser may be based on a recurrent neural network grammar (RNNG) model, which is a type of recurrent and recursive LSTM algorithm. More information about natural language understanding can be found in U.S. Patent Application No. 16 / 011062 filed on June 18, 2018, U.S. Patent Application No. 16 / 025317 filed on July 2, 2018, and U.S. Patent Application No. 16 / 038120 filed on July 17, 2018, each of which is incorporated by reference.

[0128] In a particular embodiment, the identified domain, intent, and one or more slots from the NLU module 220 may be sent to the conversation engine 235. In a particular embodiment, the conversation engine 235 may manage the conversation flow and conversation state between the user and the assistant xbot 215. The conversation engine 235 may also store previous conversations between the user and the assistant xbot 215. In a particular embodiment, the conversation engine 235 may communicate with the entity resolution module 240 to resolve entities associated with one or more slots, which supports the conversation engine 235 to forward the conversation flow between the user and the assistant xbot 215. In a particular embodiment, the entity resolution module 240 may access social graphs, knowledge graphs, and concept graphs when resolving entities. An entity may include, for example, a unique user or concept, each of which may have a unique identifier (ID). As an example and not as a limitation, a knowledge graph may include multiple entities. Each entity may include a single record associated with one or more attribute values. A specific record may be associated with a unique entity identifier. For an attribute of an entity, each record may have a different value. Each attribute value may be associated with a confidence probability. The confidence probability of an attribute value indicates the probability that the value is accurate for a given attribute. Each attribute value may also be associated with a semantic weight. The semantic weight of an attribute value may indicate the degree to which the value is semantically suitable for a given attribute, taking into account all available information. For example, a knowledge graph may include an entity of the movie "The Martian" (2015), which includes information that has been extracted from multiple content sources (e.g., Facebook, Wikipedia, movie review sources, media databases, and entertainment content sources), and then deduped, parsed, and fused to generate a single unique record of a knowledge graph. An entity may be associated with a spatial attribute value indicating the genre of the movie "The Martian" (2015). More information about the knowledge graph may be found in U.S. Patent Application No. 16 / 048049 filed on July 27, 2018 and U.S. Patent Application No. 16 / 048101 filed on July 27, 2018, each of which is incorporated by reference. The entity resolution module 240 may additionally request a user profile of a user associated with a user input from the user context engine 225. In certain embodiments, entity resolution module 240 may communicate with privacy check module 245 to ensure that the resolution of the entity does not violate the privacy policy. In certain embodiments, privacy check module 245 may use an authorization / privacy server to enforce the privacy policy. As an example and not by way of limitation, the entity to be resolved may be another user who has specified in his / her privacy settings that his / her identity should not be searchable on an online social network, and therefore entity resolution module 240 may respond to the request without returning an identifier for that user.Based on information obtained from social graphs, knowledge graphs, concept graphs, and user profiles, and in compliance with applicable privacy policies, the entity resolution module 240 can therefore accurately resolve entities associated with user input in a personalized and context-aware manner. In a particular embodiment, each resolved entity can be associated with one or more identifiers hosted by the social networking system 160. As an example and not by way of limitation, the identifier can include a unique user identifier (ID). In a particular embodiment, each resolved entity can also be associated with a confidence score. More information about resolved entities can be found in U.S. Patent Application No. 16 / 048049, filed on July 27, 2018, and U.S. Patent Application No. 16 / 048072, filed on July 27, 2018, each of which is incorporated by reference.

[0129] In a particular embodiment, the conversation engine 235 can communicate with different agents based on the identified intent and domain and the resolved entity. In a particular embodiment, the agent can be an implementation of a broker between multiple content providers acting as a domain. A content provider can be an entity responsible for performing an action associated with an intent or completing a task associated with an intent. As an example and not as a limitation, multiple device-specific implementations (e.g., real-time calls to a messaging application on or to a client system 130) can be handled internally by a single agent. Alternatively, these device-specific implementations can be handled by multiple agents associated with multiple domains. In a particular embodiment, the agent may include a first-party agent 250 and a third-party agent 255. In a particular embodiment, the first-party agent 250 may include an internal agent (e.g., an agent associated with a service provided by an online social network (Messenger, Instagram)) that can be accessed and controlled by the assistant system 140. In a particular embodiment, the third-party agent 255 may include an external agent (e.g., a music streaming agent (Spotify), a ticket sales agent (Ticketmaster)) that the assistant system 140 cannot control. The first party agent 250 may be associated with a first party provider 260 that provides content objects and / or services hosted by the social networking system 160. The third party agent 255 may be associated with a third party provider 265 that provides content objects and / or services hosted by the third party system 170.

[0130] In certain embodiments, communications from the conversation engine 235 to the first party agent 250 may include a request for a specific content object and / or service provided by the first party provider 260. Thus, the first party agent 250 may retrieve the requested content object from the first party provider 260 and / or execute a task that instructs the first party provider 260 to perform the requested service. In certain embodiments, communications from the conversation engine 235 to the third party agent 255 may include a request for a specific content object and / or service provided by the third party provider 265. Thus, the third party agent 255 may retrieve the requested content object from the third party provider 265 and / or execute a task that instructs the third party provider 265 to perform the requested service. The third party agent 255 may access the privacy check module 245 to ensure that there is no privacy violation before interacting with the third party provider 265. As an example and not by way of limitation, a user associated with the user input may specify in his / her privacy settings that his / her profile information is not visible to any third party content provider. Thus, when retrieving a content object associated with a user input from a third party provider 265 , the third party agent 255 may complete the retrieval without revealing to the third party provider 265 which user is requesting the content object.

[0131] In certain embodiments, each of the first party agent 250 or the third party agent 255 may be designated for a specific domain. By way of example and not limitation, the domain may include weather, transportation, music, etc. In certain embodiments, the assistant system 140 may use multiple agents in concert to respond to user input. By way of example and not limitation, the user input may include "direct me to my next meeting". The assistant system 140 may use a calendar agent to retrieve the location of the next meeting. The assistant system 140 may then use a navigation agent to direct the user to the next meeting.

[0132] In certain embodiments, each of the first party agent 250 or the third party agent 255 may retrieve a user profile from the user context engine 225 to perform the task in a personalized and context-aware manner. By way of example and not limitation, the user input may include "book me a ride to the airport". The transportation agent may perform the task of booking a ride. The transportation agent may retrieve the user profile of the user from the user context engine 225 before booking the ride. For example, the user profile may indicate that the user prefers taxis, so the transportation agent may book a taxi for the user. As another example, context information associated with the user profile may indicate that the user is in a hurry, so the transportation agent may book a ride for the user from a ride-sharing service (e.g., Uber, Lyft) because getting a ride from a ride-sharing service may be faster than a taxi company. In certain embodiments, each of the first party agent 250 or the third party agent 255 may consider other factors when performing the task. By way of example and not limitation, the other factors may include price, ratings, efficiency, partnerships with online social networks, etc.

[0133] In certain embodiments, the dialog engine 235 may communicate with a conversation understanding writer (CU writer) 270. The dialog engine 235 may send the requested content object and / or the status of the requested service to the CU writer 270. In certain embodiments, the dialog engine 235 may send the requested content object and / or the status of the requested service to the CU writer 270.<k,c,u,d> Tuples are sent, where k indicates a knowledge source, c indicates a communication target, u indicates a user model, and d indicates a discourse model. In a specific embodiment, the CU writer 270 may include a natural language generator (NLG) 271 and a user interface (UI) payload generator 272. The natural language generator 271 may generate communication content based on the output of the dialogue engine 235. In a specific embodiment, the NLG 271 may include a content determination component, a sentence planner, and a surface realization component. The content determination component may determine the communication content based on the knowledge source, the communication target, and the user's expectations. As an example and not as a limitation, the determination may be based on description logic. The description logic may include, for example, three basic notions, which are individuals (representing objects in a domain), concepts (describing individual sets), and roles (representing binary relationships between individuals or concepts). The description logic may be characterized by a set of constructors that allow the natural language generator 271 to build complex concepts / roles from atomic concepts / roles. In a specific embodiment, the content determination component may perform the following tasks to determine the communication content. The first task may include a translation task, in which the input to the natural language generator 271 can be translated into concepts. The second task may include a selection task, in which relevant concepts can be selected from the concepts generated by the translation task based on the user model. The third task may include a verification task, in which the consistency of the selected concept can be verified. The fourth task may include an instantiation task, in which the verified concept can be instantiated into an executable file that can be processed by the natural language generator 271. The sentence planner can determine the organization of the communication content so that it is understandable to people. The surface implementation component can determine the specific words to be used, the order of sentences, and the style of the communication content. The UI payload generator 272 can determine the preferred modality of the communication content to be presented to the user. In a specific embodiment, the CU writer 270 can communicate with the privacy check module 245 to ensure that the generation of the communication content follows the privacy policy. In a specific embodiment, when generating the communication content and determining the modality of the communication content, the CU writer 270 can retrieve the user profile from the user context engine 225. Therefore, for the user, the communication content can be more natural, personalized, and context-aware. As an example and not by way of limitation, a user profile may indicate that the user prefers short sentences in conversation, and thus the generated communication content may be based on short sentences.As another example and not by way of limitation, context information associated with the user profile may indicate that the user is using a device that only outputs audio signals, so the UI payload generator 272 may determine the modality of the communication content as audio. More information about natural language generation can be found in U.S. Patent Application No. 15 / 967,279, filed on April 30, 2018, and U.S. Patent Application No. 15 / 966,455, filed on April 30, 2018, each of which is incorporated by reference.

[0134] In certain embodiments, the CU writer 270 may send the generated communication content to the assistant xbot 215. In certain embodiments, the assistant xbot 215 may send the communication content to the messaging platform 205. The messaging platform 205 may also send the communication content to the client system 130 via the assistant application 136. In an alternative embodiment, the assistant xbot 215 may send the communication content to a text-to-speech (TTS) module 275. The TTS module 275 may convert the communication content into an audio clip. The TTS module 275 may also send the audio clip to the client system 130 via the assistant application 136.

[0135] In certain embodiments, assistant xbot 215 may interact with proactive inference layer 280 without receiving user input. Proactive inference layer 280 may infer user interests and preferences based on a user profile retrieved from user context engine 225. In certain embodiments, proactive inference layer 280 may also communicate with proactive agent 285 regarding inferences. Proactive agent 285 may perform proactive tasks based on inferences. As an example and not as a limitation, proactive tasks may include sending content objects to users or providing services. In certain embodiments, each proactive task may be associated with an agenda item. Agenda items may include recurring items, such as daily summaries. Agenda items may also include one-time items. In certain embodiments, proactive agent 285 may retrieve a user profile from user context engine 225 when performing proactive tasks. Thus, proactive agent 285 may perform proactive tasks in a personalized and context-aware manner. As an example and not as a limitation, the proactive inference layer may infer that a user likes the band Maroon 5, and proactive agent 285 may generate recommendations for new songs / albums of Maroon 5 for the user.

[0136] In a particular embodiment, the active agent 285 can generate candidate entities associated with the active task based on the user profile. The generation can be based on a direct back-end query that retrieves the candidate entity from a structured data storage using a deterministic filter. Alternatively, the generation can be based on a machine learning model that is trained based on the user profile, entity attributes, and the correlation between the user and the entity. As an example and not as a limitation, the machine learning model can be based on a support vector machine (SVM). As another example and not as a limitation, the machine learning model can be based on a regression model. As another example and not as a limitation, the machine learning model can be based on a deep convolutional neural network (DCNN). In a particular embodiment, the active agent 285 can also sort the generated candidate entities based on the user profile and the content associated with the candidate entity. The sorting can be based on the similarity between the user's interest and the candidate entity. As an example and not as a limitation, the assistant system 140 can generate a feature vector representing the user's interest and a feature vector representing the candidate entity. The assistant system 140 can then calculate a similarity score (e.g., based on cosine similarity) between the feature vector representing the user's interest and the feature vector representing the candidate entity. Alternatively, the ranking may be based on a ranking model that is trained based on user feedback data.

[0137] In certain embodiments, the active task may include recommending a candidate entity to a user. The active agent 285 may schedule the recommendation, thereby associating the recommendation time with the recommended candidate entity. The recommended candidate entity may also be associated with a priority and an expiration time. In certain embodiments, the recommended candidate entity may be sent to an active scheduler. The active scheduler may determine the actual time to send the recommended candidate entity to the user based on the priority associated with the task and other relevant factors (e.g., clicks and impressions of the recommended candidate entity). In certain embodiments, the active scheduler may then send the recommended candidate entity with the determined actual time to an asynchronous tier. The asynchronous tier may temporarily store the recommended candidate entity as a job. In certain embodiments, the asynchronous tier may send the job to the conversation engine 235 at the determined actual time for execution. In alternative embodiments, the asynchronous tier may execute the job by sending it to other surface layers (e.g., other notification services associated with the social networking system 160). In certain embodiments, the conversation engine 235 may identify the conversation intent, state, and history associated with the user. Based on the conversation intent, the conversation engine 235 may select some candidate entities from the recommended candidate entities to send to the client system 130. In a particular embodiment, the conversation state and history may indicate whether the user is involved in an ongoing conversation with assistant xbot 215. If the user is involved in an ongoing conversation and the priority of the recommended task is low, the conversation engine 235 may communicate with the active scheduler to reschedule the time when the selected candidate entity is sent to the client system 130. If the user is involved in an ongoing conversation and the priority of the recommended task is high, the conversation engine 235 may initiate a new conversation session (dialogsession) with the user, in which the selected candidate entity may be presented. Therefore, the interruption of the ongoing conversation may be prevented. When it is determined that sending the selected candidate entity will not interrupt the user, the conversation engine 235 may send the selected candidate entity to the CU writer 270 to generate personalized and context-aware communication content including the selected candidate entity according to the user's privacy settings. In a particular embodiment, the CU writer 270 may send the communication content to the assistant xbot 215, which may then be sent to the client system 130 via the messaging platform 205 or the TTS module 275. More information about proactively helping users can be found in U.S. Patent Application No. 15 / 967,193, filed on April 30, 2018, and U.S. Patent Application No. 16 / 036,827, filed on July 16, 2018, each of which is incorporated by reference.

[0138] In certain embodiments, assistant xbot 215 may communicate with proactive agent 285 in response to user input. As an example and not by way of limitation, a user may ask assistant xbot 215 to set a reminder. Assistant xbot 215 may request proactive agent 285 to set such a reminder, and proactive agent 285 may later proactively perform the task of reminding the user.

[0139] In a particular embodiment, the assistant system 140 may include a summarizer 290. The summarizer 290 may provide a customized dynamic message summary to the user. In a particular embodiment, the summarizer 290 may include multiple metaagents. Multiple metaagents may use first-party agents 250, third-party agents 255, or active agents 285 to generate dynamic message summaries. In a particular embodiment, the summarizer 290 may retrieve user interests and preferences from the active inference layer 280. The summarizer 290 may then retrieve entities associated with user interests and preferences from the entity resolution module 240. The summarizer 290 may also retrieve a user profile from the user context engine 225. Based on information from the active inference layer 280, the entity resolution module 240, and the user context engine 225, the summarizer 290 may generate a personalized and context-aware summary for the user. In a particular embodiment, the summarizer 290 may send the summary to the CU writer 270. The CU writer 270 may process the summary and send the processing result to the assistant xbot 215. Assistant xbot 215 may then send the processed summary to client system 130 via messaging platform 205 or TTS module 275. More information about summarization can be found in U.S. Patent Application No. 15 / 967,290, filed April 30, 2018, which is incorporated by reference.

[0140] Figure 3An example flow chart of the assistant system 140 responding to a user request is shown. In a particular embodiment, the assistant xbot 215 can access the request manager 305 when receiving a user request. The request manager 305 can include a context extractor 306 and a conversation understanding object generator (CU object generator) 307. The context extractor 306 can extract context information associated with the user request. The context extractor 306 can also update the context information based on the assistant application 136 executed on the client system 130. As an example and not as a limitation, the update of the context information can include displaying a content item on the client system 130. As another example and not as a limitation, the update of the context information can include whether an alarm is set on the client system 130. As another example and not as a limitation, the update of the context information can include whether a song is playing on the client system 130. The CU object generator 307 can generate a specific content object related to the user request. The content object can include conversation session data and features associated with the user request, which can be shared with all modules of the assistant system 140. In particular embodiments, request manager 305 may store context information and generated content objects in data store 310 , which is a particular data store implemented in assistant system 140 .

[0141] In a particular embodiment, the request manager 305 may send the generated content object to the NLU module 220. The NLU module 220 may perform a number of steps to process the content object. In step 221, the NLU module 220 may generate a whitelist of content objects. In a particular embodiment, the whitelist may include interpretation data that matches the user request. In step 222, the NLU module 220 may perform characterization based on the whitelist. In step 223, the NLU module 220 may perform domain classification / selection on the user request based on the features generated from the characterization to classify the user request into a predefined domain. The domain classification / selection results may also be further processed based on two related processes. In step 224a, the NLU module 220 may use an intent classifier to process the domain classification / selection results. The intent classifier may determine the user intent associated with the user request. In a particular embodiment, there may be one intent classifier per domain to determine the most likely intent in a given domain. As an example and not by way of limitation, the intent classifier may be based on a machine learning model that may take the domain classification / selection results as input and calculate the probability that the input is associated with a particular predefined intent. In step 224b, the NLU module may process the domain classification / selection results using a meta-intent classifier. The meta-intent classifier may determine a category that describes the user's intent. In a particular embodiment, intents common to multiple domains may be processed by the meta-intent classifier. As an example and not by way of limitation, the meta-intent classifier may be based on a machine learning model that may take the domain classification / selection results as input and calculate the probability that the input is associated with a particular predefined meta-intent. In step 225a, the NLU module 220 may annotate one or more slots associated with the user request using a slot tagger. In a particular embodiment, the slot tagger may annotate one or more slots for n-grams requested by the user. In step 225b, the NLU module 220 may annotate one or more slots for the classification results from the meta-intent classifier using a meta-slot tagger. In a particular embodiment, the meta-slot tagger may tag common slots, such as references to items (e.g., the first one), the type of slot, the value of the slot, and the like. As an example and not by way of limitation, a user request may include "change 500 dollars in my account to Japanese yen". The intent classifier may take as input the user request and formulate it into a vector. The intent classifier may then calculate the probability that the user request is associated with different predefined intents based on a vector comparison between the vector representing the user request and the vectors representing different predefined intents. In a similar manner, the slot tagger may take as input the user request and formulate each word into a vector.The intent classifier can then calculate the probability of each word being associated with different predefined slots based on a vector comparison between the vector representing the word and the vector representing the different predefined slots. The user's intent can be classified as "changing money". The slots requested by the user can include "500", "dollars", "account", and "Japanese yen". The user's meta-intent can be classified as "financial service". The meta-slot can include "finance".

[0142] In certain embodiments, the NLU module 220 may improve the domain classification / selection of content objects by extracting semantic information from the semantic information aggregator 230. In certain embodiments, the semantic information aggregator 230 may aggregate semantic information in the following manner. The semantic information aggregator 230 may first retrieve information from the user context engine 225. In certain embodiments, the user context engine 225 may include an offline aggregator 226 and an online inference service 227. The offline aggregator 226 may process a plurality of data associated with the user collected from a previous time window. As an example and not by way of limitation, the data may include dynamic message posts / comments, interactions with dynamic message posts / comments, Instagram posts / comments, search history, etc. collected from a previous 90-day window. The processing results may be stored in the user context engine 225 as part of the user profile. The online inference service 227 may analyze the session data associated with the user received by the assistant system 140 at the current time. The analysis results may also be stored in the user context engine 225 as part of the user profile. In certain embodiments, both the offline aggregator 226 and the online inference service 227 may extract personalized features from the plurality of data. The extracted personalized features can be used by other modules of the assistant system 140 to better understand the user input. In a particular embodiment, the semantic information aggregator 230 can then process the information retrieved from the user context engine 225, i.e., the user profile, in the following steps. In step 231, the semantic information aggregator 230 can process the information retrieved from the user context engine 225 based on natural language processing (NLP). In a particular embodiment, the semantic information aggregator 230 can: tokenize the text by text normalization, extract syntax features from the text, and extract semantic features from the text based on NLP. The semantic information aggregator 230 can also extract features from context information, which is accessed from the conversation history between the user and the assistant system 140. The semantic information aggregator 230 can also perform global word embedding, domain-specific embedding, and / or dynamic embedding based on the context information. In step 232, the processing results can be annotated with entities by the entity tagger. In step 233, based on the annotations, the semantic information aggregator 230 can generate a dictionary for the retrieved information. In certain embodiments, the dictionary may include a global dictionary feature that may be dynamically updated offline. At step 234, the semantic information aggregator 230 may rank the entities tagged by the entity tagger. In certain embodiments, the semantic information aggregator 230 may communicate with different graphs 330 (Graphs) including social graphs, knowledge graphs, and concept graphs to extract ontology data related to the information retrieved from the user context engine 225. In certain embodiments, the semantic information aggregator 230 may aggregate user profiles, ranked entities, and information from graphs 330.The semantic information aggregator 230 may then send the aggregated information to the NLU module 220 to facilitate domain classification / selection.

[0143] In certain embodiments, the output of the NLU module 220 may be sent to a co-reference module 315 to interpret the references of the content object associated with the user request. In certain embodiments, the co-reference module 315 may be used to identify the item to which the user request refers. The co-reference module 315 may include reference creation 316 and reference resolution 317. In certain embodiments, the reference creation 316 may create references for the entities determined by the NLU module 220. The reference resolution 317 may accurately resolve these references. As an example and not by way of limitation, the user request may include "find me the nearest Walmart and direct me there". The co-reference module 315 may interpret "there" as "the nearest Walmart". In certain embodiments, the co-reference module 315 may access the user context engine 225 and the dialog engine 235, if necessary, to interpret the references with improved accuracy.

[0144] In a particular embodiment, the identified domains, intents, meta-intents, slots and meta-slots, and resolved references may be sent to the entity resolution module 240 to resolve related entities. The entity resolution module 240 may perform general and domain-specific entity resolution. In a particular embodiment, the entity resolution module 240 may include a domain entity resolution 241 and a general entity resolution 242. The domain entity resolution 241 may resolve entities by categorizing slots and meta-slots into different domains. In a particular embodiment, entities may be resolved based on ontology data extracted from a graph 330. The ontology data may include structural relationships between different slots / meta-slots and domains. The ontology may also include information on how slots / meta-slots may be grouped, related, and subdivided according to similarities and differences within a hierarchy that includes domains at a higher level. The general entity resolution 242 may resolve entities by categorizing slots and meta-slots into different general topics. In a particular embodiment, resolution may also be based on ontology data extracted from a graph 330. The ontology data may include structural relationships between different slots / meta-slots and general topics. The ontology may also include information about how slots / meta-slots may be grouped, related, and subdivided according to similarities and differences within a hierarchy that includes topics at a higher level. As an example and not by way of limitation, in response to an input of a query about the advantages of a Tesla car, the general entity resolution 242 may resolve a Tesla car to a vehicle, and the domain entity resolution 241 may resolve a Tesla car to an electric car.

[0145] In a particular embodiment, the output of the entity resolution module 240 may be sent to the conversation engine 235 to forward the conversation flow with the user. The conversation engine 235 may include a conversation intent resolution 236 and a conversation state updater / sorter 237. In a particular embodiment, the conversation intent resolution 236 may resolve the user intent associated with the current conversation session based on the conversation history between the user and the assistant system 140. The conversation intent resolution 236 may map the intent determined by the NLU module 220 to different conversation intents. The conversation intent resolution 236 may also sort the conversation intent based on the signals from the NLU module 220, the entity resolution module 240, and the conversation history between the user and the assistant system 140. In a particular embodiment, the conversation state updater / sorter 237 may update / sort the conversation state of the current conversation session. As an example and not as a limitation, if the conversation session ends, the conversation state updater / sorter 237 may update the conversation state to "completed". As another example and not as a limitation, the conversation state updater / sorter 237 may sort the conversation state based on the priority associated with the conversation state.

[0146] In certain embodiments, the dialog engine 235 may communicate with the task completion module 335 regarding the dialog intent and the associated content object. In certain embodiments, the task completion module 335 may rank different dialog hypotheses for different dialog intents. The task completion module 335 may include an action selection component 336. In certain embodiments, the dialog engine 235 may additionally check against the dialog strategy 320 regarding the dialog state. In certain embodiments, the dialog strategy 320 may include a data structure describing the action execution plan of the agent 340. The agent 340 may select from the registered content providers to complete the action. The data structure may be constructed by the dialog engine 235 based on the intent and one or more slots associated with the intent. The dialog strategy 320 may also include multiple goals that are interrelated by logical operators. In certain embodiments, the goal may be the output result of a part of the dialog strategy, and it may be constructed by the dialog engine 235. The goal may be represented by an identifier (e.g., a string) having one or more named parameters that parameterize the goal. As an example and not as a limitation, a goal and its associated goal parameter can be represented as {confirm_artist, parameter: {artist: "Madonna"}}. In a particular embodiment, the dialogue strategy can be represented based on a tree structure, in which the goal is mapped to the leaves. In a particular embodiment, the dialogue engine 235 can execute the dialogue strategy 320 to determine the next action to be performed. The dialogue strategy 320 may include a general strategy 321 and a domain-specific strategy 322, both of which can guide how to select the next system action based on the dialogue state. In a particular embodiment, the task completion module 335 can communicate with the dialogue strategy 320 to obtain guidance for the next system action. In a particular embodiment, the action selection component 336 can therefore select an action based on the dialogue intention, the associated content object, and the guidance from the dialogue strategy 320.

[0147] In a particular embodiment, the output of the task completion module 335 may be sent to the CU writer 270. In an alternative embodiment, the selected action may require one or more agents 340 to participate. Therefore, the task completion module 335 may notify the agent 340 of the selected action. At the same time, the conversation engine 235 may receive an instruction to update the conversation state. As an example and not as a limitation, the update may include waiting for the response of the agent. In a particular embodiment, the CU writer 270 may generate communication content for the user using NLG 271 based on the output of the task completion module 335. In a particular embodiment, the NLG 271 may use different language models and / or language templates to generate natural language output. The generation of natural language output may be application-specific. The generation of natural language output may also be personalized for each user. The CU writer 270 may also use the UI payload generator 272 to determine the modality of the generated communication content. Since the generated communication content may be considered as a response to a user request, the CU writer 270 may additionally use a response sorter 273 to sort the generated communication content. As an example and not as a limitation, the sorting may indicate the priority of the response.

[0148] In a particular embodiment, the output of the CU writer 270 may be sent to a response manager 325. The response manager 325 may perform different tasks, including storing / updating the dialog state 326 retrieved from the data storage 310 and generating a response 327. In a particular embodiment, the output of the CU writer 270 may include one or more of a natural language string, a voice, or an action with parameters. Therefore, the response manager 325 may determine what task to perform based on the output of the CU writer 270. In a particular embodiment, the generated response and communication content may be sent to the assistant xbot 215. In an alternative embodiment, if the modality of the determined communication content is audio, the output of the CU writer 270 may be sent to the TTS module 275 in addition. Then, the voice generated by the TTS module 275 and the response generated by the response manager 325 may be sent to the assistant xbot 215. Personalized gesture recognition for user interaction with assistant system

[0149] In a particular embodiment, the assistant system 140 can train a personalized gesture classification model. The training may include receiving a plurality of input tuples, each of which includes a gesture input and a voice input, determining a plurality of intents based on the voice input by the natural language understanding (NLU) module 220, accessing a general gesture classification model, associating intents with their respective gesture inputs, and training the personalized gesture classification model based on the association between the intent and the gesture input and the general gesture classification model. In a particular embodiment, the general gesture classification model can be generated by crowdsourcing about a large number of standard gestures. However, the general gesture classification model may not be able to determine the correct intent of a particular user's gesture, especially if the user is using a non-standard gesture or a gesture that is commonly used for different intents. Therefore, the training of the personalized gesture classification model can additionally utilize information cues from the user's own input in addition to the general gesture classification model, and the user's own input includes voice and gestures. In addition to using voice input alone to assist the training of the personalized gesture classification model, the assistant system 140 can alternatively train the personalized gesture classification model by jointly using voice input and gesture input. Once the personalized gesture classification model is trained, the assistant system 140 can use it in the future to determine the user's intent corresponding to his / her own gesture. Although this disclosure describes training a particular gesture classification model in a particular manner via a particular system, this disclosure contemplates describing training any suitable gesture classification model in any suitable manner via any suitable system.

[0150] Figure 4A-4C An example gesture is shown. In certain embodiments, assistant system 140 may access a plurality of input tuples associated with the first user from a data store. Each input tuple may include a gesture input and a corresponding voice input. The gesture input and the voice input may be generated by, for example, an assistant system 140 (not shown). Figure 4A-4C The input may be captured by an assistant device associated with the user, a virtual reality (VR) headset, or augmented reality (AR) smart glasses. By way of example and not limitation, the input may include: Figure 4A The “no” gesture shown (i.e., the user holds his hand up in a substantially vertical position, palm facing outward, with fingers fully extended and substantially together) and the voice input “stop”. As another example and not by way of limitation, the input may include the following: Figure 4B The “swipe” gesture shown (i.e., the user's index and middle fingers are extended and the hand moves in an arc from left to right) and the voice input “next”. As yet another example and not by way of limitation, the input may include the following: Figure 4CA “pinch” gesture (i.e., a user's thumb and index finger moving toward each other from an extended position relative to each other) and a voice input “pick up” are shown. These input tuples of gesture input and corresponding voice input may be generated based on one or more of crowdsourcing (i.e., requesting the general population to provide gesture input and corresponding voice input) or historical user interactions with assistant system 140. Although this disclosure describes accessing particular gestures and voice input in particular manners, this disclosure contemplates accessing any suitable gestures and voice input in any suitable manner.

[0151] In a particular embodiment, the assistant system 140 may then determine multiple intentions corresponding to the multiple voice inputs, respectively, through the natural language understanding (NLU) module 220. As an example and not as a limitation, the NLU module 220 may determine the intention of the aforementioned voice input "stop" as stopping a task performed by the assistant system 140, such as stopping playing music. As another example and not as a limitation, the NLU module 220 may determine the intention of the aforementioned voice input "next" as selecting the next content object suggested by the assistant system 140, such as the next image. As another example and not as a limitation, the NLU module 220 may determine the intention of the aforementioned voice input "pick up" as picking up an object in a VR game or in an AR view of a room. The assistant system 140 may then generate multiple feature representations for the multiple gesture inputs based on one or more machine learning models. The feature representation may be a piece of information related to solving a computing task associated with an application (e.g., gesture recognition). As an example and not as a limitation, a feature representation of a gesture may be generated based on a convolutional neural network, shape and texture features from 2D images and videos associated with the gesture, or a depth image associated with the gesture. In certain embodiments, assistant system 140 may then determine a plurality of gesture identifiers for the plurality of gesture inputs, respectively, based on the respective feature representations of the plurality of gesture inputs. Assistant system 140 may then associate the plurality of intents with the plurality of gesture identifiers, respectively. In certain embodiments, assistant system 140 may further train a personalized gesture classification model for the first user based on the plurality of feature representations of their respective gesture inputs and the association between the plurality of intents and their respective gesture identifiers.

[0152] Figure 5An example workflow for processing user input including gesture input is shown. Gesture input can be input that expresses an idea or meaning based on the movement of a part of the body (especially a hand or head). The gesture input can be in the form of image information, video information, motion information, or any combination thereof. In a particular embodiment, the assistant system 140 can receive user input 505 from the user from the client system 130. The assistant system 140 can send the user input 505 to the intent understanding module 510. The intent understanding module 510 can include a natural language understanding (NLU) module 220 and a gesture classification model 515. The gesture classification model 515 can be a machine learning model trained offline for identifying different categories of gestures performed by a user. As an example and not as a limitation, the gesture classification model 515 can be based on one or more of a convolutional neural network, a tensor flow, or a hidden Markov model. Based on the user input 505, the intent understanding module 510 can use different components to determine the intent 520 associated with the user input 505. If the user input 505 includes text input or voice input, the intent understanding module 510 can use the NLU module 220 to determine the intent 520. If the user input 505 includes gesture input, the intent understanding module 510 can use the gesture classification model 515 to determine the intent 520. The determined intent 520 can be associated with a confidence score that indicates the confidence level of the assistant system 140 in determining the user's intent 520. In a particular embodiment, the confidence score can be determined based on how closely the user's text input or voice input matches the known input for a given intent 520. As an example and not by way of limitation, the closeness can be based on string similarity between the text input and the known input (text). The assistant system 140 can then send the intent 520 and its confidence score to the dialogue engine 235. If the confidence score is above a threshold score, the dialogue engine 235 can determine one or more tasks 525 corresponding to the intent 520 and send them to the first party agent 250 or the third party agent 255 to perform the task 525. The execution result 530 may be sent to the CU writer 270. If the confidence score is below the threshold score, the dialog engine 235 may determine one or more suggested inputs 535 for the user, which may help the assistant system 140 determine the user's intent with a higher confidence score. The dialog engine 235 may send the suggested inputs 535 to the CU writer 270. Based on the execution result 530 or the suggested inputs 535, the CU writer 270 may generate a response 540. The response 540 may have different modalities, including one or more of text, images, videos, or gesture animations. The CU writer 270 may further send the response 540 to the client system 130.Although this disclosure describes processing user input in a particular manner via a particular system, this disclosure contemplates describing processing any suitable user input in any suitable manner via any suitable system.

[0153] In certain embodiments, the training of the personalized gesture classification model can be based on a general gesture classification model. The assistant system 140 can access a general gesture classification model corresponding to a general user group from a data storage device. Therefore, the training of the personalized gesture classification model can also be based on the general gesture classification model. In certain embodiments, the general user group can correspond to multiple second users. As an example and not as a limitation, the second user can be from a general user group (e.g., people from all over the world) or a limited user group (e.g., people in the United States). In certain embodiments, the general gesture classification model can be trained based on multiple gesture inputs from the general user group. As an example and not as a limitation, the multiple gesture inputs can include standard gestures performed by users of the general population. Therefore, if the user performs one of these standard gestures, the general gesture classification model can recognize it and determine its corresponding intention 520. The general gesture classification model can be the basis of the personalized gesture classification model, which can have the ability to recognize user gestures that appear to be consistent with the corresponding standard gestures. Although the present disclosure describes a specific general model in a specific manner, the present disclosure contemplates describing any suitable general model in any suitable manner.

[0154] In a particular embodiment, the assistant system 140 may first use a general gesture classification model as a default model to determine the category of the received gesture input and its corresponding intent 520. If the assistant system 140 successfully determines the intent 520, it may perform the corresponding task 525. If the assistant system 140 cannot determine the user's intent 520, the assistant system 140 may request the user to perform the gesture again and simultaneously tell the assistant system 140 about his / her intent 520. Thereafter, the assistant system 140 may begin the training process of a personalized gesture classification model while collecting training data from the user's voice and gestures. Although the present disclosure describes the use of a particular gesture classification model via a particular system in a particular manner, the present disclosure contemplates the use of any suitable gesture classification model via any suitable system in any suitable manner.

[0155] In a particular embodiment, the assistant system 140 may train a personalized gesture classification model in the following manner. The assistant system may first generate multiple text inputs for multiple voice inputs respectively through one or more automatic speech recognition (ASR) modules 210. Therefore, determining multiple intentions 520 corresponding to the multiple voice inputs respectively may be based on multiple text inputs of the corresponding voice inputs. In a particular embodiment, the assistant system may classify the text input using a natural language understanding (NLU) module 220 to obtain the corresponding intention 520. The assistant system 140 may then generate a feature representation for the gesture input based on one or more machine learning models. As an example and not as a limitation, the gesture input may include an image, video, motion data, skeleton data, thermal data, etc. containing a gesture. In a particular embodiment, one or more machine learning models may be based on one or more of a neural network model or a long short-term memory (LSTM) model. The assistant system 140 may additionally assign different gesture identifiers to the feature representation corresponding to the gesture input. In a particular embodiment, the assistant system 140 may then associate the intention 520 with the gesture identifier. The association between multiple intentions 520 and their respective gesture identifiers may be used to train a personalized gesture classification model. Training a personalized gesture classification model based on both the user's voice input and gesture input may be an effective solution to the technical challenge of enabling the personalized gesture classification model to recognize user intent 520 from the user's personal gestures, because the user intent 520 can be learned based on the voice input, and an association between such intent 520 and gesture input can be generated for further utilization during the training of the personalized gesture classification model. In a particular embodiment, the personalized gesture classification model can be based on a convolutional neural network or a recurrent neural network. Repeated observations of coincidence speech intents and gestures (whether intentionally generated by the first user or subconsciously) can lead to more and more training data. At the same time, as more and more training data is used, the personalized gesture classification model can be gradually optimized. As a result, the personalized gesture classification model can discover new gestures from the user and associate them with meaning (i.e., intent 520). Training a personalized gesture classification model based on both voice input and gesture input may be more effective in classifying gestures for a particular user. As an example and not by way of limitation, a user may perform the following steps: Figure 4A The "no" gesture shown, but the intention 520 may actually be "next" (which is usually the same as Figure 4B). Accordingly, using a standard gesture classification model may determine the wrong intent. In contrast, because the data used by the personalized gesture classification model is based on the user's personal way of performing such gestures, when the user performs such Figure 4A As another example and not by way of limitation, a user may perform the following: Figure 4B , but the intention 520 may not be "next", but actually "rewind a video". Similarly, when the user performs the following Figure 4B When performing the “swipe” gesture shown, the standard gesture classification model may determine an incorrect intent, but the personalized gesture classification model may effectively determine the intent 520 to “rewind the video.” As another example and not by way of limitation, a user may perform the following Figure 4C 520 may not be a "pinch" gesture, but may actually be a "zoom out". Again, when the user performs a Figure 4C When performing the “pinch in” gesture shown, using the standard gesture classification model may determine an incorrect intent, while the personalized gesture classification model may effectively determine the intent 520 to “zoom out.” Although this disclosure describes training a particular gesture classification model in a particular manner, this disclosure contemplates describing training any suitable gesture classification model in any suitable manner.

[0156] In certain embodiments, the assistant system may generate a feature representation for a gesture input by considering various information. In certain embodiments, generating each feature representation for each gesture input may include separating the gesture input into one or more components and modeling the one or more components into the feature representation of the gesture input. These components may be considered partial gestures. In certain embodiments, generating each feature representation for each gesture input may include determining instantaneous information associated with the gesture input and modeling the instantaneous information into the feature representation of the gesture input. By way of example and not limitation, a user may perform the following operations: Figure 4Arolls his / her eyes before the “no” gesture shown. Thus, assistant system 140 can consider the transient information of the eye roll that occurs before the “no” gesture when generating the feature representation to help determine the intent of rejection 520. Modeling the components of the gesture input and the transient information associated with the gesture input into the feature representation may be an effective solution to the technical challenge of generating reliable feature representations for gesture input, because separating the gesture input into the component parts can help distinguish the gesture from another gesture, and the transient information can carry contextual information indicating the user's intent 520, which may be useful for gesture classification. Although this disclosure describes generating particular feature representations in a particular manner, this disclosure contemplates describing generating any suitable feature representations in any suitable manner.

[0157] In a particular embodiment, the assistant system 140 can train a personalized gesture classification model by jointly utilizing voice input and gesture input. The assistant system 140 can extract text input from the voice input through the ASR module 210. The assistant system 140 can then generate feature representations for both text input and gesture input. After generating the feature representation, the assistant system 140 can determine the weight of the text input and the weight of the gesture input respectively. In a particular embodiment, these inputs and their respective weights can be further fed into a deep neural network framework to train a personalized gesture classification model. In a particular embodiment, the assistant system 140 can also use user feedback when training a personalized gesture classification model. The dialogue engine 235 can seek user feedback with the help of the suggested input 535, and additional training data can also be obtained through the suggested input 535. More specifically, the dialogue engine 235 can seek confirmation or disambiguation of gestures with low confidence. The dialogue engine 235 can use text or voice to request the user to confirm or disambiguate gestures that may be included in the suggested input 535. Thus, a confirmation or selection from the user can be used as an additional signal for training a personalized gesture classification model.Although this disclosure describes training a particular gesture classification model in a particular manner, this disclosure contemplates describing training any suitable gesture classification model in any suitable manner.

[0158] In certain embodiments, the personalized gesture classification model may automatically associate gesture input with intent 520 during runtime. In certain embodiments, the assistant system 140 may receive a new gesture input from a first user from a client system 130 associated with the first user. The assistant system 140 may then determine an intent 520 corresponding to the new gesture input for the new gesture input based on the personalized gesture classification model. In certain embodiments, the assistant system 140 may further perform one or more tasks 525 based on the determined intent 520. Thus, the assistant system 140 may have a technical advantage of enriching the user's experience of the assistant system 140 by enabling the user to interact with the assistant system 140 with gestures in addition to traditional inputs (e.g., text and voice), for which the assistant system 140 may accurately recognize the user's gesture and perform a task 525 corresponding to the recognized gesture. Although the present disclosure describes the use of a particular gesture classification model in a particular manner, the present disclosure contemplates the use of any suitable gesture classification model in any suitable manner.

[0159] Figure 6 An example method 600 for training a personalized gesture classification model is shown. The method may begin at step 610, where assistant system 140 may access multiple input tuples associated with a first user from a data storage, where each input tuple includes a gesture input and a corresponding voice input. At step 620, assistant system 140 may determine multiple intents 520 corresponding to the multiple voice inputs, respectively, through natural language understanding (NLU) module 220. At step 630, assistant system 140 may generate multiple feature representations for the multiple gesture inputs based on one or more machine learning models. At step 640, assistant system 140 may determine multiple gesture identifiers for the multiple gesture inputs, respectively, based on the feature representations of the multiple gesture inputs. At step 650, assistant system 140 may associate the multiple intents 520 with the multiple gesture identifiers, respectively. At step 660, assistant system 140 may train a personalized gesture classification model for the first user based on the multiple feature representations of their respective gesture inputs and the associations between the multiple intents 520 and their respective gesture identifiers. Where appropriate, specific embodiments may be repeated. Figure 6 Although the present disclosure will Figure 6 The specific steps of the method are described and shown as occurring in a particular order, but the present disclosure contemplates Figure 6 Any suitable steps of the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates the method including Figure 6The example method for training a personalized gesture classification model includes specific steps of the method, but the present disclosure contemplates any suitable method for training a personalized gesture classification model including any suitable steps, where appropriate, the method may include Figure 6 All steps, some steps or may not include all steps of the method Figure 6 Furthermore, although the present disclosure describes and illustrates the steps of performing Figure 6 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Figure 6 Any suitable combination of any suitable components, devices, or systems of any suitable steps of the method.

[0160] Social Graph

[0161] Figure 7 An example social graph 700 is shown. In certain embodiments, social networking system 160 may store one or more social graphs 700 in one or more data stores. In certain embodiments, social graph 700 may include a plurality of nodes, which may include a plurality of user nodes 702 or a plurality of concept nodes 704, and a plurality of edges 706 connecting the nodes. Each node may be associated with a unique entity (i.e., a user or a concept), and each entity may have a unique identifier (ID), such as a unique number or username. For teaching purposes, the two-dimensional visual map representation is shown. Figure 7 Example social graph 700 is shown. In certain embodiments, social networking system 160, client system 130, assistant system 140, or third-party system 170 may access social graph 700 and related social graph information for use in appropriate applications. The nodes and edges of social graph 700 may be stored as data objects, for example, in a data store (e.g., a social graph database). Such a data store may include one or more searchable or queryable indexes of the nodes or edges of social graph 700.

[0162] In certain embodiments, user nodes 702 may correspond to users of social networking system 160 or assistant system 140. As an example and not by way of limitation, a user may be an individual (human user), an entity (e.g., a business, company, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with or through social networking system 160 or assistant system 140. In certain embodiments, when a user registers an account with social networking system 160, social networking system 160 may create a user node 702 corresponding to the user and store the user node 702 in one or more data stores. Users and user nodes 702 described herein may refer to registered users and user nodes 702 associated with registered users, where appropriate. Additionally or alternatively, users and user nodes 702 described herein may refer to users who are not registered with social networking system 160, where appropriate. In certain embodiments, user nodes 702 may be associated with information provided by a user or information collected by various systems, including social networking system 160. As an example and not by way of limitation, a user may provide his or her name, profile picture, contact information, date of birth, gender, marital status, family status, occupation, educational background, preferences, interests, or other demographic information. In certain embodiments, user node 702 may be associated with one or more data objects corresponding to information associated with the user. In certain embodiments, user node 702 may correspond to one or more web interfaces.

[0163] In certain embodiments, concept nodes 704 may correspond to concepts. By way of example and not limitation, a concept may correspond to a place (such as, for example, a movie theater, a restaurant, a landmark, or a city); a website (such as, for example, a website associated with social networking system 160 or a third-party website associated with a web application server); an entity (such as, for example, a person, a business, a group, a sports team, or a celebrity); a resource (such as, for example, an audio file, a video file, a digital photo, a text file, a structured document, or an application), which may be located within social networking system 160 or on an external server (e.g., a web application server); real estate or intellectual property (such as, for example, a sculpture, a painting, a movie, a game, a song, an idea, a photo, or a written work); a game; an activity; an idea or theory; another suitable concept; or two or more such concepts. Concept nodes 704 may be associated with information about the concept provided by a user or information collected by various systems, including social networking system 160 and assistant system 140. By way of example and not limitation, information for a concept may include a name or title; one or more images (e.g., an image of a book cover); a location (e.g., an address or geographic location); a website (which may be associated with a URL); contact information (e.g., a phone number or email address); other suitable concept information; or any suitable combination of such information. In particular embodiments, a concept node 704 may be associated with one or more data objects that correspond to the information associated with the concept node 704. In particular embodiments, a concept node 704 may correspond to one or more web interfaces.

[0164] In certain embodiments, a node in social graph 700 may represent a web interface (which may be referred to as a "profile interface") or be represented by a web interface. The profile interface may be hosted by social networking system 160 or assistant system 140 or accessible to social networking system 160 or assistant system 140. The profile interface may also be hosted on a third-party website associated with third-party system 170. As an example and not as a limitation, a profile interface corresponding to a particular external web interface may be a particular external web interface, and the profile interface may correspond to a particular concept node 704. The profile interface may be viewable by all other users or a selected subset of other users. As an example and not as a limitation, user node 702 may have a corresponding user profile interface, where the corresponding user may add content, make statements, or otherwise express him or herself. As another example and not as a limitation, concept node 704 may have a corresponding concept profile interface, where one or more users may add content, make statements, or express themselves, particularly with respect to the concept corresponding to concept node 704.

[0165] In certain embodiments, concept node 704 may represent a third-party web interface or resource hosted by third-party system 170. The third-party web interface or resource may include content representing an action or activity, selectable icons or other icons or other interactive objects (which may be implemented, for example, with JavaScript, AJAX, or PHP code), and other elements. As an example and not by way of limitation, the third-party web interface may include selectable icons such as “Like,” “Check-in,” “Eat,” “Recommend,” or other suitable actions or activities. A user viewing the third-party web interface may perform an action by selecting one of the icons (e.g., “Check-in”), causing client system 130 to send a message to social-networking system 160 indicating the user's action. In response to the message, social-networking system 160 may create an edge (e.g., a check-in-type edge) between user node 702 corresponding to the user and concept node 704 corresponding to the third-party web interface or resource, and store edge 706 in one or more data stores.

[0166] In particular embodiments, a pair of nodes in the social graph 700 may be connected to each other via one or more edges 706. An edge 706 connecting a pair of nodes may represent a relationship between the pair of nodes. In particular embodiments, an edge 706 may include or represent one or more data objects or attributes corresponding to the relationship between a pair of nodes. As an example and not by way of limitation, a first user may indicate that a second user is a "friend" of the first user. In response to the indication, the social networking system 160 may send a "friend request" to the second user. If the second user confirms the "friend request", the social networking system 160 may create an edge 706 in the social graph 700 connecting the first user's user node 702 to the second user's user node 702, and store the edge 706 as social graph information in one or more data stores 164. Figure 7In the example of , the social graph 700 includes an edge 706 indicating a friend relationship between the user nodes 702 of user "A" and user "B", and an edge indicating a friend relationship between the user nodes 702 of user "C" and user "B". Although the present disclosure describes or illustrates a particular edge 706 with a particular attribute connecting a particular user node 702, the present disclosure contemplates any suitable edge 706 with any suitable attribute connecting the user nodes 702. As an example and not by way of limitation, the edge 706 may represent a friendship, a family relationship, a business or employment relationship, a fan relationship (including, for example, likes, etc.), a follower relationship, a visitor relationship (including, for example, visits, views, check-ins, shares, etc.), a subscriber relationship, a superior / subordinate relationship, a reciprocal relationship, a non-reciprocal relationship, another suitable type of relationship, or two or more such relationships. In addition, although the present disclosure generally describes nodes as being connected, the present disclosure also describes users or concepts as being connected. Herein, references to connected users or concepts may refer to nodes corresponding to those users or concepts connected by one or more edges 706 in social graph 700 , where appropriate.

[0167] In certain embodiments, an edge 706 between a user node 702 and a concept node 704 may represent a particular action or activity performed by a user associated with the user node 702 toward a concept associated with the concept node 704. By way of example and not limitation, Figure 7 As shown, a user can "Like," "Attend," "Play," "Listen to," "Cook," "Work at," or "Watch" a concept, each of which can correspond to an edge type or subtype. The concept profile interface corresponding to concept node 704 may include, for example, a selectable "check-in" icon (such as, for example, a clickable "check-in" icon) or a selectable "add to favorites" icon. Similarly, after the user clicks on these icons, social networking system 160 may create a "Favorites" edge or a "Check-in" edge in response to a user action corresponding to the corresponding action. As another example and not by way of limitation, a user (user "C") may use a particular application (Spotify, which is an online music application) to listen to a particular song ("Imagine"). In this case, social networking system 160 may create a "Listen" edge 706 and a "Use" edge (such as, for example, a clickable "Check-in" icon) between user node 702 corresponding to the user and concept nodes 704 corresponding to the song and application. Figure 7 ), to indicate that the user listened to the song and used the application. In addition, social networking system 160 can create a "play" edge 706 (as shown in FIG. 7 ) between concept nodes 704 corresponding to the song and the application. Figure 7as shown), to indicate that a particular song is being played by a particular application. In this case, the "played" edge 706 corresponds to an action performed by an external application (Spotify) on an external audio file (the song "Imagine"). Although this disclosure describes a particular edge 706 having particular attributes that connects user node 702 and concept node 704, this disclosure contemplates any suitable edge 706 having any suitable attributes that connects user node 702 and concept node 704. Additionally, although this disclosure describes an edge between user node 702 and concept node 704 that represents a single relationship, this disclosure contemplates an edge between user node 702 and concept node 704 that represents one or more relationships. By way of example and not limitation, edge 706 can represent that the user likes and uses a particular concept. Alternatively, another edge 706 can represent each type of relationship (or multiple single relationships) between user node 702 and concept node 704 (as shown between user node 702 of user "E" and concept node 704 of "Spotify"). Figure 7 as shown, between user node 702 of user "E" and concept node 704 of "Spotify").

[0168] In a particular embodiment, the social networking system 160 can create an edge 706 between a user node 702 and a concept node 704 in the social graph 700. By way of example and not limitation, a user who views a concept profile interface (such as, for example, by using a web browser or a dedicated application hosted by the user's client system 130) can indicate that he or she likes the concept represented by concept node 704 by clicking or selecting a "like" icon, which can cause the user's client system 130 to send a message to the social networking system 160 indicating that the user likes the concept associated with the concept profile interface. In response to the message, the social networking system 160 can create an edge 706 between the user node 702 associated with the user and concept node 704, as shown by the "like" edge 706 between the user node and concept node 704. In a particular embodiment, the social networking system 160 can store the edge 706 in one or more data stores. In a particular embodiment, the edge 706 can be automatically formed by the social networking system 160 in response to a particular user action. By way of example and not limitation, if a first user uploads a picture, watches a movie, or listens to a song, an edge 706 can be formed between the user node 702 corresponding to the first user and the concept nodes 704 corresponding to those concepts. Although this disclosure describes forming a particular edge 706 in a particular manner, this disclosure contemplates forming any suitable edge 706 in any suitable manner.

[0169] Vector Spaces and Embeddings

[0170] Figure 8An example view of vector space 800 is shown. In certain embodiments, objects or n-grams may be represented in a d-dimensional vector space, where d represents any suitable number of dimensions. Although vector space 800 is shown as a three-dimensional space, this is for illustrative purposes only, as vector space 800 may have any suitable dimensions. In certain embodiments, n-grams may be represented in vector space 800 as vectors, which are referred to as term embeddings. Each vector may include coordinates corresponding to a particular point in vector space 800 (i.e., the end point of the vector). By way of example and not limitation, Figure 8 As shown, vectors 810, 820, and 830 can be represented as points in vector space 800. n-grams can be mapped to corresponding vector representations. By way of example and not limitation, by applying a function defined by a dictionary n-gramt1 and n-gramt2 can be mapped to vectors in vector space 800 respectively. and Make and As another example and not by way of limitation, a dictionary trained to map text to a vector representation may be utilized, or such a dictionary itself may be generated by training. As another example and not by way of limitation, a model (e.g., Word2vec) may be used to map n-grams to vector representations in vector space 800. In certain embodiments, n-grams may be mapped to vector representations in vector space 800 using a machine learning model (e.g., a neural network). The machine learning model may have been trained using a sequence of training data (e.g., a corpus of multiple objects each including n-grams).

[0171] In certain embodiments, an object may be represented in vector space 800 as a vector, which is referred to as a feature vector or object embedding. By way of example and not limitation, by applying the function Objects e1 and e2 can be mapped to vectors in vector space 800 respectively. and Make and In certain embodiments, objects may be mapped to vectors based on one or more properties, attributes, or characteristics of the object, the object's relationship to other objects, or any other suitable information associated with the object. By way of example and not limitation, the function Objects can be mapped to vectors by feature extraction, which can start from an initial measurement data set and construct derived values ​​(e.g., features). As an example and not by way of limitation, objects including videos or images can be mapped to vectors by using algorithms to detect or isolate various desired parts or shapes of objects. The features used to calculate the vectors can be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transforms, edge direction, varying intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transforms (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information. As another example and not by way of limitation, objects including audio data can be mapped to vectors based on features (e.g., spectral slope, pitch coefficient, audio spectrum centroid, audio spectrum envelope, Mel-frequency cepstrum, or any other suitable information). In a particular embodiment, when an object has data that is too large to be effectively processed or includes redundant data, the function The transformed reduced feature set (e.g., feature selection) can be used to map objects to vectors. In certain embodiments, the function An object e can be mapped to a vector based on one or more n-grams associated with the object e. Although this disclosure describes representing n-grams or objects in a vector space in a particular manner, this disclosure contemplates representing n-grams or objects in a vector space in any suitable manner.

[0172] In certain embodiments, social networking system 160 may calculate a similarity metric for the vectors in vector space 800. The similarity metric may be cosine similarity, Minkowski distance, Mahalanobis distance, Jaccard similarity coefficient, or any suitable similarity metric. By way of example and not limitation, and The similarity measure can be cosine similarity As another example and not by way of limitation, and The similarity measure can be the Euclidean distance The similarity measure of two vectors may represent how similar two objects or n-grams corresponding to the two vectors, respectively, are to each other, as measured by the distance between the two vectors in vector space 800. As an example and not by way of limitation, vector 810 and vector 820 may correspond to objects that are more similar to each other than objects corresponding to vector 810 and vector 830, based on the distance between the respective vectors. Although this disclosure describes calculating a similarity measure between vectors in a particular manner, this disclosure contemplates calculating a similarity measure between vectors in any suitable manner.

[0173] More information about vector spaces, embeddings, feature vectors, and similarity measures can be found in U.S. patent application Ser. No. 14 / 949,436, filed on Nov. 23, 2015, U.S. patent application Ser. No. 15 / 286,315, filed on Oct. 5, 2016, and U.S. patent application Ser. No. 15 / 365,789, filed on Nov. 30, 2016, each of which is incorporated by reference.

[0174] Artificial Neural Networks

[0175] Fig. 9 An example artificial neural network ("ANN") 900 is shown. In particular embodiments, an ANN may refer to a computational model that includes one or more nodes. The example ANN 900 may include an input layer 910, hidden layers 920, 930, 940, and an output layer 950. Each layer of the ANN 900 may include one or more nodes, such as node 905 or node 915. In particular embodiments, each node of the ANN may be connected to another node of the ANN. As an example and not by way of limitation, each node of the input layer 910 may be connected to one or more nodes of the hidden layer 920. In particular embodiments, one or more nodes may be bias nodes (e.g., nodes in a layer that are not connected to any node in the previous layer and do not receive input from it). In particular embodiments, each node in each layer may be connected to one or more nodes of the previous or next layer. Although Fig. 9 A particular ANN having a particular number of layers, a particular number of nodes, and particular connections between nodes is depicted, but the present disclosure contemplates any suitable ANN having any suitable number of layers, any suitable number of nodes, and any suitable connections between nodes. By way of example and not limitation, although Fig. 9 The connection between each node of the input layer 910 and each node of the hidden layer 920 is depicted, but one or more nodes of the input layer 910 may not be connected to one or more nodes of the hidden layer 920.

[0176] In certain embodiments, the ANN may be a feedforward ANN (e.g., an ANN without cycles or loops, where communication between nodes flows in one direction starting from the input layer and proceeding to successive layers). As an example and not by way of limitation, the input to each node of the hidden layer 920 may include the output of one or more nodes of the input layer 910. As another example and not by way of limitation, the input to each node of the output layer 950 may include the output of one or more nodes of the hidden layer 940. In certain embodiments, the ANN may be a deep neural network (e.g., a neural network including at least two hidden layers). In certain embodiments, the ANN may be a deep residual network. The deep residual network may be a feedforward ANN including hidden layers organized into residual blocks. The input to each residual block after the first residual block may be a function of the output of the previous residual block and the input of the previous residual block. As an example and not by way of limitation, the input to residual block N may be F(x)+x, where F(x) may be the output of residual block N-1, and x may be the input to residual block N-1. Although the present disclosure describes a particular ANN, the present disclosure contemplates any suitable ANN.

[0177] In certain embodiments, an activation function may correspond to each node of the ANN. The activation function of a node may define the output of the node for a given input. In certain embodiments, the input of a node may include a set of inputs. As an example and not by way of limitation, the activation function may be an identity function, a binary step function, a logistic function, or any other suitable function. As another example and not by way of limitation, the activation function of node k may be a sigmoid function Hyperbolic tangent function Rectifier F k (s k )=max(0,s k ) or any other suitable function F k (s k ), where s k can be an effective input to node k. In certain embodiments, the inputs to the activation functions corresponding to the nodes can be weighted. Each node can generate an output using a corresponding activation function based on the weighted inputs. In certain embodiments, each connection between nodes can be associated with a weight. As an example and not by way of limitation, a connection 925 between node 905 and node 915 can have a weighting factor of 0.4, which can indicate that the output of node 905 multiplied by 0.4 is used as an input to node 915. As another example and not by way of limitation, the output y of node k is k It can be k =F k (s k ), where F kcan be the activation function corresponding to node k, s k =∑ j (w jk x j ) can be a valid input to node k, x j can be the output of node j connected to node k, and w jk It can be a weighted coefficient between node j and node k. In a specific embodiment, the input of the node of the input layer can be based on a vector representing an object. Although the present disclosure describes specific inputs and outputs of nodes, the present disclosure contemplates any suitable inputs and outputs of nodes. In addition, although the present disclosure can describe specific connections and weights between nodes, the present disclosure contemplates any suitable connections and weights between nodes.

[0178] In a particular embodiment, the ANN may be trained using training data. As an example and not as a limitation, the training data may include the input and expected output of the ANN 900. As another example and not as a limitation, the training data may include vectors, each vector representing a training object and an expected label for each training object. In a particular embodiment, the training ANN may include modifying the weights associated with the connections between the nodes of the ANN by optimizing the objective function. As an example and not as a limitation, a training method (e.g., a conjugate gradient method, a gradient descent method, a stochastic gradient descent) may be used to back propagate the sum of squared errors (e.g., using a cost function that minimizes the sum of squared errors) as distance measurements between each vector representing the training object. In a particular embodiment, the ANN may be trained using a discarding technique. As an example and not as a limitation, one or more nodes may be temporarily ignored during training (e.g., not receiving input and not generating output). For each training object, one or more nodes of the ANN may have a certain probability of being ignored. The nodes ignored for a particular training object may be different from the nodes ignored for other training objects (e.g., nodes may be temporarily ignored on an object-by-object basis). Although this disclosure describes training the ANN in a particular manner, this disclosure contemplates training the ANN in any suitable manner.

[0179] privacy

[0180] In a particular embodiment, one or more objects (e.g., content or other types of objects) of a computing system may be associated with one or more privacy settings. One or more objects may be stored on or otherwise associated with any suitable computing system or application, such as, for example, a social networking system 160, a client system 130, an assistant system 140, a third-party system 170, a social networking application, an assistant application, a messaging application, a photo sharing application, or any other suitable computing system or application. Although the examples discussed herein are in the context of an online social network, these privacy settings may be applied to any other suitable computing system. The privacy settings (or "access settings") of an object may be stored in any suitable manner, such as, for example, associated with an object, indexed on an authorization server, in another suitable manner, or any suitable combination thereof. Privacy settings about an object may specify how the object (or specific information associated with the object) may be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, executed, displayed, or identified) in an online social network. An object may be described as being "visible" to a particular user or other entity when the privacy settings of the object allow that user or other entity to access the object. As an example and not by way of limitation, a user of an online social network may specify privacy settings on a user profile page that identify a set of users that may access work experience information on the user profile page, thereby excluding other users from accessing that information.

[0181] In certain embodiments, the privacy settings of an object may specify a "blocked list" of users or other entities that should not be allowed to access certain information associated with the object. In certain embodiments, the blacklist may include third-party entities. The blacklist may specify one or more users or entities to which the object is invisible. As an example and not by way of limitation, a user may specify a set of users that may not access an album associated with the user, thereby excluding those users from accessing the album (while also potentially allowing certain users not in the specified set of users to access the album). In certain embodiments, privacy settings may be associated with certain social graph elements. The privacy settings of a social graph element (e.g., a node or edge) may specify how the social graph element, information associated with the social graph element, or an object associated with the social graph element may be accessed using an online social network. As an example and not by way of limitation, a specific concept node 704 corresponding to a specific photo may have a privacy setting that specifies that the photo can only be accessed by the user tagged in the photo and the friends of the user tagged in the photo. In certain embodiments, privacy settings may allow users to opt-in or opt-out of having their content, information, or actions stored / recorded by the social networking system 160 or assistant system 140 or shared with other systems (e.g., third-party systems 170). Although this disclosure describes using particular privacy settings in particular manners, this disclosure contemplates using any suitable privacy settings in any suitable manner.

[0182] In certain embodiments, privacy settings may be based on one or more nodes or edges of the social graph 700. Privacy settings may be specified for one or more edges 706 or edge types of the social graph 700, or for one or more nodes 702, 704 or node types of the social graph 700. A privacy setting applied to a particular edge 706 connecting two nodes may control whether the relationship between two entities corresponding to the two nodes is visible to other users of the online social network. Similarly, a privacy setting applied to a particular node may control whether a user or concept corresponding to the node is visible to other users of the online social network. As an example and not by way of limitation, a first user may share an object to the social networking system 160. The object may be associated with a concept node 704 connected to the user node 702 of the first user by an edge 706. The first user may specify a privacy setting that applies to a particular edge 706 of the concept node 704 connected to the object, or may specify a privacy setting that applies to all edges 706 connected to the concept node 704. As another example and not by way of limitation, the first user may share a collection of objects (e.g., a collection of images) of a particular object type. The first user may specify privacy settings for all objects of that particular object type associated with the first user to have a particular privacy setting (e.g., specifying that all images posted by the first user are visible only to the first user's friends and / or users tagged in the images).

[0183] In certain embodiments, social-networking system 160 may present a "privacy wizard" to the first user (e.g., within a web page, module, one or more dialog boxes, or any other suitable interface) to assist the first user in specifying one or more privacy settings. The privacy wizard may display instructions, appropriate privacy-related information, current privacy settings, one or more input fields for accepting one or more inputs from the first user (which specifies a change or confirmation of a privacy setting), or any suitable combination thereof. In certain embodiments, social-networking system 160 may provide the first user with a "dashboard" function that may display the first user's current privacy settings to the first user. The dashboard function may be displayed to the first user at any appropriate time (e.g., after input from the first user invoking the dashboard function, after a particular event or triggering action occurs). The dashboard function may allow the first user to modify one or more current privacy settings of the first user at any time in any suitable manner (e.g., redirecting the first user to the privacy wizard).

[0184] The privacy settings associated with an object may specify any suitable granularity at which access is allowed or denied. As an example and not by way of limitation, access or denial of access may be specified for specific users (e.g., only me, my roommate, my boss), users within a particular degree of separation (e.g., friends, friends of friends), user groups (e.g., a gaming club, my family), user networks (e.g., employees of a particular employer, students or alumni of a particular university), all users ("public"), no users ("private"), users of third-party systems 170, specific applications (e.g., third-party applications, external websites), other suitable entities, or any suitable combination thereof. Although the present disclosure describes a specific granularity at which access is allowed or denied, the present disclosure contemplates any suitable granularity at which access is allowed or denied.

[0185] In certain embodiments, one or more servers 162 may be authorization / privacy servers for implementing privacy settings. In response to a request from a user (or other entity) for a particular object stored in data storage 164, social networking system 160 may send a request for the object to data storage 164. The request may identify a user associated with the request, and the object may be sent to the user (or the user's client system 130) only if the authorization server determines that the user is authorized to access the object based on the privacy settings associated with the object. If the requesting user is not authorized to access the object, the authorization server may prevent the requested object from being retrieved from data storage 164, or may prevent the requested object from being sent to the user. In a search-query context, an object may be provided as a search result only if the querying user is authorized to access the object, for example, if the privacy settings of the object allow it to be revealed to the querying user, discovered by the querying user, or otherwise visible to the querying user. In certain embodiments, the object may represent content visible to the user through the user's dynamic message. As an example and not by way of limitation, one or more objects may be visible to the user's "Trending" page. In a particular embodiment, an object may correspond to a particular user. An object may be content associated with a particular user, or may be an account of a particular user or information stored on a social networking system 160 or other computing system. As an example and not by way of limitation, a first user may view one or more second users of an online social network through a “People You May Know” feature of the online social network or by viewing a list of friends of the first user. As an example and not by way of limitation, a first user may specify that they do not wish to see objects associated with a particular second user in their dynamic messages or friend lists. If the privacy settings of an object do not allow it to be revealed to the user, discovered by the user, or visible to the user, the object may be excluded from the search results. Although the present disclosure describes implementing privacy settings in a particular manner, the present disclosure contemplates implementing privacy settings in any suitable manner.

[0186] In certain embodiments, different objects of the same type associated with a user may have different privacy settings. Different types of objects associated with a user may have different types of privacy settings. As an example and not as a limitation, a first user may specify that the first user's status update is public, but any images shared by the first user are only visible to the first user's friends on an online social network. As another example and not as a limitation, a user may specify different privacy settings for different types of entities (such as individual users, friends of friends, followers, user groups, or corporate entities). As another example and not as a limitation, a first user may specify a group of users who can view a video posted by the first user while preventing the video from being visible to the first user's employer. In certain embodiments, different privacy settings may be provided for different user groups or user demographics. As an example and not as a limitation, a first user may specify that other users who attend the same university as the first user can view the first user's photos, but other users who are family members of the first user cannot view those same photos.

[0187] In certain embodiments, social networking system 160 may provide one or more default privacy settings for each object of a certain object type. The privacy settings of an object that are set as default may be changed by a user associated with the object. As an example and not by way of limitation, all images posted by a first user may have a default privacy setting of only visible to the first user's friends, and for a certain image, the first user may change the privacy setting of the image to be visible to friends and friends of friends.

[0188] In certain embodiments, privacy settings may allow a first user to specify (e.g., by opting out, by not opting in) whether social-networking system 160 or assistant system 140 may receive, collect, record, or store certain objects or information associated with the user for any purpose. In certain embodiments, privacy settings may allow a first user to specify whether certain applications or processes may access, store, or use certain objects or information associated with the user. Privacy settings may allow a first user to opt in or opt out of having an object or information accessed, stored, or used by a certain application or process. Social-networking system 160 or assistant system 140 may access such information in order to provide certain functionality or services to the first user, while social-networking system 160 or assistant system 140 may not access the information for any other purpose. Prior to accessing, storing, or using such an object or information, social-networking system 160 or assistant system 140 may prompt the user to provide a privacy setting that specifies which applications or processes, if any, may access, store, or use the object or information before any such action is allowed. As an example and not by way of limitation, a first user may transmit a message to a second user via an application associated with an online social network (e.g., a messaging app), and may specify a privacy setting that social networking system 160 or assistant system 140 should not store such messages.

[0189] In certain embodiments, a user may specify whether a particular type of object or information associated with a first user may be accessed, stored, or used by social networking system 160 or assistant system 140. As an example and not by way of limitation, a first user may specify that images sent by the first user via social networking system 160 or assistant system 140 may not be stored by social networking system 160 or assistant system 140. As another example and not by way of limitation, a first user may specify that messages sent from the first user to a particular second user may not be stored by social networking system 160 or assistant system 140. As yet another example and not by way of limitation, a first user may specify that all objects sent via a particular application may be saved by social networking system 160 or assistant system 140.

[0190] In certain embodiments, the privacy settings may allow the first user to specify whether a particular object or information associated with the first user may be accessed from a particular client system 130 or third-party system 170. The privacy settings may allow the first user to opt-in or opt-out of accessing objects or information from a particular device (e.g., a phone book on the user's smartphone), from a particular application (e.g., a messaging app), or from a particular system (e.g., an email server). The social networking system 160 or assistant system 140 may provide a default privacy setting for each device, system, or application, and / or may prompt the first user to specify a specific privacy setting for each context. As an example and not by way of limitation, the first user may utilize a location services feature of the social networking system 160 or assistant system 140 to provide recommendations for restaurants or other places near the user. The first user's default privacy settings may specify that the social networking system 160 or assistant system 140 may use location information provided from the first user's client device 130 to provide location-based services, but the social networking system 160 or assistant system 140 may not store the first user's location information or provide it to any third-party system 170. The first user may then update the privacy settings to allow the third-party image sharing application to use the location information to geotag the photos.

[0191] In certain embodiments, privacy settings may allow a user to specify one or more geographic locations from which an object may be accessed. Access or denial of access to an object may depend on the geographic location of the user attempting to access the object. As an example and not as a limitation, a user may share an object and specify that only users in the same city may access or view the object. As another example and not as a limitation, a first user may share an object and specify that the object is visible to a second user only when the first user is in a particular location. If the first user leaves the particular location, the object may no longer be visible to the second user. As another example and not as a limitation, the first user may specify that the object is visible only to a second user within a threshold distance from the first user. If the first user subsequently changes location, the original second user who could access the object may lose access, while a new group of second users may gain access when they enter within a threshold distance of the first user.

[0192] In certain embodiments, social networking system 160 or assistant system 140 may have functionality that can use a user's personal or biometric information as input for user authentication or experience personalization purposes. Users may choose to utilize these features to enhance their experience on online social networks. As an example and not by way of limitation, a user may provide personal or biometric information to social networking system 160 or assistant system 140. The user's privacy settings may specify that such information may only be used for specific processes (such as authentication) and may also specify that such information may not be shared with any third-party system 170 or may not be used for other processes or applications associated with social networking system 160 or assistant system 140. As another example and not by way of limitation, social networking system 160 may provide a user with the ability to provide a voiceprint recording to an online social network. As an example and not by way of limitation, if a user wishes to utilize this functionality of an online social network, the user may provide a voice recording of his or her own voice to provide a status update on the online social network. The recording of the voice input may be compared to the user's voiceprint to determine what words the user said. The user's privacy settings may specify that such voice recordings may be used only for voice input purposes (e.g., authenticating the user, sending voice messages, improving voice recognition for use of voice manipulation features of the online social network), and further specify that such voice recordings may not be shared with any third-party system 170, or may not be used by other processes or applications associated with the social networking system 160. As another example and not by way of limitation, the social networking system 160 may provide the user with the ability to provide a reference image (e.g., a facial profile, a retinal scan) to the online social network. The online social network may compare the reference image with an image input received later (e.g., for authenticating the user, tagging the user in a photo). The user's privacy settings may specify that such images may only be used for limited purposes (e.g., authentication, tagging the user in a photo), and further specify that such images may not be shared with any third-party system 170, or may not be used by other processes or applications associated with the social networking system 160.

[0193] System and method

[0194] Fig.10An example computer system 1000 is shown. In a particular embodiment, one or more computer systems 1000 perform one or more steps of one or more methods described or shown herein. In a particular embodiment, one or more computer systems 1000 provide functionality described or shown herein. In a particular embodiment, software running on one or more computer systems 1000 performs one or more steps of one or more methods described or shown herein, or provides functionality described or shown herein. A particular embodiment includes one or more portions of one or more computer systems 1000. Herein, references to computer systems may include computing devices, and vice versa, where appropriate. In addition, references to computer systems may include one or more computer systems, where appropriate.

[0195] The present disclosure contemplates any suitable number of computer systems 1000. The present disclosure contemplates computer systems 1000 taking any suitable physical form. By way of example and not limitation, computer system 1000 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 1000 may include one or more computer systems 1000; may be monolithic or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not as a limitation, one or more computer systems 1000 may perform one or more steps of one or more methods described or shown herein in real time or in batch mode. Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or shown herein at different times or at different locations.

[0196] In a particular embodiment, computer system 1000 includes a processor 1002, a memory 1004, a storage device 1006, an input / output (I / O) interface 1008, a communication interface 1010, and a bus 1012. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0197] In a particular embodiment, the processor 1002 includes hardware for executing instructions (e.g., those that constitute a computer program). As an example and not by way of limitation, to execute instructions, the processor 1002 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 1004, or a storage device 1006; decode these instructions and execute them; and then write one or more results to an internal register, an internal cache, a memory 1004, or a storage device 1006. In a particular embodiment, the processor 1002 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1002 that includes any suitable number of any suitable internal caches. As an example and not by way of limitation, the processor 1002 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1004 or the storage device 1006, and the instruction cache may speed up the retrieval of those instructions by the processor 1002. The data in the data cache may be: a copy of data in the memory 1004 or storage device 1006 for operation by instructions executed at the processor 1002; the result of a previous instruction executed at the processor 1002, for access by a subsequent instruction executed at the processor 1002 or for writing to the memory 1004 or storage device 1006; or other suitable data. The data cache may speed up read or write operations performed by the processor 1002. The TLB may speed up virtual address translations for the processor 1002. In a particular embodiment, the processor 1002 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1002 including any suitable number of any suitable internal registers. Where appropriate, the processor 1002 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or include one or more processors 1002. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0198] In a particular embodiment, the memory 1004 includes a main memory for storing instructions for execution by the processor 1002 or data for operation by the processor 1002. As an example and not by way of limitation, the computer system 1000 can load instructions from the storage device 1006 or another source (such as, for example, another computer system 1000) to the memory 1004. The processor 1002 can then load the instructions from the memory 1004 to an internal register or an internal cache. To execute the instructions, the processor 1002 can retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 1002 can write one or more results (which can be intermediate results or final results) to the internal register or the internal cache. The processor 1002 can then write one or more of these results to the memory 1004. In a particular embodiment, the processor 1002 executes only instructions in one or more internal registers or internal caches or in the memory 1004 (rather than the storage device 1006 or elsewhere), and operates only on data in one or more internal registers or internal caches or in the memory 1004 (rather than the storage device 1006 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 1002 to the memory 1004. As described below, the bus 1012 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 1002 and the memory 1004 and facilitate access to the memory 1004 requested by the processor 1002. In a particular embodiment, the memory 1004 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1004 may include one or more memories 1004. Although this disclosure describes and illustrates particular memories, this disclosure contemplates any suitable memories.

[0199] In a particular embodiment, the storage device 1006 includes a large-capacity storage device for data or instructions. As an example and not as a limitation, the storage device 1006 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the storage device 1006 may include a removable or non-removable (or fixed) medium. Where appropriate, the storage device 1006 may be inside or outside the computer system 1000. In a particular embodiment, the storage device 1006 is a non-volatile solid-state memory. In a particular embodiment, the storage device 1006 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM), or a flash memory, or a combination of two or more of these. The present disclosure contemplates a large-capacity storage device 1006 in any suitable physical form. Where appropriate, storage 1006 may include one or more storage control units that facilitate communications between processor 1002 and storage 1006. Where appropriate, storage 1006 may include one or more storage devices 1006. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0200] In a particular embodiment, I / O interface 1008 includes hardware, software, or both that provide one or more interfaces for communication between computer system 1000 and one or more I / O devices. Where appropriate, computer system 1000 may include one or more of these I / O devices. One or more of these I / O devices may enable communication between a person and computer system 1000. As an example and not as a limitation, I / O devices may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a static camera, a stylus, a tablet computer, a touch screen, a tracking ball, a video camera, another suitable I / O device, or a combination of two or more of these devices. I / O devices may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1008 for them. Where appropriate, I / O interface 1008 may include one or more devices or software drivers that enable processor 1002 to drive one or more of these I / O devices. Where appropriate, I / O interface 1008 may include one or more I / O interfaces 1008. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0201] In a particular embodiment, the communication interface 1010 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1000 and one or more other computer systems 1000 or one or more networks. As an example and not by way of limitation, the communication interface 1010 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (e.g., a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 1010 for it. As an example and not by way of limitation, the computer system 1000 may communicate with one or more parts of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1000 may communicate with a wireless PAN (WPAN) such as, for example, a Bluetooth WPAN, a WI-FI network, a WI-MAX network, a cellular telephone network such as, for example, a Global System for Mobile Communications (GSM) network, or other suitable wireless networks, or a combination of two or more of these. Where appropriate, the computer system 1000 may include any suitable communication interface 1010 for any of these networks. Where appropriate, the communication interface 1010 may include one or more communication interfaces 1010. Although this disclosure describes and illustrates particular communication interfaces, this disclosure contemplates any suitable communication interface.

[0202] In a particular embodiment, bus 1012 includes hardware, software, or both that couples components of computer system 1000 to each other. By way of example and not limitation, bus 1012 may include an accelerated graphics port (AGP) or other graphics bus, an extended industry standard architecture (EISA) bus, a front side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or any other suitable bus, or a combination of two or more of these. Where appropriate, bus 1012 may include one or more buses 1012. Although the present disclosure describes and illustrates a particular bus, the present disclosure contemplates any suitable bus or interconnect.

[0203] Herein, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital (SECURE DIGITAL) cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, the computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0204] Other Miscellaneous

[0205] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated by context. Thus, herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated by context. Furthermore, "and" is both conjunctive and several, unless expressly indicated otherwise or indicated by context. Thus, herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated by context.

[0206] The scope of the present disclosure includes all changes, substitutions, variations, alterations and modifications to the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the corresponding embodiments herein as including specific parts, elements, features, functions, operations or steps, any of these embodiments may include any combination or permutation of any parts, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, references to devices or systems or parts of devices or systems that are suitable for, arranged to, capable of, configured to, implemented to, operable to, or operated to perform specific functions in the appended claims include the device, system, component, whether it or that specific function is activated, turned on or unlocked, as long as the device, system or component is adjusted, arranged, enabled, configured, implemented, operable, or operated in this way. In addition, although the present disclosure describes or illustrates specific embodiments as providing specific advantages, specific embodiments may provide some, all or none of these advantages.

Claims

1. A method comprising: Receiving a first gesture input from a user and a voice input from the user; determining user intent based at least in part on the speech input; determining, based at least in part on the first gesture input, a gesture identifier associated with the first gesture input; associating the gesture identifier with the user intent; receiving a second gesture input from the user; determining whether the second gesture input corresponds to the gesture identifier; In response to determining that the second gesture input corresponds to the gesture identifier, a task is performed based at least in part on the user intent associated with the gesture identifier.

2. The method according to claim 1, wherein: The gesture identifier corresponds to one of a pinch gesture, a swipe gesture, or an open palm gesture.

3. The method according to claim 1, wherein: The computer system includes smart glasses.

4. The method according to claim 1, wherein: An assistant system performs the task, and the task includes presenting a response to the user via the computer system.

5. The method according to claim 4, wherein: The response includes at least one of text, image, video, or gesture animation.

6. The method according to claim 4, wherein: The assistant system is external to the computer system and communicatively coupled to the computer system via a network.

7. The method according to claim 6, wherein: The assistant system includes an artificial neural network.

8. The method of claim 1, wherein: The first gesture input includes an arc-shaped movement of a hand of the user extending at least two fingers, The user intent corresponds to the next instruction, and The gesture identifier is determined to be a slide gesture.

Images