Context-driven conversations in noisy environment of assistant system

Abstract

A context-driven conversation in a noisy environment of an assistant system is provided. In one embodiment, a method includes accessing, from a head-mounted device associated with a user, an audio signal corresponding to a voice input from the user; determining an audio environment metric corresponding to the speech input based on the audio signal; calculating a confidence score indicating a likelihood that an assistant system associated with the head-mounted device understands the speech input; generating a response based on the confidence score and the audio environment metric; and providing instructions to the head-mounted device to present the response.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. patent application No. 18 / 352,889, filed on July 14, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to database and file management within a network environment, and more particularly to hardware and software for intelligent assistant systems. Background Art

[0004] The assistant system can provide information or services on behalf of the user based on a combination of the following: user input, location awareness, and the ability to access information (e.g., weather conditions, traffic jams, news, stock prices, user schedules, retail prices, etc.) from various online sources. User input can include text (e.g., online chat), voice, images, actions, or a combination thereof (especially in instant messaging applications or other applications). The assistant system can perform concierge-type services (e.g., booking dinner, purchasing event tickets, making travel arrangements) or provide information based on user input. The assistant system can also perform management tasks or data processing tasks based on online information and events without user initiation or interaction. Examples of those tasks that can be performed by the assistant system can include schedule management (e.g., sending reminders to users to postpone dinner dates due to traffic conditions, updating the schedules of both parties, and changing restaurant reservation times). The assistant system can be implemented through a combination of the following items: computing devices, application programming interfaces (APIs), and a large number of applications on user devices.

[0005] A social networking system, which may include a social networking website, may enable its users (e.g., individuals or organizations) to interact with the social networking system and with each other through the social networking system. The social networking system may use input from the user to create and store user profiles associated with the user in the social networking system. The user profile may include demographic information, communication channel information, and information about the user's personal interests. The social networking system may also use input from the user to create and store a record of the user's relationships with other users in the social networking system, as well as provide services for facilitating social interaction between or among users (e.g., profile / news feed postings, photo sharing, event organization, messaging, games, or advertising).

[0006] The social networking system may send content or messages related to the services of the social networking system to the user's mobile device or other computing device through one or more networks. The user may also install a software application on the user's mobile device or other computing device to access the user's user profile and other data within the social networking system. The social networking system may generate a set of personalized content objects (e.g., a news feed of comprehensive reports of other users connected to the user) for display to the user. Summary of the invention

[0007] In a particular embodiment, the assistant system can help users obtain information or services. The assistant system can enable users to interact with the assistant system through user input in various modes (e.g., audio, voice, text, image, video, gesture, motion, position, orientation) in a stateful and multi-round dialogue to get help from the assistant system. As an example and not limitation, the assistant system can support unimodal input (e.g., voice input only), multimodal input (e.g., voice input and text input), mixed / multimodal input, or any combination thereof. The user input provided by the user can be associated with a specific assistant-related task, and can, for example, include user requests (e.g., verbal requests for information or action execution), and user interactions with assistant applications associated with the assistant system (e.g., selecting user interface (UI) elements by touch or gesture), or any other type of suitable user input that can be detected and understood by the assistant system (e.g., user movement detected by the user's client device). The assistant system can create and store such user profiles: the user profile includes both personal information and context information associated with the user. In certain embodiments, the assistant system may analyze user input using natural-language understanding (NLU). For a more personalized and context-aware understanding, the analysis may be based on the user's user profile. The assistant system may resolve entities associated with the user input based on the analysis. In certain embodiments, the assistant system may interact with different agents to obtain information or services associated with the resolved entity. The assistant system may generate responses about the information or services for the user by using natural-language generation (NLG). The assistant system may manage and advance the flow of conversations with the user by using dialogue management techniques through interaction with the user. In certain embodiments, the assistant system may also help the user understand the acquired information effectively and efficiently by summarizing the information. The assistant system may also help the user interact more with online social networks by providing tools that help the user interact with online social networks (e.g., create posts, comments, messages). The assistant system may also help the user manage different tasks, such as keeping track of events. In certain embodiments, the assistant system may actively perform tasks related to the user's interests and preferences based on the user profile at a time relevant to the user without user input. In certain embodiments, the assistant system may check privacy settings to ensure that access to a user's profile or other user information and performance of different tasks are permitted based on the user's privacy settings.

[0008] In a particular embodiment, the assistant system can help users through a hybrid architecture built on both client-side processes and server-side processes. The client-side process and the server-side process can be two parallel workflows for processing user input and providing assistance to the user. In a particular embodiment, the client-side process can be executed locally on a client system associated with the user. In contrast, the server-side process can be executed remotely on one or more computing systems. In a particular embodiment, an arbitrator on the client system can: coordinate the reception of user input (e.g., an audio signal), determine whether to use a client-side process, a server-side process, or both to respond to the user input, and analyze the processing results from each process. The arbitrator can instruct the agent on the client side or the server side to perform tasks associated with the user input based on the aforementioned analysis. The execution results can be further rendered as the output of the client system. The assistant system can effectively help users optimize the use of computing resources by utilizing both client-side processes and server-side processes, while protecting user privacy and enhancing security.

[0009] In a particular embodiment, when a user uses an assistant-enabled head-mounted device such as smart glasses, the assistant system can improve the response to the user's voice commands in a noisy environment. The assistant system can access the audio signal collected by the microphone of the head-mounted device, measure the background noise using different frame-based environmental metrics, and use the measurement results to drive a meaningful response to the user (i.e., a contextual response based on the noisy environment). The assistant system can use noise suppression to improve automatic speech recognition (ASR) in a noisy environment, automatically increase the volume of the assistant's response when the user is in a noisy environment, and successfully resume the task instead of simply failing the task. As an example and not limitation, a user in a noisy environment speaks a voice command such as "Hey Assistant, take a picture." The assistant system can respond with "It's too noisy. I can't understand you" (i.e., an out-of-context response). Similarly, as the task resumes, the assistant system may say "It's noisy, but do you want to take a picture?" (i.e., a low-confidence successful contextual response with explicit confirmation). Although this disclosure describes a particular response in a particular context being generated by a particular system in a particular manner, this disclosure contemplates any suitable response in any suitable context being generated by any suitable system in any suitable manner.

[0010] In a particular embodiment, the assistant system can access one or more audio signals corresponding to a voice input from a user from a head-mounted device associated with the user. The assistant system can then determine one or more audio environment metrics corresponding to the voice input based on the one or more audio signals. The assistant system can then calculate a confidence score indicating the likelihood that the assistant system associated with the head-mounted device understands the voice input. In a particular embodiment, the assistant system can generate a response based on the confidence score and the one or more audio environment metrics. The assistant system can also provide instructions to the head-mounted device to present the response.

[0011] There are certain technical challenges in generating contextual responses in a noisy environment. One technical challenge may include determining that the user is in a noisy environment. A solution for solving this challenge proposed by the embodiments disclosed herein may be to use an audio environment metric to determine the noise level associated with the received audio signal, because the audio environment metric can provide effective measurements of ambient noise loudness, ambient wind energy level, and voice activity. Another technical challenge may include determining what type of response is provided to the user in a noisy environment. A solution for solving this challenge proposed by the embodiments disclosed herein may be to use a model-based dialog (MBD) system, because a model-based dialog (MBD) system can train a lightweight model to select an action based on a set of declared input features, wherein the training is based on real interaction data to optimize task success in combination with ambient noise, wind noise, and voice activity.

[0012] Certain embodiments disclosed herein may provide one or more technical advantages. The technical advantages of each embodiment may include improving the ASR model to make it robust to background noise, because the assistant system can use noise suppression in a noisy environment to improve the ASR, so that the input of the assistant system (i.e., ASR token) has higher quality. Another technical advantage of each embodiment may include providing better assistant feedback to the user, because when the assistant system is listening to the user but cannot understand the command due to background noise, the assistant system can let the user understand, and also automatically increase the volume of the assistant text-to-speech (TTS) response when the user is in a noisy environment. Another technical advantage of each embodiment may include creating a more successful assistant interaction, because the assistant system can use the environmental context when deciding how to respond to low / medium confidence utterances for both single-turn interaction and multi-turn interaction. Certain embodiments disclosed herein may not provide the above technical advantages, or provide some or all of the above technical advantages. In view of the drawings, description and claims of the present disclosure, one or more other technical advantages are obvious to those skilled in the art.

[0013] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to these embodiments. A specific embodiment may include all, some, or not include the components, elements, features, functions, operations, or steps in the embodiments disclosed herein. The embodiments according to the present invention are specifically disclosed in the attached claims for methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or backreferences in the attached claims are selected only for formal reasons. However, any subject matter resulting from intentional reference to any previous claim (especially multiple dependencies) may also be claimed, so that any combination of multiple claims and multiple features thereof are disclosed and may be claimed regardless of the dependencies selected in the attached claims. The subject matter that can be claimed includes not only multiple combinations of multiple features stated in the attached claims, but also any other combination of multiple features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any embodiment and any feature described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or any feature described or depicted herein or in any combination with any feature in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 An example network environment associated with an assistant system is shown.

[0015] Figure 2 An example architecture for an assistant system is shown.

[0016] Figure 3 An example flow diagram of an assistant system is shown.

[0017] Figure 4 An example task-centric flow chart for processing user input is shown.

[0018] Figure 5 An example flow diagram for a server-side context-driven dialog is shown.

[0019] Figure 6 An example pipeline for utilizing an audio environment metric on a frame is shown.

[0020] Figure 7 An example metadata encoded audio stream is shown.

[0021] Figure 8 An example user flow with logic for graceful out-of-domain responses is shown.

[0022] Fig. 9 An example user flow with logic for a recovered out-of-domain response with low confidence is shown.

[0023] Fig.10 An example user flow with logic for a recovered out-of-domain response with medium confidence is shown.

[0024] Fig.11 An example high-level flow for gracefully generating out-of-domain responses using a context-driven model-based dialogue (MBD) system is shown.

[0025] FIG. 12A to FIG. 12C An example scenario of contextual response in a noisy environment is shown.

[0026] Fig.13 An example method for generating contextual responses in a noisy environment is shown.

[0027] Fig.14 An example computer system is shown. DETAILED DESCRIPTION

[0028] System Overview

[0029] Figure 1 An example network environment 100 associated with an assistant system is shown. The network environment 100 includes a client system 130, an assistant system 140, a social networking system 160, and a third-party system 170 connected to each other via a network 110. Figure 1 A particular arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110 is shown, but the present disclosure contemplates any suitable arrangement of client system 130, assistant system 140, social networking system 160, third-party system 170, and network 110. By way of example and not limitation, two or more of client system 130, social networking system 160, assistant system 140, and third-party system 170 may be directly connected to each other, bypassing network 110. As another example, two or more of client system 130, assistant system 140, social networking system 160, and third-party system 170 may be physically or logically co-located, in whole or in part, with each other. Furthermore, although Figure 1A particular number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110 are shown, but the present disclosure contemplates any suitable number of client systems 130, assistant systems 140, social networking systems 160, third-party systems 170, and networks 110. By way of example and not limitation, network environment 100 may include multiple client systems 130, multiple assistant systems 140, multiple social networking systems 160, multiple third-party systems 170, and multiple networks 110.

[0030] The present disclosure contemplates any suitable network 110. By way of example and not limitation, one or more portions of network 110 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a Public Switched Telephone Network (PSTN), a network based on cellular technology, a network based on satellite communication technology, another network 110, or a combination of two or more such networks 110.

[0031] Links 150 may connect client systems 130, assistant systems 140, social networking systems 160, and third-party systems 170 to communication network 110, or to each other. The present disclosure contemplates any suitable links 150. In particular embodiments, one or more links 150 include one or more wired (e.g., Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)) links, one or more wireless (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or one or more optical (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In certain embodiments, one or more links 150 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 150, or a combination of two or more such links 150. The multiple links 150 need not all be identical throughout the network environment 100. In one or more aspects, one or more first links 150 can be different from one or more second links.

[0032] In certain embodiments, client system 130 may be any suitable electronic device that includes hardware, software, or embedded logic components or a combination of two or more such components and is capable of performing the functions implemented or supported by client system 130. By way of example and not limitation, client system 130 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a smart speaker, a smart watch, smart glasses, augmented reality (AR) smart glasses, virtual reality (VR) head-mounted viewer (headset), other suitable electronic devices, or any suitable combination thereof. In certain embodiments, client system 130 may be a smart assistant device. More information about the smart assistant device can be found in the following documents: U.S. Patent Application No. 15 / 949,011, filed on April 9, 2018, U.S. Patent Application No. 16 / 153,574, filed on October 5, 2018, U.S. Design Patent Application No. 29 / 631,910, filed on January 3, 2018, U.S. Design Patent Application No. 29 / 631,747, filed on January 2, 2018, U.S. Design Patent Application No. 29 / 631,913, filed on January 3, 2018, and U.S. Design Patent Application No. 29 / 6631,914, filed on January 3, 2018, each of which is incorporated by reference. The present disclosure contemplates any suitable client system 130. In a particular embodiment, the client system 130 can enable a network user at the client system 130 to access the network 110. Client system 130 may also enable the user to communicate with other users at other client systems 130 .

[0033] In certain embodiments, the client system 130 may include a web browser 132 and may have one or more add-ons, plug-ins, or other extensions. A user at the client system 130 may enter a Uniform Resource Locator (URL) or other address to direct the web browser 132 to a specific server (e.g., server 162, or a server associated with a third-party system 170), and the web browser 132 may generate a Hyper Text Transfer Protocol (HTTP) request and transmit the HTTP request to the server. The server may receive the HTTP request and transmit one or more Hyper Text Markup Language (HTML) files to the client system 130 in response to the HTTP request. The client system 130 may render a web page interface (e.g., a web page) based on the HTML file from the server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not limitation, a web page interface may be rendered according to an HTML file, an Extensible Hyper Text Markup Language (XHTML) file, or an Extensible Markup Language (XML) file, depending on specific needs. Such an interface may also execute scripts, and combinations of markup languages ​​and scripts, etc. In this document, where appropriate, a reference to a web page interface includes one or more corresponding source files (which a browser may use to render the web page interface), and vice versa.

[0034] In certain embodiments, client system 130 may include social networking application 134 installed on client system 130. A user at client system 130 may use social networking application 134 to access an online social network. A user at client system 130 may use social networking application 134 to communicate with the user's social connections (e.g., friends, followers, followed accounts, contacts, etc.). A user at client system 130 may also use social networking application 134 to interact with a plurality of content objects (e.g., posts, news articles, ephemeral content, etc.) on an online social network. By way of example and not limitation, a user may use social networking application 134 to browse trending topics and breaking news.

[0035] In a particular embodiment, the client system 130 may include an assistant application 136. A user at the client system 130 may use the assistant application 136 to interact with the assistant system 140. In a particular embodiment, the assistant application 136 may include an assistant xbot function as a front-end interface for interacting with a user of the client system 130 (which includes receiving user input and presenting output). In a particular embodiment, the assistant application 136 may include a standalone application. In a particular embodiment, the assistant application 136 may be integrated into a social network application 134 or another suitable application (e.g., a messaging application). In a particular embodiment, the assistant application 136 may also be integrated into the client system 130, an assistant hardware device, or any other suitable hardware device. In a particular embodiment, the assistant application 136 may also be a part of the assistant system 140. In a particular embodiment, the assistant application 136 may be accessed through a web browser 132. In a particular embodiment, a user may interact with the assistant system 140 in the following manner: providing user input to the assistant application 136 through various modalities (e.g., audio, voice, text, vision, image, video, gesture, motion, activity, position, orientation). The assistant application 136 can transmit user input to the assistant system 140 (e.g., through the assistant xbot). The assistant system 140 can generate a response based on the user input. The assistant system 140 can send the generated response to the assistant application 136. The assistant application 136 can then present the response to the user at the client system 130 in various modalities (e.g., audio, text, image, and video). As an example and not a limitation, the user can interact with the assistant system 140 in the following manner: provide user input to the assistant xbot through the microphone of the client system 130 (e.g., a verbal request for information about the current state of nearby vehicle traffic). The assistant application 136 can then transmit the user input to the assistant system 140 through the network 110. The assistant system 140 can analyze the user input accordingly, generate a response based on the analysis of the user input (e.g., vehicle traffic information obtained from a third-party source), and transmit the generated response back to the assistant application 136. Assistant application 136 may then present the generated response to the user in any suitable manner (e.g., display on a display of client system 130: a text-based push notification and / or one or more images of a local map showing nearby vehicular traffic).

[0036] In certain embodiments, client system 130 may implement wake-up word detection techniques to allow a user to conveniently activate assistant system 140 using one or more wake-up words associated with assistant system 140. By way of example and not limitation, a system audio API on client system 130 may continuously monitor user input comprising audio data (e.g., multiple frames of voice data) received at client system 130. In this example, the wake-up word associated with assistant system 140 may be a voice phrase "Hey Assistant". In this example, when the system audio API on client system 130 detects the voice phrase "Hey Assistant" in the monitored audio data, assistant system 140 may be activated for subsequent interaction with the user. In alternative embodiments, similar detection techniques may be implemented to activate assistant system 140 using specific non-audio user input associated with assistant system 140. For example, the non-audio user input may be a specific visual signal detected by a low-power sensor (e.g., a camera) of client system 130. By way of example and not limitation, the visual signal may be a static image (e.g., a bar code, a quick response (QR) code, a universal product code (UPC)), the user's location (e.g., the user's gaze at client system 130), a user action (e.g., the user pointing at an object), or any other suitable visual signal.

[0037] In certain embodiments, client system 130 may include rendering device 137 and, optionally, companion device 138. Rendering device 137 may be configured to render output generated by assistant system 140 to a user. Companion device 138 may be configured to perform computations locally on companion device 138 (i.e., on the device) in certain circumstances, such as when rendering device 137 is unable to perform computations associated with a particular task (e.g., communication with assistant system 140). In certain embodiments, client system 130, rendering device 137, and / or companion device 138 may each be a suitable electronic device including hardware, software, or embedded logic components or a combination of two or more such components, and client system 130, rendering device 137, and / or companion device 138 may be capable of independently or collaboratively performing the functions implemented or supported by client system 130 as described herein. By way of example and not limitation, the client system 130, the rendering device 137, and / or the supporting device 138 may each include a computer system, such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a smart speaker, a virtual reality (VR) head mounted viewer, an augmented reality (AR) smart glasses, other suitable electronic devices, or any suitable combination thereof. In a particular embodiment, one or more of the client system 130, the rendering device 137, and the supporting device 138 may operate as a smart assistant device. By way of example and not limitation, the rendering device 137 may include smart glasses, and the supporting device 138 may include a smart phone. By way of another example and not limitation, the rendering device 137 may include a smart watch, and the supporting device 138 may include a smart phone. By way of yet another example and not limitation, the rendering device 137 may include smart glasses, and the supporting device 138 may include a smart remote control for the smart glasses. As yet another example and not limitation, rendering device 137 may include a VR / AR head-mounted viewer, and companion device 138 may include a smartphone.

[0038] In certain embodiments, a user may use rendering device 137 or companion device 138, either alone or in combination, to interact with assistant system 140. In certain embodiments, one or more of client system 130, rendering device 137, and companion device 138 may implement a multi-stage wake-up word detection model to enable a user to conveniently activate assistant system 140 by continuously monitoring one or more wake-up words associated with assistant system 140. In the first stage of the wake-up word detection model, rendering device 137 may receive audio user input (e.g., multiple frames of voice data). If a wireless connection between rendering device 137 and companion device 138 is available, an application on rendering device 137 may transmit the received audio user input to a companion application on companion device 138 via a wireless connection. In the second stage of the wake-up word detection model, a companion application on companion device 138 may process the received audio user input to detect a wake-up word associated with assistant system 140. Then, the companion application on companion device 138 may transmit the detected wake-up word to a server associated with assistant system 140 via wireless network 110. In the third stage of the wake-up word detection model, a server associated with assistant system 140 may perform keyword verification on the detected wake-up word to verify whether the user wants to activate assistant system 140 and receive help from assistant system 140. In alternative embodiments, any of the processing, detection, or keyword verification may be performed by rendering device 137 and / or companion device 138. In particular embodiments, when assistant system 140 has been activated by the user, the application on rendering device 137 may be configured to receive user input from the user, and the companion application on companion device 138 may be configured to process the user input (e.g., user request) received by the application on rendering device 137. In particular embodiments, rendering device 137 and companion device 138 may be associated (i.e., paired) with each other via one or more wireless communication protocols (e.g., Bluetooth).

[0039] The following example workflow illustrates how the rendering device 137 and the companion device 138 may process user input provided by a user. In this example, an application on the rendering device 137 may receive user input including a user request directed to the rendering device 137. The application on the rendering device 137 may then determine the state of the wireless connection between the rendering device 137 and the companion device 138 (i.e., the tethering state). If the wireless connection between the rendering device 137 and the companion device 138 is not available, the application on the rendering device 137 may transmit the user request (optionally including additional data and / or contextual information available to the rendering device 137) to the assistant system 140 via the network 110. The assistant system 140 may then generate a response to the user request and transmit the generated response back to the rendering device 137. The rendering device 137 may then present the response to the user in any suitable manner. Alternatively, if a wireless connection between rendering device 137 and companion device 138 is available, the application on rendering device 137 can transmit the user request (optionally including additional data and / or context information available to rendering device 137) to the companion application on companion device 138 via the wireless connection. The companion application on companion device 138 can then transmit the user request (optionally including additional data and / or context information available to companion device 138) to assistant system 140 via network 110. Assistant system 140 can then generate a response to the user request and transmit the generated response back to companion device 138. The companion application on companion device 138 can then transmit the generated response to the application on rendering device 137. Rendering device 137 can then present the response to the user in any suitable manner. In the previous example workflow, rendering device 137 and companion device 138 can each perform one or more calculations and / or processing at each corresponding step of the workflow. In certain embodiments, the performance of computations and / or processing disclosed herein may be adaptively switched between the rendering device 137 and the companion device 138 based at least in part on the device state of the rendering device 137 and / or the companion device 138, the task associated with the user input, and / or one or more additional factors. By way of example and not limitation, one factor may be the signal strength of the wireless connection between the rendering device 137 and the companion device 138. For example, if the signal strength of the wireless connection between the rendering device 137 and the companion device 138 is strong, the computations and processing may be adaptively switched to be substantially performed by the companion device 138, e.g., to benefit from the greater processing power of the central processing unit (CPU) of the companion device 138. Alternatively, if the signal strength of the wireless connection between the rendering device 137 and the companion device 138 is weak, the computations and processing may be adaptively switched to be substantially performed by the rendering device 137 in an independent manner.In certain embodiments, if the client system 130 does not include the companion device 138, the aforementioned calculations and processing may be performed solely by the rendering device 137 in an independent manner.

[0040] In certain embodiments, assistant system 140 can assist users in performing various assistant-related tasks. Assistant system 140 can interact with social networking system 160 and / or third-party system 170 when performing these assistant-related tasks.

[0041] In certain embodiments, social networking system 160 may be a network-addressable computing system that may host an online social network. Social networking system 160 may generate, store, receive, and transmit social networking data, such as user profile data, concept profile data, social graph information, or other suitable data related to an online social network. Social networking system 160 may be accessed directly by other components in network environment 100 or through network 110. By way of example and not limitation, client system 130 may access social networking system 160 directly or through network 110 using web browser 132 or a native application associated with social networking system 160 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In certain embodiments, social networking system 160 may include one or more servers 162. Each server 162 may be a single server or a distributed server across multiple computers or multiple data centers. By way of example and not limitation, each server 162 may be a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 162 may include hardware, software, or embedded logic components, or a combination of two or more of these components, for performing the appropriate functions implemented or supported by the server 162. In a particular embodiment, the social networking system 160 may include one or more data repositories 164. The data repository 164 may be used to store various types of information. In a particular embodiment, the information stored in the data repository 164 may be organized according to a particular data structure. In a particular embodiment, each data repository 164 may be a relational database, a columnar database, a correlation database, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. Particular embodiments may provide an interface that enables client system 130 , social networking system 160 , assistant system 140 , or third-party system 170 to manage, retrieve, modify, add, or delete information stored in data repository 164 .

[0042] In certain embodiments, social networking system 160 may store one or more social graphs in one or more data stores 164. In certain embodiments, a social graph may include a plurality of nodes, which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept) and a plurality of edges connecting the nodes. Social networking system 160 may provide users of an online social network with the ability to communicate and interact with other users. In certain embodiments, a user may join an online social network through social networking system 160, and may subsequently add connections (e.g., relationships) with a plurality of other users in social networking system 160 to whom they want to be connected. As used herein, the term "friend" may refer to any other user in social networking system 160 with whom a user has formed a connection, association, or relationship through social networking system 160.

[0043] In particular embodiments, social-networking system 160 may provide users with the ability to take actions on various types of items or objects supported by social-networking system 160. By way of example and not limitation, these items and objects may include groups or social networks to which users of social-networking system 160 may belong, events or calendar entries that may be of interest to the users, computer-based applications that the users may use, transactions that allow users to buy or sell items through the service, interactions with advertisements that are actionable by the users, or other suitable items or objects. Users may interact with anything that can be represented in social-networking system 160 or represented by an external system of third-party system 170 that is separate from social-networking system 160 and coupled to social-networking system 160 via network 110.

[0044] In certain embodiments, social networking system 160 may be able to link various entities. By way of example and not limitation, social networking system 160 may enable users to interact with each other and receive content from third-party systems 170 or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0045] In certain embodiments, third-party system 170 may include one or more types of servers, one or more data repositories, one or more interfaces (including, but not limited to, APIs), one or more web services, one or more content sources, one or more networks, or any other suitable component with which, for example, a server may communicate. Third-party system 170 may be operated by an entity that is different from the entity that operates social networking system 160. However, in certain embodiments, social networking system 160 and third-party system 170 may operate in conjunction with each other to provide social networking services to users of social networking system 160 or users of third-party system 170. In this sense, social networking system 160 may provide a platform or backbone that other systems (e.g., third-party system 170) may use to provide social networking services and functionality to users on the Internet.

[0046] In certain embodiments, third-party system 170 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be transmitted to client system 130. By way of example and not limitation, a content object may include information related to an item or activity of interest to a user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not limitation, a content object may include an incentive content object, such as a coupon, discount ticket, gift certificate, or other suitable incentive object. In certain embodiments, a third-party content provider may use one or more third-party agents to provide content objects and / or services. A third-party agent may be an implementation that is hosted and executed on third-party system 170.

[0047] In certain embodiments, social networking system 160 also includes user-generated content objects that can enhance a user's interaction with social networking system 160. User-generated content can include any content that a user can add, upload, send, or "post" to social networking system 160. By way of example and not limitation, a user transmits a post from client system 130 to social networking system 160. Posts can include data such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to social networking system 160 by third parties through "communication channels" (e.g., news feeds or streams).

[0048] In certain embodiments, the social networking system 160 may include various servers, subsystems, programs, modules, logs, and data repositories. In certain embodiments, the social networking system 160 may include one or more of the following: a web server, an action logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object disclosure log, an inference module, an authorization / privacy server, a search module, an advertising positioning module, a user interface module, a user profile repository, a contact repository, a third-party content repository, or a location repository. The social networking system 160 may also include suitable components, such as a web interface, a security mechanism, a load balancer, a failover server, a management and network operation console, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 160 may include one or more user profile memories for storing user profiles. User profiles may include, for example, biometric information, personal background information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, in-law relationships, or locations). Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not limitation, if a user "likes" an item about a shoe brand, the category may be the brand, or may be a general category of "shoes" or "clothing". A contact repository may be used to store contact information about a user. The contact information may indicate users with similar or common work experience, group membership, hobbies, educational history, or users who are related or share common attributes in any way. The contact information may also include user-defined connections between different users and content (both internal and external). A web server may be used to link the social networking system 160 to one or more client systems 130 or one or more third-party systems 170 via a network 110. The web server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 160 and one or more client systems 130. The API request server may allow, for example, an assistant system 140 or a third-party system 170 to access information from the social networking system 160 by calling one or more APIs. An action logger may be used to receive information from a web server about a user's actions to log in or out of the social networking system 160. In conjunction with the action log, a third-party content object log that a user has disclosed to a third-party content object may be maintained. The notification controller may provide information about the content object to the client system 130. The information may be pushed to the client system 130 as a notification, or may be pulled from the client system 130 in response to user input including a user request received from the client system 130. The authorization server may be used to enforce one or more privacy settings of users of the social networking system 160.The user's privacy settings may determine how specific information associated with the user may be shared. The authorization server may allow the user to opt in or opt out of having their actions recorded by the social networking system 160 or shared with other systems (e.g., third-party systems 170), for example, by setting appropriate privacy settings. A third-party content object repository may be used to store content objects received from third parties (e.g., third-party systems 170). A location repository may be used to store location information received from a client system 130 associated with a user. The advertising pricing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0049] Assistant System

[0050] Figure 2 An example architecture 200 of an assistant system 140 is shown. In certain embodiments, the assistant system 140 can assist a user in obtaining information or services. The assistant system 140 can enable a user to interact with the assistant system 140 in a stateful and multi-turn conversation through user input in various modalities (e.g., audio, voice, text, vision, image, video, gesture, action, activity, location, orientation) to receive assistance from the assistant system 140. As an example and not limitation, the user input can include audio input based on user voice (e.g., verbal commands), which can be processed by a system audio API (application programming interface) on the client system 130. The system audio API can perform techniques including echo cancellation, noise removal, beamforming, self-user voice activation, speaker recognition, voice activity detection (VAD), and / or any other suitable acoustic technology to generate audio data that can be easily processed by the assistant system 140. In particular embodiments, assistant system 140 may support unimodal input (e.g., voice input only), multimodal input (e.g., voice input and text input), hybrid / multimodal input, or any combination thereof. In particular embodiments, user input may be user-generated input sent to assistant system 140 in a single turn. User input provided by a user may be associated with a particular assistant-related task and may include, for example, user requests (e.g., verbal requests for information or performance of an action), and user interactions with assistant applications 136 associated with assistant system 140 (e.g., selection of a UI element by touch or gesture), or any other type of suitable user input detectable and understandable by assistant system 140 (e.g., user movements detected by a user's client device 130).

[0051] In certain embodiments, the assistant system 140 may create and store such a user profile: the user profile includes both personal information and contextual information associated with the user. In certain embodiments, the assistant system 140 may use natural language understanding (NLU) technology to analyze user input. For a more personalized and context-aware understanding, the analysis may be based at least in part on the user's user profile. The assistant system 140 may resolve entities associated with the user input based on the analysis. In certain embodiments, the assistant system 140 may interact with different agents to obtain information or services associated with the resolved entities. The assistant system 140 may generate responses about information or services for the user by using natural language generation (NLG). Through interaction with the user, the assistant system 140 may use dialogue management technology to manage and forward the dialogue flow with the user. In certain embodiments, the assistant system 140 may also help the user to effectively and efficiently understand the information obtained by summarizing the information. The assistant system 140 may also help the user to participate more in online social networks by providing tools that help the user interact with online social networks (e.g., create posts, comments, messages). The assistant system 140 may also help the user manage different tasks, such as keeping track of events. In certain embodiments, the assistant system 140 may proactively perform pre-authorized tasks related to the user's interests and preferences based on the user profile at a time relevant to the user, without user input. In certain embodiments, the assistant system 140 may check privacy settings to ensure that access to the user's profile or other user information and performance of different tasks are allowed based on the user's privacy settings. More information about helping users based on privacy settings can be found in the following document: U.S. Patent Application No. 16 / 182,542, filed on November 6, 2018, which is incorporated by reference.

[0052] In certain embodiments, assistant system 140 can assist users through an architecture built on client-side processes and server-side processes that can operate in various operating modes. Figure 2, the client-side process is shown above the dashed line 202, and the server-side process is shown below the dashed line 202. The first operating mode (i.e., on-device mode) can be a workflow in which the assistant system 140 processes user input and provides assistance to the user by primarily or exclusively executing the client-side process locally on the client system 130. For example, if the client system 130 is not connected to the network 110 (i.e., when the client system 130 is offline), the assistant system 140 can process user input using only the client-side process in the first operating mode. The second operating mode (i.e., cloud mode) can be a workflow in which the assistant system 140 processes user input and provides assistance to the user by primarily or exclusively executing the server-side process on one or more remote servers (e.g., servers associated with the assistant system 140). As Figure 2 As shown, the third operating mode (i.e., hybrid mode) can be a parallel workflow in which the assistant system 140 processes user input and provides assistance to the user by executing client-side processes locally on the client system 130 in combination with server-side processes on one or more remote servers (e.g., servers associated with the assistant system 140). For example, both the client system 130 and the server associated with the assistant system 140 can perform automatic speech recognition (ASR) processes and natural language understanding (NLU) processes, but the client system 130 can delegate the server associated with the assistant system 140 to perform dialog, agent, and natural language generation (NLG) processes.

[0053] In certain embodiments, the selection of the operating mode may be based at least in part on the device state, the task associated with the user input, and / or one or more additional factors. By way of example and not limitation, as described above, one factor may be the network connection state of client system 130. For example, if client system 130 is not connected to network 110 (i.e., when client system 130 is offline), assistant system 140 may process the user input in a first operating mode (i.e., on-device mode). By way of another example and not limitation, another factor may be based on a measurement of the available battery power (i.e., battery state) of client system 130. For example, if client system 130 needs to conserve battery power (e.g., when client system 130 has a minimum available battery power or the user has indicated a desire to conserve the battery power of client system 130), assistant system 140 may process the user input in a second operating mode (i.e., cloud mode) or a third operating mode (i.e., hybrid mode) to perform less power-consuming operations on client system 130. By way of yet another example and not limitation, another factor may be one or more privacy constraints (e.g., specified privacy settings, applicable privacy policies). For example, if one or more privacy constraints limit or prevent certain data from being sent to a remote server (e.g., a server associated with assistant system 140), assistant system 140 may process user input in a first operating mode (i.e., on-device mode) to protect user privacy. As yet another example and not limitation, another factor may be unsynchronized context data between client system 130 and a remote server (e.g., a server associated with assistant system 140). For example, it may be determined that client system 130 and a server associated with assistant system 140 have inconsistent, missing, and / or inconsistent context data, and assistant system 140 may process user input in a third operating mode (i.e., hybrid mode) to reduce the likelihood of inadequate analysis associated with the user input. As yet another example and not limitation, another factor may be a measurement of the latency of a connection between client system 130 and a remote server (e.g., a server associated with assistant system 140). For example, if the task associated with the user input may significantly benefit from and / or require timely or immediate execution (e.g., a photo collection task), assistant system 140 may process the user input in a first operating mode (i.e., in device mode) to ensure timely execution of the task. As yet another example and not limitation, another factor may be: for a feature related to the task associated with the user input, whether the feature is supported only by a remote server (e.g., a server associated with assistant system 140).For example, if the relevant feature requires advanced technical functions (e.g., high-performance processing capabilities, fast update cycles), and the advanced technical functions are only supported by a server associated with the assistant system 140 and not by the client system 130 at the time of user input, the assistant system 140 can process the user input in the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode) in order to benefit from the relevant feature.

[0054] In certain embodiments, the on-device coordinator 206 on the client system 130 can coordinate the receipt of user input and can determine which of the above-described operating modes should be used to process or continue processing the user input at one or more decision points in the example workflow. As described above, the selection of the operating mode can be based at least in part on the device state, the task associated with the user input, and / or one or more additional factors. By way of example and not limitation, reference is made to Figure 2 In the workflow architecture shown in , after receiving user input from a user, at decision point (D0) 205, the on-device coordinator 206 may determine whether to start processing the user input in a first operating mode (i.e., device mode), a second operating mode (i.e., cloud mode), or a third operating mode (i.e., hybrid mode). For example, at decision point (D0) 205, if the client system 130 is not connected to the network 110 (i.e., when the client system 130 is offline), if one or more privacy constraints explicitly require on-device processing (e.g., adding or removing another person to a private call between multiple users), or if the user input is associated with a task that does not require or benefit from server-side processing (e.g., setting a reminder or calling another user), the on-device coordinator 206 may select the first operating mode (i.e., device mode). As another example, at decision point (D0) 205, if the client system 130 needs to conserve battery power (e.g., when the client system 130 has a minimum available battery power or the user has expressed a desire to conserve battery power of the client system 130) or needs to limit additional utilization of computing resources (e.g., when other processes running on the client device 130 (e.g., an SMS messaging application) require high CPU utilization), the on-device coordinator 206 may select the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode).

[0055] In a specific embodiment, if Figure 2As shown, if the on-device coordinator 206 determines at decision point (D0) 205 that the user input should be processed using the first operating mode (i.e., on-device mode) or the third operating mode (i.e., hybrid mode), the client-side process can continue. By way of example and not limitation, if the user input includes voice data, the voice data can be received at a local automatic speech recognition (ASR) module 208a on the client system 130. The ASR module 208a can allow the user to dictate and transcribe the speech into written text, synthesize a document into an audio stream, or issue commands that are recognized by the system as such written text and audio streams.

[0056] In certain embodiments, the output of the ASR module 208a can be sent to a local natural language understanding (NLU) module 210a. The NLU module 210a can perform named entity resolution (NER), or named entity resolution can be performed by an entity resolution module 212a, as described below. In certain embodiments, one or more of an intent, a slot, or a domain can be the output of the NLU module 210a.

[0057] In a particular embodiment, user input may include non-voice data that may be received at a local context engine 220a. As an example and not limitation, non-voice data may include location, vision, touch, gesture, world update, social update, context information, information about people, activity data, and / or any other suitable type of non-voice data. Non-voice data may also include sensory data received by a sensor (e.g., microphone, camera) of the client system 130, which may be accessed subject to privacy constraints and may be further analyzed by computer vision techniques. In a particular embodiment, computer vision techniques may include object detection, scene recognition, hand tracking, eye tracking, and / or any other suitable computer vision techniques. In a particular embodiment, non-voice data may be subject to geometric construction, which may include using any suitable type of data collected by the client system 130 to construct objects around the user. As an example and not limitation, the user may be wearing AR glasses, and geometric construction may be used to determine the spatial position of surfaces and items (e.g., floors, walls, user's hands). In certain embodiments, the non-voice data may be inertial data collected by AR glasses or VR head mounted view devices, and the inertial data may be data associated with linear and angular motion (e.g., measurements associated with user body motion). In certain embodiments, the context engine 220a may determine various types of events and contexts based on the non-voice data.

[0058] In certain embodiments, the output of the NLU module 210a and / or the context engine 220a may be sent to the entity resolution module 212a. The entity resolution module 212a may resolve entities associated with one or more slots output by the NLU module 210a. In certain embodiments, each resolved entity may be associated with one or more entity identifiers. As an example and not limitation, the identifier may include a unique user identifier (ID) corresponding to a particular user (e.g., a unique username or user ID number of the social networking system 160). In certain embodiments, each resolved entity may also be associated with a confidence score. More information about resolving entities may be found in the following documents: U.S. Patent No. 10,803,050, filed on July 27, 2018, and U.S. Patent Application No. 16 / 048072, filed on July 27, 2018, each of which is incorporated by reference.

[0059] In certain embodiments, at decision point (D0) 205, the on-device coordinator 206 may determine that the user input should be processed in the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode). In these operating modes, the user input may be processed by certain server-side modules in a manner similar to the client-side process described above.

[0060] In particular embodiments, if the user input includes voice data, the user input voice data may be received at a remote automatic speech recognition (ASR) module 208 b on a remote server, such as a server associated with assistant system 140. ASR module 208 b may allow the user to dictate and have the speech transcribed into written text, synthesize a document into an audio stream, or issue commands that are recognized by the system as such written text or audio stream.

[0061] In certain embodiments, the output of the ASR module 208b may be sent to a remote natural language understanding (NLU) module 210b. In certain embodiments, the NLU module 210b may perform named entity resolution (NER), or named entity resolution may be performed by an entity resolution module 212b of a dialog manager 216b as described below. In certain embodiments, one or more of an intent, a slot, or a domain may be the output of the NLU module 210b.

[0062] In certain embodiments, the user input may include non-speech data, which may be received at a remote context engine 220b. In certain embodiments, the remote context engine 220b may determine various types of events and contexts based on the non-speech data. In certain embodiments, the output of the NLU module 210b and / or the context engine 220b may be sent to a remote dialog manager 216b.

[0063] In certain embodiments, as described above, the on-device coordinator 206 on the client system 130 can coordinate receiving the user input and can determine, at one or more decision points in the example workflow, which of the above-described operating modes should be used to process or continue processing the user input. As further described above, the selection of the operating mode can be based at least in part on the device state, the task associated with the user input, and / or one or more additional factors. By way of example and not limitation, continue to refer to Figure 2 In the workflow architecture shown in , after the entity resolution module 212a generates an output or an empty output, the on-device coordinator 206 can determine at decision point (D1) 215 whether to continue processing the user input in the first operating mode (i.e., the device mode), the second operating mode (i.e., the cloud mode), or the third operating mode (i.e., the hybrid mode). For example, at decision point (D1) 215, if the identified intent is associated with a latency-sensitive processing task (e.g., taking a photo, pausing a timer), the on-device coordinator 206 may select the first operating mode (i.e., the device mode). As another example and not limitation, if the on-device processing on the client system 130 does not support messaging tasks, the on-device coordinator 206 may select the third operating mode (i.e., the hybrid mode) to process user input associated with a messaging request. As yet another example, at decision point (D1) 215, if the task being processed requires access to a social graph, a knowledge graph, or a concept graph that is not stored on the client system 130, the on-device coordinator 206 may select the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode). Alternatively, if a sufficient version of the information graph (e.g., a small version and / or a bootstrapped version of the knowledge graph) exists on the client system 130 (the information graph includes the necessary information for the task), the on-device coordinator 206 may alternatively select the first operating mode (i.e., on-device mode).

[0064] In a specific embodiment, if Figure 2As shown, if the on-device coordinator 206 determines at decision point (D1) 215 that the first operating mode (i.e., the on-device mode) or the third operating mode (i.e., the hybrid mode) should be used to continue processing, the client-side process can continue. As an example and not a limitation, the output from the entity resolution module 212a can be sent to the on-device dialogue manager 216a. In a particular embodiment, the on-device dialogue manager 216a may include a dialogue state tracker 218a and an action selector 222a. The on-device dialogue manager 216a may have complex dialogue logic and product-related business logic to manage the dialogue state and dialogue flow between the user and the assistant system 140. The on-device dialogue manager 216a may include full functionality for end-to-end integration and multi-round support (e.g., confirmation, disambiguation). The on-device dialogue manager 216a can also be lightweight in terms of computing constraints and resources (including memory, computing (CPU), and binary size constraints). The on-device dialogue manager 216a can also be extensible to improve the developer experience. In particular embodiments, on-device dialog manager 216a can benefit assistant system 140 by, for example, providing offline support to reduce network connectivity issues (e.g., unstable or unavailable network connections), using client-side processes to prevent privacy-sensitive information from being transmitted out of client system 130, and providing a stable user experience in high-latency sensitive scenarios.

[0065] In certain embodiments, the on-device dialogue manager 216a may also perform false trigger reduction. The implementation of false trigger reduction may detect and prevent false triggers (e.g., accidental wake words) from user input that would also invoke the assistant system 140, and may further prevent the assistant system 140 from generating data records based on false triggers, which may be inaccurate and / or may be subject to privacy constraints. As an example and not limitation, if a user is in a voice call, the user's conversation during the voice call may be considered private, and false trigger reduction may limit the detection of wake words to audio user input received locally by the user's client system 130. In certain embodiments, the on-device dialogue manager 216a may implement false trigger reduction based on a nonsense detector. If the nonsense detector determines with high confidence that the received wake word is logically and / or contextually unreasonable at the point in time when the wake word is received from the user, the on-device dialogue manager 216a may determine that the user does not want to invoke the assistant system 140.

[0066] In certain embodiments, due to the limited computing power of the client system 130, the on-device dialog manager 216a may perform on-device learning based on a learning algorithm specifically customized for the client system 130. As an example and not a limitation, a federated learning technique may be implemented by the on-device dialog manager 216a. Federated learning is a specific category of distributed machine learning techniques that can train machine learning models using decentralized data stored on terminal devices (e.g., mobile phones). In certain embodiments, the on-device dialog manager 216a may use a federated user representation learning model to extend existing neural network personalization techniques to implement federated learning of the on-device dialog manager 216a. Federated user representation learning may personalize the federated learning model by learning task-specific user representations (i.e., embeddings) and / or by personalizing model weights. Federated user representation learning is simple, scalable, privacy-preserving, and resource-efficient. Federated user representation learning may separate model parameters into federated parameters and privacy parameters. Privacy parameters (e.g., private user embeddings) may be trained locally on the client system 130 instead of being transmitted to a remote server (e.g., a server associated with the assistant system 140) or averaged by a remote server. Instead, the federated parameters may be trained remotely on the server. In certain embodiments, the on-device dialog manager 216a may use an active federated learning model that may transfer a global model trained on a remote server to the client system 130 and compute gradients locally on the client system 130. Active federated learning may enable the on-device dialog manager 216a to minimize the transmission costs associated with downloading the model and uploading the gradients. For active federated learning, in each round, the client system 130 may be selected in a semi-random manner based at least in part on a probability conditioned on the current model and the data on the client system 130 to optimize the efficiency for training the federated learning model.

[0067] In certain embodiments, as the user interacts with the world and assistant system 140 interacts with the user, conversation state tracker 218a can track state changes over time. By way of example and not limitation, conversation state tracker 218a can track, for example, what the user is talking about, who the user is with, where the user is, what task is currently being performed, and where the user is looking, in accordance with applicable privacy policies.

[0068] In certain embodiments, at decision point (D1) 215, the on-device coordinator 206 may determine to forward the user input to a server in a second operating mode (i.e., cloud mode) or a third operating mode (i.e., hybrid mode). As an example and not limitation, if a particular function or process (e.g., messaging) is not supported on the client system 130, the on-device coordinator 206 may determine to use the third operating mode (i.e., hybrid mode) at decision point (D1) 215. In certain embodiments, the on-device coordinator 206 may cause the output from the NLU module 210a, the context engine 220a, and the entity resolution module 212a to be forwarded to the entity resolution module 212b of the remote dialog manager 216b through the dialog manager proxy 224 to continue the processing. The dialog manager proxy 224 may be a communication channel for exchanging information / events between the client system 130 and the server. In certain embodiments, the dialog manager 216b may also include a remote arbitrator 226b, a remote dialog state tracker 218b, and a remote action selector 222b. In certain embodiments, at decision point (D0) 205, assistant system 140 may have already begun processing user input in a second operating mode (i.e., cloud mode), and at decision point (D1) 215, on-device coordinator 206 may determine to continue processing the user input based on the second operating mode (i.e., cloud mode). Thus, outputs from NLU module 210b and context engine 220b may be received at remote entity resolution module 212b. Remote entity resolution module 212b may have similar functionality to local entity resolution module 212a, which may include resolving entities associated with slots. In certain embodiments, entity resolution module 212b may access one or more of a social graph, a knowledge graph, or a concept graph when resolving entities. Outputs from entity resolution module 212b may be received at arbitrator 226b.

[0069] In certain embodiments, the remote arbitrator 226b may be responsible for selecting between client-side upstream results and server-side upstream results (e.g., results from the NLU module 210a / 210b, results from the entity resolution module 212a / 212b, and results from the context engine 220a / 220b). The arbitrator 226b may send the selected upstream result to the remote dialog state tracker 218b. In certain embodiments, similar to the local dialog state tracker 218a, the remote dialog state tracker 218b may convert the upstream result into a candidate task using a task specification and resolve arguments using entity resolution.

[0070] In a particular embodiment, at decision point (D2) 225, the device coordinator 206 can determine whether to continue to process the user input based on the first operating mode (i.e., the on-device mode) or forward the user input to the server in the third operating mode (i.e., the hybrid mode). The decision may depend on, for example, whether the client-side process can successfully resolve the task and slot, whether there is a valid task policy with specific feature support, and / or the context difference between the client-side process and the server-side process. In a particular embodiment, the decision made at decision point (D2) 225 can be for a multi-round scenario. In a particular embodiment, there may be at least two possible scenarios. In the first scenario, the assistant system 140 may have started to process the user input in the first operating mode (i.e., the on-device mode) using the client-side dialog state. If the assistant system 140 decides at some point to switch to having the remote server process the user input, the assistant system 140 can create a programmed / predefined task with the current task state and forward the programmed / predefined task to the remote server. For subsequent rounds, the assistant system 140 can continue processing in the third operating mode (i.e., the hybrid mode) using the server-side dialog state. In another scenario, assistant system 140 may have already begun processing user input in the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode), and may rely substantially on the server-side dialog state for all subsequent turns. If on-device coordinator 206 determines to continue processing user input based on the first operating mode (i.e., device mode), output from dialog state tracker 218a may be received at action selector 222a.

[0071] In certain embodiments, at decision point (D2) 225, on-device coordinator 206 may determine to forward the user input to a remote server and continue processing the user input in a second operating mode (i.e., cloud mode) or a third operating mode (i.e., hybrid mode). Assistant system 140 may create a programmed / predefined task with a current task state and forward the programmed / predefined task to a server, which may be received at action selector 222b. In certain embodiments, assistant system 140 may have already begun processing the user input in a second operating mode (i.e., cloud mode), and on-device coordinator 206 may determine at decision point (D2) 225 to continue processing the user input in the second operating mode (i.e., cloud mode). Thus, output from dialog state tracker 218b may be received at action selector 222b.

[0072] In a particular embodiment, the action selector 222a / 222b can perform interaction management. The action selector 222a / 222b can determine and trigger a set of general executable actions. These actions can be performed on the client system 130 or on a remote server. As an example and not a limitation, these actions can include providing information or suggestions to the user. In a particular embodiment, these actions can interact with the agent 228a / 228b, the user and / or the assistant system 140 itself. These actions can include such actions: the action includes one or more of slot requests, confirmations, disambiguation or agent execution. These actions can be independent of the underlying implementation of the action selector 222a / 222b. For more complex scenarios (such as multi-round tasks or tasks with complex business logic), the local action selector 222a can call one or more local agents 228a, and the remote action selector 222b can call one or more remote agents 228b to perform these actions. Agent 228a / 228b can be called by task ID, and any action can be routed to the correct agent 228a / 228b using the task ID. In a specific embodiment, agent 228a / 228b can be configured to act as a broker between multiple content providers of a domain. A content provider can be an entity responsible for performing an action associated with an intent or completing a task associated with the intent. In a specific embodiment, agent 228a / 228b can provide several functions for assistant system 140, including, for example, local template generation, task-specific business logic, and querying external APIs. Agent 228a / 228b can use the context from dialog state tracker 218a / 218b when performing the action of a task, and can also update dialog state tracker 218a / 218b. In a specific embodiment, agent 228a / 228b can also generate a partial payload from a dialog action.

[0073] In a particular embodiment, the local agent 228a may have different implementations compiled / registered for different platforms (e.g., smart glasses and VR head-mounted viewer). In a particular embodiment, multiple device-specific implementations (e.g., real-time calls to the client system 130 or a messaging application on the client system 130) may be handled internally by a single agent 228a. Alternatively, the device-specific implementation may be handled by multiple agents 228a associated with multiple domains. As an example and not limitation, calling the agent 228a on the smart glasses may be implemented in a different manner than calling the agent 228a on the smartphone. Different platforms may also use different numbers of agents 228a. Agent 228a may also be cross-platform (i.e., different operating systems on the client system 130). In addition, agent 228a may have a minimized startup time or binary size impact. The local agent 228a may be suitable for specific use cases. As an example and not limitation, one use case may be an emergency call to the client system 130. As another example and not limitation, another use case may be responding to user input without a network connection. As another example and not limitation, another use case may be that certain domains / tasks may be privacy sensitive and sending user input to a remote server may be prohibited.

[0074] In a particular embodiment, the local action selector 222a can call the local delivery system 230a to perform the action, and the remote action selector 222b can call the remote delivery system 230b to perform the action. Once the delivery system 230a / 230b receives the trigger signal from the conversation state tracker 218a / 218b, it can deliver the predefined event by performing the corresponding action. The delivery system 230a / 230b can ensure that the event is delivered to the host with an active connection. As an example and not limitation, the delivery system 230a / 230b can broadcast to all online devices belonging to a user. As another example and not limitation, the delivery system 230a / 230b can deliver the event to a target-specific device. The delivery system 230a / 230b can also use the latest device context to render the payload.

[0075] In a particular embodiment, the on-device dialog manager 216a may also include a separate local action execution module, and the remote dialog manager 216b may also include a separate remote action execution module. The local execution module and the remote action execution module may have similar functions. In a particular embodiment, the action execution module may call the agent 228a / 228b to perform a task. The action execution module may also execute a set of general executable actions determined by the action selector 222a / 222b. The set of executable actions may interact with the agent 228a / 228b, the user, and the assistant system 140 itself through the delivery system 230a / 230b.

[0076] In a particular embodiment, if the user input is processed using the first operating mode (i.e., the on-device mode), the results from the agent 228a and / or the delivery system 230a can be returned to the device dialogue manager 216a. The on-device dialogue manager 216a can then instruct the local arbitrator 226a to generate a final response based on these results. The arbitrator 226a can summarize these results and evaluate them. As an example and not a limitation, the arbitrator 226a can sort the results in response to the user input and select the best result. If the user request is processed in the second operating mode (i.e., the cloud mode), the results from the agent 228b and / or the delivery system 230b can be returned to the remote dialogue manager 216b. The remote dialogue manager 216b can instruct the arbitrator 226a to generate a final response based on these results through the dialogue manager agent 224. Similarly, the arbitrator 226a can analyze these results and select the best result to provide to the user. If the user input is processed based on the third operating mode (i.e., hybrid mode), both the client-side results and the server-side results (e.g., from the agent 228a / 228b and / or the delivery system 230a / 230b) can be provided to the arbitrator 226a through the on-device dialog manager 216a and the remote dialog manager 216b, respectively. The arbitrator 226a can then select between the client-side results and the server-side results to determine the final results to be presented to the user. In a particular embodiment, the logic for deciding between these results can depend on the particular use case.

[0077] In a particular embodiment, the local arbitrator 226a may generate a response based on the final result and send the response to the rendering output module 232. The rendering output module 232 may determine how to render the output in a manner suitable for the client system 130. As an example and not limitation, for a VR head-mounted viewer or AR smart glasses, the rendering output module 232 may determine to render the output using a visual-based modality (e.g., an image or video clip), which may be displayed by the VR head-mounted viewer or AR smart glasses. As another example, the response may be rendered as an audio signal that may be played by the user through the VR head-mounted viewer or AR smart glasses. As yet another example, the response may be rendered as augmented reality data for enhancing the user experience.

[0078] In certain embodiments, in addition to determining the operating mode for processing user input, the on-device coordinator 206 may also determine whether to process the user input on the rendering device 137, process the user input on the companion device 138, or process the user request on a remote server. The rendering device 137 and / or the companion device 138 may each use the assistant stack to process the user input in a manner similar to that disclosed above. As an example and not limitation, the on-device coordinator 206 may determine that a portion of the processing should be completed on the rendering device 137, a portion of the processing should be completed on the companion device 138, and the remainder of the processing should be completed on the remote server.

[0079] In certain embodiments, assistant system 140 may have various capabilities including audio cognition, visual cognition, signal intelligence, reasoning, and memory. In certain embodiments, the audio cognition capability may enable assistant system 140 to, for example, understand user input associated with various domains of different languages, understand and summarize conversations, perform on-device audio cognition for complex commands, recognize users by voice, extract topics from conversations and auto-tagged portions of conversations, enable audio interactions without a wake word, filter and amplify user voice from ambient noise and conversations, and / or understand which client system 130 a user is talking to when multiple client systems 130 are nearby.

[0080] In certain embodiments, the capabilities of visual cognition may enable assistant system 140 to: for example, identify interesting objects in the world through a combination of existing machine learning models and one-shot learning, identify interesting moments and automatically capture them, achieve semantic understanding across multiple visual frames spanning different time segments, provide platform support for additional capabilities in place recognition or object recognition, recognize a full range of settings and micro-locations including personalized locations, recognize complex activities, recognize complex gestures for controlling client system 130, process images / videos (e.g., with motion, capture angles, resolution) from egocentric cameras with similar accuracy and speed levels associated with images with lower resolution, perform one-shot registration and recognition of places and objects, and / or perform visual recognition on client system 130.

[0081] In certain embodiments, the assistant system 140 may utilize computer vision techniques to achieve visual cognition. In addition to computer vision techniques, the assistant system 140 may also explore options that can supplement these techniques to expand the recognition of objects. In certain embodiments, the assistant system 140 may use supplementary signals, such as optical character recognition (OCR) of object labels, GPS signals for location recognition, and / or signals from the user's client system 130 for identifying the user. In certain embodiments, the assistant system 140 may perform general scene recognition (e.g., home space, work space, public space) to set the user's context and narrow the computer vision search space to identify possible objects or people. In certain embodiments, the assistant system 140 may guide users to train the assistant system 140. For example, crowdsourcing may be used to allow users to tag objects and help the assistant system 140 recognize more objects over time. As another example, when using the assistant system 140, users may register their personal objects as part of the initial setup. The assistant system 140 may also allow users to provide positive / negative signals for the objects they interact with to train and improve their personalized models.

[0082] In particular embodiments, signals intelligence capabilities may enable assistant system 140 to, for example, determine user location, understand date / time, determine home location, understand the user's schedule and future destinations, integrate richer voice understanding to identify settings / context by voice alone, and / or build signals intelligence models that can be personalized at runtime for a user's individual routines.

[0083] In certain embodiments, reasoning capabilities may enable assistant system 140 to, for example, extract previous conversation threads at any point in the future, synthesize all signals to understand micro-context and personalized context, learn interaction patterns and preferences based on a user's historical behavior and accurately suggest interactions that they may value, generate highly predictive proactive suggestions based on micro-context understanding, understand what content a user may want to watch at what time of day, and / or understand changes in the context and how that change may affect what the user wants.

[0084] In certain embodiments, memory capabilities may enable assistant system 140 to, for example, remember social connections that a user has previously visited or interacted with, write to memory and query memory at will (i.e., open dictation and automatic tagging), extract richer preferences based on previous interactions and long-term learning, remember the user's life history, extract rich information from egocentric data streams and automatic catalogs, and / or write to memory in a structured form to form rich short-term memory, episodic memory, and long-term memory.

[0085] Figure 3 An example flow chart 300 of an assistant system 140 is shown. In certain embodiments, the assistant service module 305 may access a request manager 310 upon receiving user input. In certain embodiments, the request manager 310 may include a context extractor 312 and a conversational understanding (CU) object generator (CU object generator) 314. The context extractor 312 may extract context information associated with the user input. The context extractor 312 may also update the context information based on the assistant application 136 executed on the client system 130. As an example and not limitation, the update to the context information may include displaying a content item on the client system 130. As another example and not limitation, the update to the context information may include whether a reminder is set on the client system 130. As another example and not limitation, the update to the context information may include whether a song is playing on the client system 130. The CU object generator 314 may generate a specific CU object related to the user input. The CU object may include conversational session data and features associated with the user input, which may be shared with all modules of the assistant system 140. In particular embodiments, request manager 310 may store the context information and the generated CU objects in data store 320 , which is a particular data store implemented in assistant system 140 .

[0086] In a particular embodiment, the request manager 310 may send the generated CU object to the NLU module 210. The NLU module 210 may perform multiple steps to process the CU object. The NLU module 210 may first traverse the CU object in the allow list / block list 330. In a particular embodiment, the allow list / block list 330 may include interpretation data that matches the user input. Then, the NLU module 210 may perform characterization 332 of the CU object. Then, the NLU module 210 may perform domain classification / selection 334 on the user input based on the features generated by the characterization 332 to classify the user input into a predefined domain. In a particular embodiment, the domain may represent a social context of interaction (e.g., education), or a namespace of a set of intents (e.g., music). The domain classification / selection results may be further processed based on two related processes. In one process, the NLU module 210 may use a meta-intent classifier 336a to process the domain classification / selection results. The meta-intent classifier 336a may determine a category that describes the user intent. An intent may be an element in a predefined semantic intent classification that may indicate the purpose of a user's interaction with the assistant system 140. The NLU module 210 may classify the user input as a member of a predefined classification. For example, the user input may be "play Beethoven's Fifth," and the NLU module 210 may classify the input as having the intent [IN:play_music]. In a particular embodiment, intents common to multiple domains may be processed by a meta-intent classifier 336a. As an example and not limitation, the meta-intent classifier 336a may be based on a machine learning model that may take a domain classification / selection result as input and calculate the probability that the input is associated with a particular predefined meta-intent. The NLU module 210 may then use a meta-slot annotator 338a to annotate one or more meta-slots for the classification results from the meta-intent classifier 336a. A slot may be a named substring corresponding to a string representing a basic semantic entity within the user input. For example, the slot for "pizza" may be [SL:food] ([SL:dish]). In particular embodiments, a set of valid or expected named slots may be conditional on the intent being classified. As an example and not limitation, for the intent [IN: play music], a valid slot may be [SL: song_name]. In particular embodiments, the meta-slot tagger 338a may tag common slots, such as references to items (e.g., first), types of slots, values ​​of slots, etc. In particular embodiments, the NLU module 210 may use an intent classifier 336b to process domain classification / selection results. The intent classifier 336b may determine a user intent associated with a user input. In particular embodiments, for each domain, there may be an intent classifier 336b to determine the most likely intent in a given domain.As an example and not limitation, the intent classifier 336b can be based on a machine learning model that can take the domain classification / selection result as input and calculate the probability that the input is associated with a specific predefined intent. Then, the NLU module 210 can use the slot tagger 338b to tag one or more slots associated with the user input. In a specific embodiment, the slot tagger 338b can tag one or more slots for the n-gram of the user input. As an example and not limitation, the user input can include "convert 500 US dollars in my account to Japanese yen". The intent classifier 336b can take the user input as input and represent it as a vector. Then, the intent classifier 336b can calculate the probability that the user input is associated with different predefined intents based on a vector comparison between the vector representing the user input and the vector representing different predefined intents. In a similar manner, the slot tagger 338b can take the user input as input and represent each word as a vector. Then, the slot tagger 338b can calculate the probability that each word is associated with different predefined slots based on a vector comparison between the vector representing the word and the vector representing different predefined slots. The user's intent may be classified as "Exchange currency". The user's input slots may include "500", "US dollars", "account", and "Japanese yen". The user's meta-intent may be classified as "Financial services". The meta-slot may include "Finance".

[0087] In certain embodiments, the natural language understanding (NLU) module 210 may also extract information from one or more of a social graph, a knowledge graph, or a concept graph, and may retrieve a user profile stored locally on the client system 130. The NLU module 210 may also consider contextual information when analyzing user input. The NLU module 210 may further process information from these different sources by identifying and summarizing information, annotating n-grams of user input, ranking n-grams with confidence scores based on the summarized information, and representing the ranked n-grams as features that can be used by the NLU module 210 to understand the user input. In certain embodiments, the NLU module 210 may identify one or more of domains, intents, or slots from the user input in a personalized and context-aware manner. As an example and not limitation, the user input may include "tell me how to get to a coffee shop." The NLU module 210 may identify a specific coffee shop that the user wants to go to based on the user's personal information and associated contextual information. In certain embodiments, the NLU module 210 may include a dictionary for a specific language, a parser, and grammatical rules that divide sentences into internal representations. The NLU module 210 may also include one or more programs that perform naive semantics or stochastic semantic analysis, and may also use pragmatics to understand user input. In a particular embodiment, the parser may be based on a deep learning architecture that includes multiple long-short term memory (LSTM) networks. As an example and not limitation, the parser may be based on a recurrent neural network grammar (RNNG) model, which is a recursive and cyclic LSTM algorithm. More information about natural language understanding (NLU) can be found in the following documents: U.S. Patent Application No. 16 / 011062, filed on June 18, 2018, U.S. Patent Application No. 16 / 025317, filed on July 2, 2018, and U.S. Patent Application No. 16 / 038120, filed on July 17, 2018, each of which is incorporated by reference.

[0088] In a particular embodiment, the output of the NLU module 210 may be sent to the entity resolution module 212 to resolve related entities. Entities may include, for example, unique users or concepts, each of which may have a unique identifier (ID). These entities may include one or more of the following: real-world entities (from a general knowledge base), user entities (from a user memory), contextual entities (device context / conversation context), or value resolution (numbers, date time, etc.). In a particular embodiment, the entity resolution module 212 may include domain entity resolution 340 and general entity resolution 342. The entity resolution module 212 may perform general entity resolution and domain-specific entity resolution. General entity resolution 342 may resolve entities by classifying slots and meta-slots into different general topics. Domain entity resolution 340 may resolve entities by classifying slots and meta-slots into different domains. As an example and not limitation, in response to the input of a query for the advantages of a particular brand of electric vehicles, general entity resolution 342 may resolve the referenced electric vehicle brand as a vehicle, and domain entity resolution 340 may resolve the referenced electric vehicle brand as an electric vehicle.

[0089] In a particular embodiment, entities can be parsed based on knowledge 350 about the world and the user. Assistant system 140 can extract ontology data from graph 352. As an example and not limitation, graph 352 can include one or more of a knowledge graph, a social graph, or a concept graph. Ontology data can include structural relationships between different slots / meta slots and domains. Ontology data can also include information such as how slots / meta slots can be grouped, how slots / meta slots can be related within a hierarchy (wherein a higher level includes a domain), and how slots / meta slots can be subdivided according to similarities and differences. For example, a knowledge graph can include multiple entities. Each entity can include a single record associated with one or more attribute values. The particular record can be associated with a unique entity identifier. Each record can have different attribute values ​​for the entity. Each attribute value can be associated with a confidence probability and / or a semantic weight. The confidence probability for an attribute value indicates: for a given attribute, the probability that the value is accurate. The semantic weight for an attribute value can indicate: considering all available information, how semantically appropriate the value is for a given attribute. For example, a knowledge graph may include an entity for a book named "BookName", which may include information extracted from multiple content sources (e.g., online social networks, online encyclopedias, book review sources, media databases, and entertainment content sources), which may be deduplicated, parsed, and fused to generate a single unique record of the knowledge graph. In this example, an entity named "BookName" may be associated with a "fantasy" attribute value of a "genre" entity attribute. More information about knowledge graphs can be found in the following documents: U.S. Patent Application No. 16 / 048049, filed on July 27, 2018, and U.S. Patent Application No. 16 / 048101, filed on July 27, 2018, each of which is incorporated by reference.

[0090] In certain embodiments, assistant user memory (AUM) 354 may include user context memory that helps determine how to more effectively assist users. AUM 354 may be a central location for storing user data, retrieving user data, indexing user data, and searching user data. By way of example and not limitation, AUM 354 may store information such as contacts, photos, reminders, and the like. In addition, AUM 354 may automatically synchronize data to servers and other devices (only for non-sensitive data). By way of example and not limitation, if a user sets a nickname for a contact on one device, all devices may synchronize and obtain the nickname based on AUM 354. In certain embodiments, AUM 354 may first prepare events, user status, reminders, and trigger status for storage in a data repository. A memory node identifier (ID) may be created to store an entry object in AUM 354, where an entry may be a piece of information about a user (e.g., a photo, reminder, etc.). By way of example and not limitation, the first few bits of a memory node ID may indicate that this is a memory node ID type, the last few bits may be a user ID, and the last few bits may be a creation time. AUM 354 can then index this data so that it can be retrieved as needed. For this purpose, an index ID can be created. In a specific embodiment, given an "index keyword" (e.g., PHOTO_LOCATION) and an "index value" (e.g., "San Francisco"), AUM 354 can obtain a list of memory IDs with this attribute (e.g., photos in San Francisco). As an example and not limitation, the first few bits can indicate that this is an index ID type, the second few bits can be a user ID, and the third few bits can encode the "index keyword" and "index value". AUM 354 can also use a flexible query language for information retrieval. For this purpose, a relationship index ID can be created. In a specific embodiment, given a source memory node and edge type, AUM 354 can obtain the memory IDs of all target nodes with outgoing edges of this type from the source. As an example and not limitation, the first few bits can indicate that this is a relationship index ID type, the second few bits can be a user ID, and the third few bits can be the source node ID and edge type. In a specific embodiment, AUM 354 can help detect concurrent updates of different events. More information about episodic memory can be found in U.S. patent application Ser. No. 16 / 552,559, filed Aug. 27, 2019, which is incorporated by reference.

[0091] In certain embodiments, the entity resolution module 212 may use different techniques to resolve different types of entities. For real-world entities, the entity resolution module 212 may use a knowledge graph to resolve the scope of an entity, such as "music track", "movie", etc. For user entities, the entity resolution module 212 may use user memory or some proxy to resolve the scope to a user-specific entity, such as "contact", "reminder", or "relationship". For contextual entities, the entity resolution module 212 may perform coreference based on information from the context engine 220 to resolve references to entities in the context, such as "he", "she", "first", or "last". In certain embodiments, for coreference, the entity resolution module 212 may create references for entities determined by the NLU module 210. The entity resolution module 212 may then accurately resolve these references. By way of example and not limitation, user input may include "find me the nearest grocery store and direct me there". Based on coreference, the entity resolution module 212 may interpret "there" as "the nearest grocery store". In certain embodiments, coreference may depend on information from the context engine 220 and the dialog manager 216 to interpret references with increased accuracy. In certain embodiments, the entity resolution module 212 may also resolve entities based on context (device context or dialog context), such as entities shown on the screen or entities from the last dialog history. For value resolution, the entity resolution module 212 may resolve the mentioned value into a precise value in a standard form, such as a numeric value, date time, address, etc.

[0092] In certain embodiments, the entity resolution module 212 may first perform a check on applicable privacy constraints to ensure that performing entity resolution does not violate any applicable privacy policies. By way of example and not limitation, the entity to be resolved may be another user who has specified in its privacy settings that its identity should not be searchable on an online social network. In this case, the entity resolution module 212 may not return an entity identifier for the user in response to the user input. By leveraging the described information obtained from the social graph, knowledge graph, concept graph, and user profile, and by complying with any applicable privacy policies, the entity resolution module 212 may resolve entities associated with the user input in a personalized, context-aware, and privacy-preserving manner.

[0093] In a particular embodiment, the entity resolution module 212 can work with the ASR module 208 to perform entity resolution. The following example shows how the entity resolution module 212 can resolve entity names. The entity resolution module 212 can first expand the name associated with the user into its corresponding normalized text form as a phonetic consonant representation, which can be phonetically transcribed using a double metaphone algorithm. Then, the entity resolution module 212 can determine the n-best set of candidate transcriptions and perform a parallel understanding process on all phonetic transcriptions in the n-best set of candidate transcriptions. In a particular embodiment, each transcription that is parsed to the same intent can then be transformed (collapsed) into a single intent. Then, each intent can be assigned a score corresponding to the highest scoring candidate transcription of the intent. During the transition, the entity resolution module 212 can identify various possible text transcriptions associated with each slot, which are associated by boundary timing offsets associated with the transcription of the slot. Then, the entity resolution module 212 can extract a subset of possible candidate transcriptions for each slot from multiple (e.g., 1000) candidate transcriptions, regardless of whether they are classified into the same intent. In this way, slots and intents can be a scored list of phrases. In certain embodiments, a new task or an ongoing task that can handle the intent (e.g., a message composition task for an intent to send a message to another user) can be identified and associated with the intent. Figure 1The identified task can then trigger the entity resolution module 212 by providing the entity resolution module 212 with a score list of phrases associated with one of its slots and the category for which it should be resolved. As an example and not limitation, if the entity attribute is specified as "friend", the entity resolution module 212 can traverse each candidate list of items in the same extension that can be run when the matcher is compiled. Each candidate extension of the item can be matched in a pre-compiled trie matching structure. Matches can be scored using a function based at least in part on the transcribed input, the form of the match, and the friend's name. As another example and not limitation, if the entity attribute is specified as "celebrity / prominent person", the entity resolution module 212 can perform a parallel search for the knowledge graph for each candidate item set of the slot output from the ASR module 208. The entity resolution module 212 can score the match based on the popularity of the person matched and the score signal provided by the ASR. In a specific embodiment, when a memory category is specified, the entity resolution module 212 can perform the same search for the user memory. The entity resolution module 212 can crawl backwards in the user's memory and try to match each memory (e.g., a person recently mentioned in a conversation, or a person seen and recognized through a visual signal, etc.). For each entity, the entity resolution module 212 can use a match similar to how it matches friends (i.e., voice). In a particular embodiment, the score can include a time decay factor associated with the most recent time the name was previously mentioned. The entity resolution module 212 can also combine, sort, and deduplicate all matches. In a particular embodiment, a task can receive a candidate set. When there are multiple high-scoring candidates, the entity resolution module 212 can perform user-facilitated disambiguation (e.g., obtaining real-time user feedback on these candidates from the user).

[0094] In a particular embodiment, the context engine 220 can help the entity resolution module 212 improve entity resolution. The context engine 220 may include an offline aggregator and an online reasoning service. The offline aggregator can process multiple data associated with the user collected from a previous time window. As an example and not limitation, the data may include news push posts / comments collected within a predetermined time frame (e.g., from a previous 90-day window), interactions with news push posts / comments, search history, etc. The processing results may be stored in the context engine 220 as part of the user profile. The user profile of the user may include user profile data, which includes personal background information, social information, and context information associated with the user. The user profile data may also include user interests and preferences on multiple topics aggregated through conversations on news push, search logs, messaging platforms, etc. The use of the user profile may be subject to privacy constraints to ensure that the user's information can only be used for his / her benefit and cannot be shared with anyone else. More information about user profiles can be found in the following documents: U.S. Patent Application No. 15 / 967,239, filed on April 30, 2018, which is incorporated by reference. In certain embodiments, the online reasoning service can analyze the conversation data associated with the user received by the assistant system 140 at the current time. The analysis results can also be stored in the context engine 220 as part of the user profile. In certain embodiments, both the offline aggregator and the online reasoning service can extract personalized features from multiple data. Other modules of the assistant system 140 can use the extracted personalized features to better understand the user input. In certain embodiments, the entity resolution module 212 can process the information (e.g., user profile) from the context engine 220 in the following steps based on natural language processing (NLP). In certain embodiments, the entity resolution module 212 can tag the text by text normalization, extract syntactic features from the text, and extract semantic features from the text based on NLP. The entity resolution module 212 can also extract features from context information, which is accessed from the conversation history between the user and the assistant system 140. The entity resolution module 212 can also perform global word embedding, domain-specific embedding, and / or dynamic embedding based on context information. The processing results can be annotated with entities by an entity tagger. Based on the annotations, the entity resolution module 212 can generate a dictionary. In a specific embodiment, the dictionary can include global dictionary features that can be dynamically updated offline. The entity resolution module 212 can sort the entities marked by the entity tagger. In a specific embodiment, the entity resolution module 212 can communicate with different graphs 352 including one or more of a social graph, a knowledge graph, or a concept graph to extract ontology data related to the information retrieved from the context engine 220.In certain embodiments, the entity resolution module 212 can also resolve entities based on the user profile, the ranked entities, and information from the graph 352 .

[0095] In a particular embodiment, the entity resolution module 212 can be driven by a task (corresponding to the agent 228). This reversal of the processing order can make it possible for the domain knowledge present in the task to be applied to pre-filter or bias a set of resolution targets when it is obvious and appropriate to do so. As an example and not limitation, for the utterance "Who is John?", there is no explicit category implied in the utterance. Therefore, the entity resolution module 212 can resolve "John" for all content. As another example and not limitation, for the utterance "Send a message to John", the entity resolution module 212 can easily determine that "John" refers to a person who can send and receive messages. Therefore, the entity resolution module 212 can bias the resolution towards friends. As another example and not limitation, for "What is John's most famous album?" In order to resolve "John", the entity resolution module 212 can first determine the task corresponding to the utterance, that is, to find music albums. The entity resolution module 212 can determine that entities related to music albums include singers, producers, and recording studios. Therefore, the entity resolution module 212 can search in these types of entities in the music domain to resolve "John".

[0096] In a particular embodiment, the output of the entity resolution module 212 can be sent to the dialogue manager 216 to advance the dialogue flow with the user. The dialogue manager 216 can be an asynchronous state machine that repeatedly updates the state and selects an action based on the new state. The dialogue manager 216 can also store previous dialogues between the user and the assistant system 140. In a particular embodiment, the dialogue manager 216 can perform dialogue optimization. Dialogue optimization involves understanding and identifying the challenge of the most likely branch options in the dialogue with the user. As an example and not a limitation, the assistant system 140 can implement dialogue optimization techniques to avoid the need to confirm who the user wants to call, because the assistant system 140 can determine the high confidence that the person inferred based on the context and available data is the intended recipient. In a particular embodiment, the dialogue manager 216 can implement a reinforcement learning framework to improve dialogue optimization. The dialogue manager 216 can include a dialogue intent parsing 356, a dialogue state tracker 218, and an action selector 222. In a particular embodiment, the dialogue manager 216 can perform the selected action and then call the dialogue state tracker 218 again until the selected action requires a user response or there is no more action to perform. Each action selected may depend on the results of the execution of previous actions. In a particular embodiment, the conversation intent parsing 356 may parse the user intent associated with the current conversation session based on the conversation history between the user and the assistant system 140. The conversation intent parsing 356 may map the intent determined by the NLU module 210 to different conversation intents. The conversation intent parsing 356 may also sort the conversation intents based on signals from the NLU module 210, the entity resolution module 212, and the conversation history between the user and the assistant system 140.

[0097] In a particular embodiment, the dialog state tracker 218 can use a set of operators to track the dialog state. The operator can include the data and logic required to update the dialog state. Each operator can act as a delta of the dialog state after processing the user input just received. In a particular embodiment, the dialog state tracker 218 can include a task tracker, which can be based on task specifications and different rules. The dialog state tracker 218 can also include a slot tracker and a co-referencing component, which can be based on rules and / or based on recent time. The co-referencing component can help the entity resolution module 212 to resolve entities. In an alternative embodiment, using the co-referencing component, the dialog state tracker 218 can replace the entity resolution module 212, and can resolve any reference / mention and keep tracking state. In a particular embodiment, the dialog state tracker 218 can use the task specification to convert the upstream results into candidate tasks, and use entity resolution to resolve parameters. Both user status (e.g., the user's current activity) and task status (e.g., trigger conditions) can be tracked. Given the current state, the dialog state tracker 218 can generate candidate tasks that the assistant system 140 can process and perform for the user. By way of example and not limitation, the candidate tasks can include "give advice", "get weather information", or "take a photo". In certain embodiments, the dialog state tracker 218 can generate candidate tasks based on available data from, for example, a knowledge graph, user memory, and user task history. In certain embodiments, the dialog state tracker 218 can then use the parsed parameters to parse the trigger object. By way of example and not limitation, the user input "remind me to call mom tonight when she is online and I am at home" can be converted from NLU output to trigger representation by the dialog state tracker 218, as shown in Table 1 below:

[0098]

[0099] Table 1: Example of transformation from NLU output to trigger representation

[0100] In the above example, "Mom", "Home", and "Tonight" are represented by their respective entities: personEntity, locationEntity, datetimeEntity.

[0101] In certain embodiments, the dialog manager 216 may map events determined by the context engine 220 to actions. By way of example and not limitation, an action may be a natural language generation (NLG) action, a display or overlay, a device action, or a retrieval action. The dialog manager 216 may also perform context tracking and interaction management. Context tracking may include aggregating real-time event streams into a unified user state. Interaction management may include selecting the best action in each state. In certain embodiments, the dialog state tracker 218 may perform context tracking (i.e., tracking events related to the user). To support processing of the event stream, the dialog state tracker 218a may use event processors (e.g., for disambiguation, confirmation, request) that may consume various types of events and update the internal assistant state. There may be one or more processors for each event type. Each event processor may be modifying a certain fragment of the assistant state. In certain embodiments, the event processors may be operating on a disjoint subset of the state (i.e., only one processor may have write access to a specific field in the state). In certain embodiments, all event processors may have the opportunity to process a given event. By way of example and not limitation, dialog state tracker 218 may run all event handlers in parallel on each event, and may then merge the state updates proposed by the various event handlers (eg, for each event, a majority of handlers may return a NULL update).

[0102] In certain embodiments, the dialog state tracker 218 can be used as any programming processor (logic) that needs to be versioned. In certain embodiments, instead of directly changing the dialog state, the dialog state tracker 218 can be a marginal effect-independent component and can generate the n best candidates for the dialog state update operator (which proposes an update to the dialog state). The dialog state tracker 218 can include an intent parser that contains logic to handle different types of NLU intents and generate operators based on the dialog state. In certain embodiments, the logic can be organized by an intent processor, such as a disambiguation intent processor for processing intents when the assistant system 140 requests disambiguation, a confirmation intent processor including the logic for processing confirmation, etc. The intent parser can combine the turn intent with the dialog state to generate a contextual update of the dialog with the user. Then, the slot resolution component can recursively resolve the slots in the update operator using a resolution provider including a knowledge graph and a domain agent. In certain embodiments, the dialog state tracker 218 can update / sort the dialog state of the current dialog session. As an example and not limitation, if the dialog session ends, the dialog state tracker 218 may update the dialog state to “complete.” As another example and not limitation, the dialog state tracker 218 may sort the dialog states based on the priorities associated with the dialog states.

[0103] In certain embodiments, the dialog state tracker 218 may communicate with the action selector 222 about the dialog intent and the associated content object. In certain embodiments, the action selector 222 may rank different dialog hypotheses for different dialog intents. The action selector 222 may employ candidate operators for the dialog state and query the dialog strategy 360 to determine what action should be performed. In certain embodiments, the dialog strategy 360 may be a tree-based strategy, which is a pre-built dialog plan. Based on the current dialog state, the dialog strategy 360 may select a node to execute and generate a corresponding action. As an example and not limitation, a tree-based strategy may include a topic grouping node and a dialog action (leaf) node. In certain embodiments, the dialog strategy 360 may also include a data structure describing the execution plan of the agent 228 for the action. The dialog strategy 360 may also include multiple targets that are related to each other by logical operators. In certain embodiments, the target may be the result of a portion of the dialog strategy, and the target may be constructed by the dialog manager 216. The target may be represented by an identifier (e.g., a string) having one or more named parameters that parameterize the target. By way of example and not limitation, a goal with its associated goal parameters may be represented as {confirm_artist, parameters: {artist: “artist name”}}. In particular embodiments, goals may be mapped to leaves of a tree of a tree structure representation of dialog policies 360.

[0104] In certain embodiments, assistant system 140 may use hierarchical dialog strategies 360, where generic strategies 362 handle cross-domain business logic and task strategies 364 handle task / domain specific logic. Generic strategies 362 may be used for actions that are not specific to individual tasks. Generic strategies 362 may be used to determine task stacking and switching, active tasks, notifications, etc. Generic strategies 362 may include: handling low confidence intents, internal errors, unacceptable user responses with retries, and / or skipping or inserting confirmations based on ASR or NLU confidence scores. Generic strategies 362 may also include logic to sort dialog state update candidates from the output of dialog state tracker 218, and may pick one to update (e.g., pick the first ranked task intent). In certain embodiments, assistant system 140 may have a specific interface for generic strategies 362 that allows decentralized cross-domain policies / business rules (particularly those found in dialog state tracker 218) to be merged into the functionality of action selector 222. The interface for the generic policy 362 may also allow the creation of independent sub-policy units that can be bound to specific situations or clients (e.g., policy functionality that can be easily turned on or off based on the client, situation). The interface for the generic policy 362 may also allow for the provision of policy hierarchies (i.e., multiple policy units) with back-off, where highly specialized policy units that handle specific situations are backed up by more generic policies 362 that apply to a wider range of situations. In this case, the generic policy 362 may instead include intent or task-specific policies.

[0105] In certain embodiments, task policies 364 may include logic for action selectors 222 based on the task and the current state. Task policies 364 may be dynamic and self-organizing. In certain embodiments, types of task policies 364 may include one or more of the following: (1) manually created tree-based dialogue plans; (2) coded policies that directly implement an interface for generating actions; (3) slot-filling tasks specified by a configurator; or (4) policies based on machine learning models learned from data. In certain embodiments, assistant system 140 may use rule-based logic to bootstrap new domains and then use machine learning models to refine task policies 364. In certain embodiments, general policies 362 may select an operator from candidate operators to update the dialogue state, and then task policies 364 may select a user-oriented action. Once a task is active in a dialogue state, the corresponding task policy 364 may be queried to select the correct action.

[0106] In certain embodiments, the action selector 222 may select an action based on one or more of the following: an event determined by the context engine 220, a dialog intent and state, an associated content object, and guidance from a dialog strategy 360. Each dialog strategy 360 may subscribe to a specific condition on a state field. After the event is processed and the state is updated, the action selector 222 may run a fast search algorithm (e.g., similar to Boolean satisfiability) to identify which strategies should be triggered based on the current state. In certain embodiments, if multiple strategies are triggered, the action selector 222 may use a tie-breaker mechanism to select a specific strategy. Alternatively, the action selector 222 may use a more sophisticated approach as follows: the approach may preview each strategy and then select a specific strategy that may be determined to have a high probability of success. In certain embodiments, mapping events to actions may bring several technical advantages to the assistant system 140. One technical advantage may include that each event may be a state update from the user or the user's physical / digital environment that may or may not trigger an action from the assistant system 140. Another technical advantage may include the possibility to handle fast bursts of events (e.g., a user enters a new building and sees many people) by first consuming all events to update the state, and then triggering one or more actions based on the final state. Another technical advantage may include consuming all events into a single global assistant state.

[0107] In certain embodiments, action selector 222 may take a dialog state update operator as part of the input to select a dialog action. Execution of a dialog action may generate a set of expectations to instruct dialog state tracker 218 to process future turns. In certain embodiments, expectations may be used to provide context to dialog state tracker 218 when processing user input from the next turn. As an example and not limitation, a slot request dialog action may have an expectation to verify the value of the requested slot. In certain embodiments, both dialog state tracker 218 and action selector 222 may not change the dialog state before executing the selected action. This may allow assistant system 140 to execute dialog state tracker 218 and action selector 222 for processing inferred ASR results and performing n-best ranking using previews.

[0108] In a particular embodiment, the action selector 222 may call different agents 228 to perform the task. Meanwhile, the dialog manager 216 may receive an instruction to update the dialog state. As an example and not a limitation, the update may include waiting for a response from the agent 228. The agent 228 may select from among the registered content providers to complete the action. The data structure may be constructed by the dialog manager 216 based on the intent and one or more slots associated with the intent. In a particular embodiment, the agent 228 may include a first-party agent and a third-party agent. In a particular embodiment, the first-party agent may include an internal agent (e.g., an agent associated with a service provided by an online social network (e.g., a messaging service or a photo sharing service)) that may be accessed and controlled by the assistant system 140. In a particular embodiment, the third-party agent may include an external agent (e.g., a third-party online music application agent, a ticketing agent) that the assistant system 140 cannot control. The first-party agent may be associated with a first-party provider that provides content objects and / or services hosted by the social networking system 160. The third-party agent may be associated with a third-party provider that provides content objects and / or services hosted by a third-party system 170. In a particular embodiment, each of the first-party agent or the third-party agent may be specified for a particular domain. By way of example and not limitation, domains may include weather, traffic, music, shopping, social, video, photo, event, location, and / or work. In certain embodiments, assistant system 140 may collaboratively use multiple agents 228 to respond to user input. By way of example and not limitation, user input may include “direct me to my next meeting.” Assistant system 140 may use a calendar agent to retrieve the location of the next meeting. Assistant system 140 may then use a navigation agent to direct the user to the next meeting.

[0109] In a particular embodiment, the dialog manager 216 may support multiple rounds of compositional parsing of slot mentions. For compositional parsing from the NLU module 210, the parser may recursively parse nested slots. The dialog manager 216 may also support disambiguation of nested slots. As an example and not limitation, the user input may be "remind me to call Alex". The parser may need to know which Alex to call before creating an actionable reminder to-do entity. When further user clarification is necessary for a particular slot, the parser may pause parsing and set the parsing state. The general strategy 362 may check the parsing state and create corresponding dialogue actions for user clarification. The dialog manager 216 may update the nested slots in the dialog state tracker 218 based on the user input and the most recent dialogue actions. This functionality may allow the assistant system 140 to interact with the user so that it can not only collect missing slot values, but also reduce the ambiguity of more complex / ambiguous utterances to complete the task. In certain embodiments, the dialog manager 216 may also support requesting missing slots in nested intents and multi-intent user inputs (e.g., "Take this photo and send it to Dad"). In certain embodiments, the dialog manager 216 may support machine learning models for a more robust dialog experience. By way of example and not limitation, the dialog state tracker 218 may use a neural network-based model (or any other suitable machine learning model) to model beliefs over task assumptions. As another example and not limitation, for the action selector 222, the highest priority policy unit may include a whitelist / blacklist override that may have to be done by design; the medium priority unit may include a machine learning model designed for action selection; and the lower priority unit may include a rule-based fallback when the machine learning model chooses not to handle a situation. In certain embodiments, a general policy unit based on a machine learning model may help the assistant system 140 reduce redundant disambiguation or confirmation steps, thereby reducing the number of turns to execute user input.

[0110] In certain embodiments, the action determined by the action selector 222 may be sent to the delivery system 230. The delivery system 230 may include a CU editor 370, a response generation component 380, a dialog state writing component 382, ​​and a text-to-speech (TTS) component 390. Specifically, the output of the action selector 222 may be received at the CU editor 370. In certain embodiments, the output from the action selector 222 may be represented as<k,c,u,d> tuple, where k indicates the knowledge source, c indicates the communication target, u indicates the user model, and d indicates the discourse model.

[0111] In a particular embodiment, the CU editor 370 may use a natural language generation (NLG) component 372 to generate communication content for a user. In a particular embodiment, the NLG component 372 may use different language models and / or language templates to generate natural language output. The generation of natural language output may be application-specific. The generation of natural language output may also be personalized for each user. In a particular embodiment, the NLG component 372 may include a content determination component, a sentence planner, and a surface implementation component. The content determination component may determine the communication content based on knowledge sources, communication goals, and user expectations. As an example and not a limitation, the determination may be based on description logic. Description logic may include, for example, three basic concepts (motion), which are individuals (representing objects in a domain), concepts (describing a collection of individuals), and roles (representing binary relationships between individuals or concepts). Description logic may be characterized by a set of constructors that allow a natural language generator to build complex concepts / roles from atomic concepts / roles. In a particular embodiment, the content determination component may perform the following tasks to determine the communication content. The first task may include a conversion task, in which the input of the NLG component 372 may be converted into a concept. The second task may include a selection task in which relevant concepts may be selected from the concepts generated from the conversion task based on the user model. The third task may include a verification task in which the coherence of the selected concepts may be verified. The fourth task may include an instantiation task in which the verified concepts may be instantiated into executable files that can be processed by the NLG component 372. The sentence planner may determine the organization of the communication content so that it can be understood by humans. The surface implementation component may determine the specific words to be used, the order of sentences, and the style of the communication content.

[0112] In a particular embodiment, the CU editor 370 may also use a UI payload generator 374 to determine the modality of the generated communication content. Since the generated communication content can be considered as a response to user input, the CU editor 370 may also use a response sorter 376 to sort the generated communication content. As an example and not a limitation, the sorting may indicate the priority of the response. In a particular embodiment, the CU editor 370 may include a natural language synthesis (natural-language synthesis, NLS) component, which may be separated from the NLG component 372. The NLS component may specify the properties (including, gender, volume, rhythm, style or range) of the synthesized speech generated by the CU editor 370 to customize the response for a specific user, task or agent. The NLS component may adjust the language synthesis without participating in the implementation of the associated task. In a particular embodiment, the CU editor 370 may check the privacy constraints associated with the user to ensure that the generation of the communication content follows the privacy policy. More information about customized natural language generation (NLG) can be found in the following documents: U.S. Patent Application No. 15 / 967,279 filed on April 30, 2018, and U.S. Patent Application No. 15 / 966,455 filed on April 30, 2018, which are incorporated by reference.

[0113] In certain embodiments, the delivery system 230 may perform different tasks based on the output of the CU editor 370. These tasks may include writing (i.e., storing / updating) the dialog state into the data repository 330 using the dialog state writing component 382, ​​and generating a response using the response generating component 380. In certain embodiments, if the modality of the determined communication content is audio, the output of the CU editor 370 may also be sent to the TTS component 390. In certain embodiments, the output from the delivery system 230 may then be sent back to the dialog manager 216, and the output from the delivery system 230 may include one or more of a response generated by the TTS component 390, the communication content, or the speech.

[0114] In certain embodiments, the coordinator 206 may determine whether to process the user input on the client system 130, on the server, or in a third mode of operation (i.e., a hybrid mode) based on the output of the entity resolution module 212. In addition to determining how to process the user input, the coordinator 206 may also receive results from the agent 228 and / or from the delivery system 230 provided by the dialog manager 216. The coordinator 206 may then forward these results to the arbitrator 226. The arbitrator 226 may aggregate these results, analyze them, select the best results, and provide the selected results to the rendering output module 232. In certain embodiments, the arbitrator 226 may query the dialog strategy 360 for guidance in analyzing these results. In certain embodiments, the rendering output module 232 may generate a response suitable for the client system 130.

[0115] Figure 4An example task-centric flowchart 400 for processing user input is shown. In certain embodiments, the assistant system 140 can help users not only with voice-initiated experiences, but also with more proactive multimodal experiences initiated when understanding the user context. In certain embodiments, the assistant system 140 can rely on assistant tasks for such purposes. Assistant tasks can be such central concepts that are shared throughout the assistant stack to understand user intent, interact with the user and the world, and complete the correct task for the user. In certain embodiments, assistant tasks can be the original unit of assistant functions. Assistant functions can include data extraction, updating a certain state, executing a certain command, or complex tasks composed of a set of smaller tasks. Correctly and successfully completing tasks to pass values ​​to users may be the goal of optimizing the assistant system 140. In certain embodiments, assistant tasks can be defined as functions or features. If multiple product surfaces have exactly the same requirements, assistant tasks can be shared across multiple product surfaces, so the assistant tasks can be easily tracked. Assistant tasks can also be passed from one device to another and are easily obtained by another device midway through the task because the original units are consistent. In addition, the consistent format of assistant tasks can allow developers working in different modules in the assistant stack to design around it more easily. In addition, it can also allow task sharing. As an example and not limitation, if a user is listening to music on smart glasses, the user can say "play this music on my phone." In the case where the phone has not yet been awakened or there is a task to perform, the smart glasses can formulate a task that is provided to the phone, which is then executed by the phone to start playing music. In a specific embodiment, if each surface has different expected behaviors, each surface can retain assistant tasks separately. In a specific embodiment, the assistant system 140 can identify the correct task based on user input or other signals of different modalities, conduct a dialogue to collect all necessary information, and use an action selector 222 implemented inside or outside a server or local product surface to complete the task. In a specific embodiment, the assistant stack may include a set of processing components that are used to wake up, recognize user input, understand user intent, reason about tasks, and complete tasks to generate natural language responses using voice.

[0116] In certain embodiments, the user input may include a voice input. The voice input may be received at the ASR module 208 to extract a text transcription from the voice input. The ASR module 208 may use a statistical model to determine the most likely sequence of words corresponding to a given portion of speech received as an audio input by the assistant system 140. The model may include one or more of a hidden Markov model, a neural network, a deep learning model, or any combination thereof. The received audio input may be encoded as digital data at a specific sampling rate (e.g., 16 kilohertz (kHz), 44.1kHz, or 96kHz) and using a specific number of bits representing each sample (e.g., 8 bits, 16 bits out of 24 bits).

[0117] In certain embodiments, the ASR module 208 may include one or more of the following: a grapheme-to-phoneme (G2P) model, a pronunciation learning model, a personalized acoustic model, a personalized language model (PLM), or an end-pointing model. In certain embodiments, the grapheme-to-phoneme (G2P) model may be used to determine a user's grapheme-to-phoneme style (i.e., what a particular word might sound like when a particular user says it). In certain embodiments, a personalized acoustic model may be a model of the relationship between an audio signal and the sound of a speech unit in a language. Thus, such a personalized acoustic model may recognize how a user's voice sounds. A personalized acoustic model may be generated using training data (e.g., a training speech received as an audio input and a corresponding speech unit corresponding to the speech). A personalized acoustic model may be trained or refined using the voice of a particular user to recognize the user's voice. In certain embodiments, the personalized language model may then determine the most likely phrase corresponding to the speech unit recognized for a particular audio input. A personalized language model may be a model of the probability that various word sequences may appear in the language. A personalized language model can be used to match the sound of a speech unit in an audio input with a sequence of words, and a greater weight can be assigned to a sequence of words that is more likely to be a phrase in the language. Then, the sequence of words with the highest weight can be selected as the text corresponding to the audio input. In a specific embodiment, the personalized language model can also be used to predict what words a user is most likely to say given a context. In a specific embodiment, an end point model can detect when the end of an utterance is reached. In a specific embodiment, during the client-side process, the assistant system 140 can optimize the personalized language model at runtime based at least in part on the limited computing power of the client system 130. As an example and not a limitation, the assistant system 140 can pre-calculate multiple personalized language models for multiple possible topics that a user may talk about. When user input is associated with an assistant request, the assistant system 140 can quickly switch between pre-calculated language models and optimize these pre-calculated language models locally based on user activity at runtime. Therefore, the assistant system 140 can retain computing resources while efficiently identifying topics associated with user input. In a specific embodiment, the assistant system 140 can also dynamically re-learn user pronunciation at runtime.

[0118] In certain embodiments, user input may include non-speech input. Non-speech input may be received at context engine 220 to determine events and contexts based on the non-speech input. Context engine 220 may determine multimodal events, including speech / text intent, location updates, visual events, touch, gaze, gesture, action, device / application events, and / or any other suitable type of event. Speech / text intent may depend on ASR module 208 and NLU module 210. Location updates may be consumed by dialog manager 216 to support various active / passive scenarios. Visual events may be based on people or objects that appear in the user's field of view. These events may be consumed by dialog manager 216 and recorded in temporary user states to support visual co-reference (e.g., parsing out "that" in "how much is that shirt?" and parsing out "him" in "send him my contact information"). Gaze, gesture, and activity may cause a flag to be set in a temporary user state (e.g., the user is running), which may adjust action selector 222. For device / application events, if the application updates the device state, it can be published to the assistant system 140 so that the dialogue manager 216 can use the context (the context currently displayed to the user) to handle passive and active scenarios. As an example and not limitation, the context engine 220 can cause a push notification message to be displayed on the display screen of the user's client system 130. The user can interact with the push notification message, which can initiate a multimodal event (e.g., an event workflow for replying to a message received from another user). Other example multimodal events can include seeing a friend, seeing a landmark, being at home, running, starting a call with touch, taking a photo with touch, opening an application, etc. In a particular embodiment, the context engine 220 can also determine world / social events based on world / social updates (e.g., weather changes, friends online). Social updates can include events that the user subscribes to (e.g., friends' birthdays, posts, comments, other notifications). The dialogue manager 216 can consume these updates to trigger context-based active actions (e.g., suggesting that the user call a friend on his birthday, but only when the user is not paying attention to other things). By way of example and not limitation, receiving a message may be a social event that may trigger a task to read the message to the user.

[0119] In certain embodiments, the text transcription from the ASR module 208 may be sent to the NLU module 210. The NLU module 210 may process the text transcription and extract the user intent (i.e., multiple intents), and resolve the slots or resolve the results based on the language ontology. In certain embodiments, the intents and slots from the NLU module 210, and / or the events and contexts from the context engine 220 may be sent to the entity resolution module 212. In certain embodiments, the entity resolution module 212 may resolve entities associated with the user input based on the output from the NLU module 210 and / or the context engine 220. The entity resolution module 212 may resolve entities using different techniques, including accessing user memories from the assistant user memory (AUM) 354. In certain embodiments, the AUM 354 may include user context memories that facilitate the entity resolution module 212 in resolving entities. The AUM 354 may be a central location for storing, retrieving, indexing, and searching user data.

[0120] In certain embodiments, the entity resolution module 212 may provide one or more of an intent, a slot, an entity, an event, a context, or a user memory to the dialog state tracker 218. The dialog state tracker 218 may accordingly identify a set of state candidates for the task, interact with the user to collect the information required to fill the state, and call the action selector 222 to complete the task. In certain embodiments, the dialog state tracker 218 may include a task tracker 410. The task tracker 410 may track the task state associated with the assistant task. In certain embodiments, the task state may be a data structure that persists across multiple interaction rounds and is updated in real time to collect the state of the task during the entire interaction. The task state may include all current information related to the state of task execution, such as parameters, confirmation status, confidence scores, etc. Any errors or outdated information in the task state may result in a failure or error in the task execution. The task state may also be used as a set of context information for many other components (e.g., ASR module 208, NLU module 210, etc.).

[0121] In a particular embodiment, the task tracker 410 may include an intent processor 411, a task candidate ranking module 414, a task candidate generation module 416, and a merging layer 419. In a particular embodiment, a task may be identified by its ID name. If the task ID is not explicitly set in the task specification (e.g., dialogue strategy 360, agent execution, NLG dialogue action, etc.), the task ID may be used to associate the corresponding component asset. Therefore, the output from the entity resolution module 212 may be received by the task ID resolution component 417 of the task candidate generation module 416 to resolve the task ID of the corresponding task. In a particular embodiment, the task ID resolution component 417 may call the task specification manager API 430 to access the trigger specification and deployment specification for resolving the task ID. Given these specifications, the task ID resolution component 417 may resolve the task ID using intent, slot, dialogue state, context, and user memory.

[0122] In a particular embodiment, the technical specification of a task can be defined by a task specification. The assistant system 140 can use the task specification to trigger the task, conduct a dialog dialogue, and find the correct execution module (e.g., agent 228) to perform the task. The task specification can be the implementation of the product requirement document. The task specification can be used as a requirement for a general contract and all component agreements. The task specification can be considered as the assembly specification of the product, and all development partners deliver these modules based on the specification. In a particular embodiment, the assistant task can be defined in the implementation by the specification. As an example and not a limitation, the task rule can be defined as the following categories. One category can be a basic task scheme, which includes basic identification information, such as a scheme of ID, name, and input parameters. Another category can be a trigger specification on how to trigger a task, such as an intention, an event message ID, etc. Another category can be a dialogue specification for the dialogue manager 216 to dialogue with the user and the system. Another category can be an execution specification on how to execute and complete a task. Another category can be a deployment specification on how to deploy functions to certain surfaces, locals, and user groups.

[0123] In a particular embodiment, the task specification manager API 430 can be an API for accessing the task specification manager. The task specification manager can be a module in the runtime stack that is used to load specifications from all tasks and provide an interface to access all task specifications to obtain detailed information or generate task candidates. In a particular embodiment, the task specification manager can access all components in the runtime stack through the task specification manager API 430. The task specification manager can include a set of static utility functions to utilize the task specification manager to manage tasks, such as filtering task candidates through the platform. Before the task specification is landed, the assistant system 140 can also dynamically load the task specification to support end-to-end development during the development phase.

[0124] In certain embodiments, task specifications may be grouped by domain and stored in runtime configuration 435. The runtime stack may load all task specifications from runtime configuration 435 during build time. In certain embodiments, in runtime configuration 435, for a domain, there may be a cconf file and a cinc file (e.g., sidechef_task.cconf and sidechef_task.inc). By way of example and not limitation, <domian>_tasks.cconf can include all the details of the task specification. As another example and not limitation, if this feature is not yet supported, <domain>_tasks.cinc can provide a way to override the generated specification.

[0125] In a particular embodiment, task execution may require a set of parameters to execute. Therefore, parameter parsing component 418 can use the parameter specification of the parsed task ID to parse the parameter name. These parameters can be parsed based on NLU output (e.g., slot [SL: contact] ([SL:contact])), dialogue state (e.g., short-term call history), user memory (e.g., user preferences, location, long-term call history, etc.) or device context (e.g., timer state, screen content, etc.). In a particular embodiment, the parameter modality can be text, audio, image or other structured data. The mapping of slots to parameters can be defined by filling strategies and / or language ontology. In a particular embodiment, given a task trigger specification, task candidate generation module 416 can find a list of tasks to be triggered as task candidates based on the parsed task ID and parameters.

[0126] In a particular embodiment, the generated task candidates may be sent to the task candidate sorting module 414 for further sorting. The task candidate sorting module 414 may sort them using a rule-based sorter 415. In a particular embodiment, the rule-based sorter 415 may include a set of heuristics to favor certain domain tasks. The sorting logic may be described below using the principle of contextual priority. In a particular embodiment, the priority of a user-specified task may be higher than that of a foreground task. When the intent is a meta-intent, the priority of a foreground task may be higher than that of a device domain task. The priority of a device domain task may be higher than that of a task that triggers the intent domain. As an example and not limitation, if a task domain is mentioned or specified in a discourse (e.g., "create a timer in a timer application"), the sorting may pick out the task. As another example and not limitation, when the timer application is in the foreground and there is an active timer, if the task domain is in the foreground or active state, the sorting may pick out a task (e.g., "stop timer") to stop the timer. As another example and not limitation, if the intent is a general meta-intent, and when there are no other active applications or active states, and the task is device control, the sorting may pick out the task. As another example and not limitation, if the task is the same as the intent domain, the sorting can pick out the task. In a particular embodiment, the task candidate sorting module 414 can customize some more logic to check for matches of intent / slot / entity types. The sorted task candidates can be sent to the merging layer 419.

[0127] In a particular embodiment, the output from the entity resolution module 212 may be sent to the task ID resolution component 412 of the intent processor 411. Similar to the task ID resolution component 417, the task ID resolution component 412 may resolve the task ID of the corresponding task. In a particular embodiment, the intent processor 411 may also include a parameter resolution component 413. Similar to the parameter resolution component 418, the parameter resolution component 413 may resolve the parameter name using the parameter specification of the resolved task ID. In a particular embodiment, the intent processor 411 may process features that are not related to the task and may not be expressed in a task-specific task specification. The intent processor 411 may output state candidates other than task candidates, such as parameter updates, confirmation updates, disambiguation updates, etc. In a particular embodiment, some tasks may require very complex trigger conditions or very complex parameter filling logic, which may not be reused by other tasks (e.g., voice commands in calls, media tasks via [IN: PLAY_MEDIA], etc.), even if they are supported in the task specification. The intent processor 411 may also be applicable to this type of task. In certain embodiments, the results from the intent processor 411 may take precedence over the results from the task candidate ranking module 414. The results from the intent processor 411 may also be sent to the merging layer 419.

[0128] In certain embodiments, the merging layer 419 may combine the results from the intent processor 411 and the results from the task candidate ranking module 414. The dialog state tracker 218 may suggest each task as a new state for the dialog policy 360 to select from, thereby generating a list of state candidates. The merged results may be further sent to a conversational understanding reinforcement engine (CURE) tracker 420. In certain embodiments, the CURE tracker 420 may be a personalized learning process to use real-time user feedback to improve the dialog state tracker 218's determination of state candidates in different contexts. More information about the dialog understanding reinforcement engine may be found in the following documents: U.S. Patent Application No. 17 / 186,459, filed on February 26, 2021, which is incorporated by reference.

[0129] In certain embodiments, the candidate states generated by CURE tracker 420 may be sent to action selector 222. Action selector 222 may query task policy 364, which may be generated from an execution specification accessed through task specification manager API 430. In certain embodiments, the execution specification may describe how the task should be performed and what actions action selector 222 may need to take to complete the task.

[0130] In certain embodiments, the action selector 222 may determine an action associated with the system. Such an action may require the agent 228 to perform. Therefore, the action selector 222 may send the system action to the agent 228, and the agent 228 may return the results of the execution of these actions. In certain embodiments, the action selector may determine an action associated with a user or device. Such an action may require the delivery system 230 to perform. Therefore, the action selector 222 may send the user / device action to the delivery system 230, and the delivery system 230 may return the results of the execution of these actions.

[0131] The embodiments disclosed herein may include an artificial reality system, or be implemented in combination with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to the user, and the artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with collected content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of the above may be presented in a single channel or multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may also be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality, and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0132] Context-driven conversations in noisy environments

[0133] In a particular embodiment, when a user uses an assistant-enabled head-mounted device such as smart glasses, the assistant system 140 can improve the response to the user's voice commands in a noisy environment. The assistant system 140 can access the audio signal collected by the microphone of the head-mounted device, measure the background noise using different frame-based environmental metrics, and use the measurement results to drive a meaningful response to the user (i.e., a contextual response based on the noisy environment). The assistant system 140 can use noise suppression to improve automatic speech recognition (ASR) in a noisy environment, automatically increase the volume of the assistant's response when the user is in a noisy environment, and successfully resume the task instead of simply failing the task. As an example and not limitation, a user in a noisy environment speaks a voice command such as "Hey Assistant, take a picture." The assistant system 140 can respond with "It's too noisy. I can't understand you" (i.e., an out-of-context response). Similarly, as the task resumes, the assistant system 140 may say "It's noisy, but do you want to take a picture?" (i.e., a low-confidence successful contextual response with explicit confirmation). Although this disclosure describes a particular response in a particular context being generated by a particular system in a particular manner, this disclosure contemplates any suitable response in any suitable context being generated by any suitable system in any suitable manner.

[0134] In certain embodiments, assistant system 140 may access one or more audio signals corresponding to a voice input from a user from a head mounted device associated with the user. Assistant system 140 may then determine one or more audio environment metrics corresponding to the voice input based on the one or more audio signals. Assistant system 140 may then calculate a confidence score indicating the likelihood that the assistant system 140 associated with the head mounted device understands the voice input. In certain embodiments, assistant system 140 may generate a response based on the confidence score and the one or more audio environment metrics. Assistant system 140 may also provide instructions to the head mounted device to present the response.

[0135] For users of head-mounted devices such as smart glasses, one problem may be that environmental noise (especially wind noise) may distort the user's voice input and make it difficult for the wake-up word model and the ASR model to correctly recognize the user's voice. The results may include that the user's speech may be incorrectly classified as silence (SILENCE) or out of domain (OUT_OF_DOMAIN). When the speech is classified as silence, the assistant system 140 may not respond at all. When the speech is classified as out of domain, the assistant system 140 may respond with a generic out-of-domain (OOD) response (for example, "I can't do this on your glasses"), which may not explain why the in-domain command failed. Both scenarios may result in a poor user experience.

[0136] In certain embodiments, one or more audio signals may be collected by one or more microphones of the head-mounted device. To improve responses from assistant system 140 in noisy environments, the head-mounted device may use a new microphone array that enables better noise suppression (e.g., five microphones instead of three microphones). However, noise may still be a problem for assistant system 140, according to one study, where estimates of speech signal-noise-ratio (SNR) and ASR word-error-rate (WER) with a six-microphone array indicated that ASR accuracy may only be "somewhat usable" (7% to 8% WER) in cars, busy streets, restaurants, and bars.

[0137] Therefore, the embodiments disclosed herein utilize different solutions to solve this problem by inferring when the assistant system 140 cannot understand the user's voice command due to high background noise, and providing a text-to-speech (TTS) response accordingly. In a specific embodiment, the assistant system 140 can use context-driven dialogue to provide the user with feedback on the assistant interaction in a noisy environment. In a specific embodiment, the response may include one or more of the following: an indication that the assistant system 140 does not understand the user; an indication that the user's environment is too noisy; or an implicit confirmation of the command associated with the voice input. When the ambient noise is strong and the speech recognition is not reliable (i.e., the confidence score is less than the threshold score), the assistant system 140 may respond: "I can't understand, it's too noisy". The assistant system 140 can also improve the understanding of user speech in a noisy environment, including using noise suppression to improve the ASR model in a noisy environment, automatically increasing the volume of the text-to-speech (TTS) response from the assistant system 140 when the user is in a noisy environment, and successfully resuming the task instead of simply failing the task gracefully. Thus, assistant system 140 can have a technical advantage of providing better assistant feedback to the user, because assistant system 140 can make the user understand when assistant system 140 is listening to the user but cannot understand the command due to background noise, and also automatically increase the volume of the assistant text-to-speech (TTS) response when the user is in a noisy environment.

[0138] Figure 5 An example flowchart 500 for a server-side context-driven conversation is shown. At step 510, ambient noise and / or wind noise can be measured on a digital-signal-processing (DSP) unit on the head-mounted device. At step 520, the audio and metadata can be forwarded to the server via the companion device 138. At step 530, the server of the assistant system 140 can transcribe, evaluate the audio, and take action. At step 540, if the request is unclear and the environment is noisy, the assistant system 140 can respond: "I can't understand, it's too noisy." At step 550, the head-mounted device can notify the user using text to speech (TTS).

[0139] In a particular embodiment, the assistant system 140 can utilize the environmental context through the audio environment metrics on the frame, and pass them to the assistant server together with the audio stream to handle the server-side interaction. As an example and not limitation, one or more audio environment metrics may include one or more of the following: ambient loudness, wind energy, or voice activity. In a particular embodiment, the assistant system 140 can also determine the noise level associated with one or more audio signals based on the one or more audio environment metrics. The noise level may be higher than a threshold level. Using audio environment metrics to determine the noise level associated with the received audio signal can be an effective solution for solving the technical challenge of determining that the user is in a noisy environment, because the audio environment metrics can provide effective measurements of ambient noise loudness, ambient wind energy level, and voice activity.

[0140] In certain embodiments, assistant system 140 may collect average values ​​of ambient noise loudness and ambient wind energy level during each ASR session (i.e., each utterance). In certain embodiments, the average value of ambient noise loudness may be measured as decibels relative to full-intensity audio. A reading of 0 may indicate that the audio signal is as loud as the audio system can handle. A reading of -45 may indicate that the amplitude of the audio is 45 decibels (dB) lower than the full intensity. In certain embodiments, these numbers may be converted to positive values. For example, a reading of 18 means -18 dB, and a reading of 45 means -45 dB. In certain embodiments, the average value of ambient noise loudness may be measured as a decibel (dB) sound pressure level (SPL) (dBSPL), an absolute measure of sound pressure, which may always be positive and therefore directly related to the noise level.

[0141] Assistant system 140 may also measure an average of wind activity level, which is a measure between [0, 1] that predicts the probability of wind being present. In a particular embodiment, the average of wind activity level may be scaled by a factor of 1000 to accommodate the conversion from float32 to integer8. Assistant system 140 may also measure an average of voice activity level, which is an average measure of the probability [0, 1] that the user is speaking during the ASR session. The average of voice activity level may be scaled by a factor of 1000 to accommodate the conversion from float32 to integer8. Assistant system 140 may also measure an average of the device speaker volume setting (0 to 100) during each ASR session.

[0142] The assistant system 140 may also collect the percentage of time that the user may be speaking during each ASR session. The assistant system 140 may also collect the average of the device speaker volume setting (0 to 100) during each ASR session. In a particular embodiment, if the wind energy is greater than 11 decibels (dB) sound pressure level (SPL) per minute and the average speech level is less than 60 decibels (dB) sound pressure level (SPL), then the environment can be considered windy. In a particular embodiment, if the ambient loudness is not less than 54 decibels (dB) sound pressure level (SPL) per minute and the wind energy is less than 11 decibels (dB) sound pressure level (SPL) per minute, then the environment can be considered noisy but not due to wind.

[0143] Figure 6 An example pipeline 600 for utilizing an audio environment metric on a frame is shown. In a particular embodiment, audio signals collected by a microphone 602 may be provided to a DSP unit 604. The DSP unit 604 may first perform filter bank processing and echo cancellation (EC) 606 on the audio signals. In a particular embodiment, suppressing one or more noise signals from one or more audio signals may also be based on one or both of filter bank processing or echo cancellation. Then, the DSP unit 604 may perform wind removal 608 on the processed signals. After wind removal 608, the DSP unit 604 may perform beamforming 610. In a particular embodiment, suppressing one or more noise signals from one or more audio signals may include applying a beamforming algorithm to the one or more audio signals. After beamforming 610, the audio stream may directly enter stereo acquisition 612 in a microcontroller unit (MCU) 614, or be processed by noise removal 616.

[0144] In a particular embodiment, noise removal 616 can be based on a noise suppression model that determines whether there is a large amount of background noise. The noise suppression model can also filter the audio data before it enters the ASR module 208. In a particular embodiment, the assistant system 140 can determine one or more noise signals associated with one or more audio signals. The assistant system 140 can suppress the one or more noise signals from the one or more audio signals based on the noise suppression model. Therefore, the assistant system 140 can have a technical advantage of improving the ASR model to make it robust to background noise, because the assistant system 140 can use noise suppression to improve the ASR in a noisy environment, so that the input of the assistant system 140 (i.e., the ASR token) has a higher quality.

[0145] After noise removal 616a, the audio stream may be sent to a voice over internet protocol (VOIP) 618 of the MCU 614. After noise removal 616b, the audio stream may be sent to the arbitrator 206, a voice activity detector (VAD) 620, and a wake-word (WW) detector 622, respectively. The audio stream after noise removal 616b and VAD 620 may be provided to the arbitrator 206. The audio stream after the wake-word detector 622 may be sent to the assistant wake-up module 624 of the system on a chip (SoC). In a particular embodiment, the assistant system 140 may generate a transcription of the speech input based on the noise suppressed audio signal through an automatic speech recognition (ASR) model.

[0146] In a particular embodiment, the DSP unit 604 may estimate wind, noise, and activity early in the pipeline 600. After wind removal 608, the DSP unit 604 may add the estimated environmental metrics to the audio stream, which may be further provided to the arbitrator 206. In a particular embodiment, the assistant system 140 may generate one or more updated audio signals by combining one or more audio environmental metrics with one or more audio signals. The assistant system 140 may then provide the one or more updated audio signals to one or more modules associated with the assistant system 140. Thus, generating a response may also be based on the processing results of one or more of the one or more modules associated with the assistant system 140 on the one or more updated audio signals. In a particular embodiment, the voice input may be associated with a first conversation session between the user and the assistant system 140. Generating a response may also be based on one or both of the following: a previous conversation state associated with the first conversation session or a device context associated with the head mounted device.

[0147] Figure 7 An example metadata encoded audio stream is shown. In a particular embodiment, metadata may be added to include audio environment metrics in a key-value structure, including ambient noise levels in decibel (dB) sound pressure levels (SPL), wind levels in dBSPL, and voice activity from 0 to 1. The audio stream may include multiple data packets 710, 720, 730, and 740. Each data packet may be approximately 4 milliseconds of audio data. In a particular embodiment, each data packet 710, 720, 730, and 740 may include headers (8 bytes) 712, 722, 732, and 742, respectively. Each data packet 710, 720, 730, and 740 may also include metadata (64 bytes) 714, 724, 734, and 744, respectively. Each data packet 710, 720, 730, and 740 may also include coded audio (128 bytes) 716, 726, 736, and 746, respectively.

[0148] Based on the audio streams from noise removal 616b, VAD 620, and the audio stream with the audio environment metric, arbitrator 206 may forward audio stream 626 to assistant system 140. As can be seen, the audio sent to assistant system 140 may have been adjusted to reduce noise and suppress wind.

[0149] In certain embodiments, the arbitrator 206 may run in the MCU 614 to quickly process the incoming audio stream. In addition to out-of-domain requests, the arbitration logic may also include "first response wins". If the companion device 138 or the headset responds with out-of-domain while the other is processing, the arbitrator 206 may wait. In certain embodiments, the companion device 138 or the assistant system 140 may respond quickly for silent interactions. In certain embodiments, a "capture" request may be classified as out-of-domain.

[0150] After assistant activation (via wake word detection or force touch on the head mounted device), the audio stream can be processed by ASR module 208. In certain embodiments, assistant system 140 can also align the audio environment metrics with the confidence level of ASR module 208. The ASR output can then be passed to NLU module 210, which can classify the input audio stream as out-of-domain (OOD). The out-of-domain (OOD) classification and the determination of whether there is a lot of background noise can then be passed to dialog manager 216, which can determine that it is too noisy and generate a response accordingly.

[0151] In certain embodiments, when the ASR module 208 outputs a low confidence level for the text transcription of the audio input and there is a lot of background noise, the assistant system 140 can provide a fault. When the ambient noise is high (as measured by the audio environment metric) and the ASR module 208 is not trustworthy, the assistant system 140 can provide a text-to-speech (TTS) response that takes into account the background noise.

[0152] In certain embodiments, the assistant system 140 may provide different responses in different scenarios. When the ASR 208 or NLU 210 cannot understand the user's first utterance with sufficient confidence, and the average ambient loudness or average wind energy level exceeds a predetermined threshold, the assistant system 140 may provide a decent out-of-domain (OOD) response. In this scenario, the ASR module 208 may not have confidence (i.e., the assistant system 140 did not guess what the user wanted), which means that it is clearly wrong due to noise. The out-of-domain (OOD) response can be a contextual out-of-domain (OOD) response type that lets the user know that the assistant system 140 cannot respond (i.e., the assistant system does not understand the user or cannot determine which domain to call), and why the assistant system 140 cannot respond (i.e., it is too noisy to clearly hear the user's voice). As an example and not a limitation, the response can be "Sorry, I can't understand you. It's too noisy" or "Sorry, there is very strong background noise, please try your command again later." In certain embodiments, assistant system 140 may rely on certain logic with a few (eg, 1 to 3) attempts with a tapered response.

[0153] When the assistant system 140 has a certain level (albeit low) of confidence in the domain to be selected, the assistant system 140 can provide a recovered out-of-domain response. As an example and not limitation, the assistant system 140 can respond with "It's a bit noisy, but here's your answer: [normal conversation]" for confirmation. For this scenario, if the ASR module 208 has low confidence, the assistant system 140 can have a confirmation step due to the noise, such as a domain-specific confirmation step. If the ASR module 208 has medium confidence, the assistant system 140 can implicitly confirm the noise.

[0154] For a multi-turn experience, if assistant system 140 is unable to recognize an utterance during a conversation, assistant system 140 may respond based on contextual information in the particular conversation and then determine whether to gracefully terminate the conversation or resume the conversation. In certain embodiments, gracefully terminating the conversation may be similar to generating a graceful out-of-domain (OOD) response as described above. For a resumed conversation, assistant system 140 may utilize a domain-specific scripted response.

[0155] Figure 8 An example user flow with logic 800 for a graceful out-of-domain response is shown. At step 802, a wake word (e.g., "Hey Assistant") can start the user flow 800. The assistant system 140 can then determine whether a user request is recognized at step 804. If the request is recognized, the assistant system 140 can determine whether the request is out-of-domain at step 806. If the request is out-of-domain, the assistant system 140 can generate an out-of-domain (OOD) response at step 808, such as, "I can't do that on your glasses." If the request is not out-of-domain, at step 810, the user flow 800 can enter the domain. By way of example and not limitation, the domain can be acquisition (photos, videos), calling, messaging, sharing, or utility.

[0156] If the user request is not recognized at step 804, the assistant system 140 may determine the ASR confidence level at step 812. If the ASR confidence level is high, the assistant system 140 may generate a recovered out-of-domain response at step 814. If the ASR confidence level is low, the assistant system 140 may determine whether it is the first attempt at step 816. If it is the first attempt, the assistant system 140 may generate a first appropriate response at step 818, for example, "I can't hear you, it's too noisy." If it is not the first attempt, the assistant system 140 may determine whether it is the second attempt at step 820. If it is the second attempt, the assistant system 140 may generate a second appropriate response at step 822, for example, "I still can't hear you. There's too much noise." If it is not the second attempt, the assistant system 140 may determine whether it is the third attempt at step 824. If it is the third attempt, the assistant system 140 may generate a third appropriate response at step 826, for example, "It's too noisy." In certain embodiments, from step 812 to step 818, to step 822, or to step 826, assistant system 140 may explicitly have audio environment metrics so that assistant system 140 may notify the user that it is "too loud" or there is "too much noise."

[0157] Fig. 9 An example user flow with logic 900 for a recovered out-of-domain response with low confidence is shown. At step 902, a wake word (e.g., "Hey Assistant") can begin the user flow 900. The assistant system 140 can then determine whether a user request is recognized at step 904. If the request is recognized, the assistant system 140 can determine whether the request is out-of-domain at step 906. If the request is out-of-domain, the assistant system 140 can generate an out-of-domain (OOD) response at step 908, such as, "I can't do that on your glasses." If the request is not out-of-domain, then at step 910, the user flow 900 can enter the domain.

[0158] If the user request is not recognized at step 904, the assistant system 140 can determine the ASR confidence level at step 912. If there is no confidence, the assistant system 140 can generate a decent out-of-domain (OOD) response at step 914. Between steps 912 and 914, the assistant system 140 can check the audio environment metrics. If there is confidence, the assistant system 140 can determine whether the ASR has a medium confidence level at step 916. If there is a medium confidence level, the assistant system 140 can generate an out-of-domain (OOD) response with medium confidence at step 918.

[0159] If the ASR has a low confidence level, assistant system 140 can explicitly confirm the domain with the user, e.g., via a yes / no prompt. For example, at step 920, assistant system 140 can determine whether it is a capture domain. If it is a capture domain, then at step 922, assistant system 140 can generate a response to explicitly confirm the capture, e.g., "I really can't hear you. Take a picture?" or "I really can't hear you. Record a video?" User flow 900 can then proceed to step 910.

[0160] If it is not the collection domain, the assistant system 140 can determine whether it is the call domain at step 924. If it is the call domain, the assistant system 140 can generate a response at step 926 to explicitly confirm the call. Then, the user process 900 can proceed to step 910. If it is not the call domain, the assistant system 140 can determine whether it is the messaging domain at step 928. If it is the messaging domain, the assistant system 140 can generate a response at step 930 to explicitly confirm the messaging. Then, the user process 900 can proceed to step 910.

[0161] If it is not a messaging domain, the assistant system 140 can determine whether it is a sharing domain at step 932. If it is a sharing domain, the assistant system 140 can generate a response at step 934 to explicitly confirm the sharing. The user flow 900 can then proceed to step 910. If it is not a sharing domain, the assistant system 140 can determine whether it is a utility domain at step 936. If it is a utility domain, the assistant system 140 can generate a response at step 938 to explicitly confirm the utility. The user flow 900 can then proceed to step 910. If the user does not confirm, the user flow 900 can end at step 940.

[0162] Fig.10 An example user flow with logic 1000 for a recovered out-of-domain response with medium confidence is shown. At step 1002, a wake word (e.g., "Hey Assistant") can begin user flow 1000. Assistant system 140 can then determine whether a user request is recognized at step 1004. If the request is recognized, assistant system 140 can determine whether the request is out-of-domain at step 1006. If the request is out-of-domain, assistant system 140 can generate an out-of-domain (OOD) response at step 1008, such as, "I can't do that on your glasses." If the request is not out-of-domain, then at step 1010, user flow 1000 can enter the domain.

[0163] If the user request is not recognized at step 1004, the assistant system 140 can determine the ASR confidence level at step 1012. If there is no confidence, the assistant system 140 can generate a decent out-of-domain (OOD) response at step 1014. Between steps 1012 and 1014, the assistant system 140 can check the audio environment metrics. If there is confidence, the assistant system 140 can determine whether the ASR has a medium confidence level at step 1016. If the ASR has a low confidence level, the assistant system 140 can generate an out-of-domain (OOD) response with low confidence at step 1018.

[0164] If the ASR has a medium confidence level, assistant system 140 can determine whether it is a collection domain at step 1020. If it is a collection domain, assistant system 140 can generate a response at step 1022 to implicitly confirm the collection, such as "take a picture..." or "start a video...". Then, user flow 1000 can proceed to step 1010.

[0165] If it is not the collection domain, the assistant system 140 can determine whether it is the call domain at step 1024. If it is the call domain, the assistant system 140 can generate a response at step 1026 to implicitly confirm the call. Then, the user process 1000 can proceed to step 1010. If it is not the call domain, the assistant system 140 can determine whether it is the messaging domain at step 1028. If it is the messaging domain, the assistant system 140 can generate a response at step 1030 to implicitly confirm the messaging. Then, the user process 1000 can proceed to step 1010.

[0166] If it is not a messaging domain, the assistant system 140 may determine if it is a sharing domain at step 1032. If it is a sharing domain, the assistant system 140 may generate a response at step 1034 to implicitly confirm sharing. The user flow 1000 may then proceed to step 1010. If it is not a sharing domain, the assistant system 140 may determine if it is a utility domain at step 1036. If it is a utility domain, the assistant system 140 may generate a response at step 1038 to implicitly confirm utility. The user flow 1000 may then proceed to step 1010.

[0167] In certain embodiments, to enable interactions with appropriate out-of-domain responses, assistant system 140 may use a model-based dialogue (MBD) system. The MBD system may have ready-made processes for changing responses based on the history of the conversation. The MBD system may also provide facilities for customizing thresholds based on observed interaction behavior using trained models. In certain embodiments, assistant system 140 may implement hotfix rules in the MBD system to directly address interactions with noise above a threshold.

[0168] As can be seen, assistant system 140 can have another technical advantage in creating more successful assistant interactions because assistant system 140 can leverage environmental context when deciding how to respond to low / medium confidence utterances for both single-turn interactions and multi-turn interactions. For single-turn interactions where low latency is critical (e.g., photo acquisition), assistant system 140 can use environmental context and ASR confidence to complete the task in one turn instead of failing silently. For multi-turn interactions (e.g., calls and messaging), assistant system 140 can ask follow-up questions when a command cannot be understood due to background noise in order to successfully complete the task.

[0169] Fig.11 An example high-level flow 1100 for gracefully generating out-of-domain responses using a context-driven model-based dialogue (MBD) system is shown. At step 1102, a wake word (e.g., "Hey Assistant") can start the high-level flow 1100. The assistant system 140 can then determine whether the domain is recognized at step 1104. If the domain is recognized, the assistant system 140 can enter the domain at step 1106. If the domain is not recognized, the assistant system 140 can determine a confidence level at step 1108.

[0170] If the confidence level is low, the assistant system 140 can determine whether it is the first attempt at step 1110. If it is the first attempt, the assistant system 140 can send an error at step 1112. As an example and not limitation, the assistant system 140 can say "I can't hear you, it's too loud." If it is not the first attempt, the assistant system 140 can send an error at step 1114 and also provide alternative options. As an example and not limitation, the assistant system 140 can say "I still can't hear you. Try again when it's quieter, or use the capture button." After step 1110 or step 1112, the high-level process 1100 can end at step 1116.

[0171] If the confidence is medium, assistant system 140 can confirm the domain with the user at step 1118, for example, with a yes / no prompt. As an example and not limitation, assistant system 140 can say "I really can't hear you. Do you want to take a picture? Say yes or no." If the user confirms the domain, assistant system 140 can enter the domain at step 1120. If the user does not confirm the domain, assistant system 140 can end high-level process 1100 at step 1116.

[0172] If the confidence level is high, assistant system 140 can confirm the request at step 1122 and continue with high-level flow 1100. As an example and not limitation, assistant system 140 can say "It's too loud, but I heard you. Take a picture now." Assistant system 140 can then enter the domain at step 1120.

[0173] In certain embodiments, assistant system 140 may accumulate statistics of the noise environment and create an aggregate for each interaction. For users who have selected assistant logging, this may be recorded. By way of example and not limitation, the aggregate recorded may be:

[0174] {

[0175] 'speech_activity_level_min': 0,

[0176] "speech_activity_level_max": 1000,

[0177] 'speech_activity_level_average': 134.07486631016,

[0178] "wind_energy_level_min": 42,

[0179] "wind_power_level_max": 76,

[0180] "wind_energy_level_average": 56.802139037433,

[0181] "ambient_loudness_min": 54,

[0182] "ambient_loudness_max": 59,

[0183] "ambient_loudness_average": 56.83422459893

[0184] }

[0185] In a particular embodiment, the assistant system 140 may also use one or more noisy environment strategies to intercept multiple domains. The assistant system 140 may use these strategies to perform precondition checks on out-of-domain (OOD), silent (SILENT) domains, noisy environments, and ASR confidence (if out-of-domain). The assistant system 140 may then link to a model-based dialog to generate a dialog response. In a particular embodiment, using these strategies, the assistant system 140 may match appropriate specific conditions and, if the conditions pass, call a model-based dialog (MBD) system. The result may be the creation of a dialog state operator that adds a new task to a task queue and prioritizes it to specifically match tasks associated with an MBD system. The assistant system 140 may provide a cross-domain link to a single MBD task.

[0186] In certain embodiments, a model-based dialogue (MBD) system may train a lightweight model to select actions based on a set of declared input features. Context-driven dialogues may primarily look at noise levels and any recurrences (past dialogue states) to select dialogue responses. As an example and not limitation, the first response may be "I can't understand" and the second response may be "I still can't understand..." In certain embodiments, model-based dialogues may be based on predefined thresholds. In certain embodiments, model-based dialogues may train models on real interaction data to optimize task success in conjunction with ambient noise, wind noise, and voice activity. Using a model-based dialogue (MBD) system may be an effective solution to the technical challenge of determining what type of response to provide to a user in a noisy environment, because a model-based dialogue (MBD) system may train a lightweight model to select actions based on a set of declared input features, wherein the training is based on real interaction data to optimize task success in conjunction with ambient noise, wind noise, and voice activity.

[0187] In certain embodiments, the assistant system 140 can provide disambiguation for uncertain events with noise-robust responses. As an example and not a limitation, instead of "I don't understand", the assistant system 140 can ask for clarification, for example, "It's noisy, but do you want to take a photo?". The assistant system 140 can disambiguate based on previous actions. In certain embodiments, the response can include an implicit confirmation of the command associated with the voice input. The voice input can be associated with a first dialogue session between the user and the assistant system 140. Therefore, the implicit confirmation can be generated based on the previous action taken by the assistant system 140 during the first dialogue session. As an example and not a limitation, if the previous action is "take a photo", the assistant system 140 can also offer to take a photo again even if the ASR confidence is low and the noise is high. In certain embodiments, the assistant system 140 can use model-based dialogue to change the disambiguation question. As an example and not a limitation, instead of asking "John Smith" or "John Smythe", the assistant system 140 can ask, "Do you want to send a message to John Smith?", which can get a more clear answer in a noisy environment.

[0188] In an alternative embodiment, assistant system 140 can pass the output from ASR module 208 and the determination of whether there is a lot of background noise to NLU module 210. NLU module 210 can classify them as out-of-domain (OOD) or too noisy. In this scenario, NLU module 210 can be trained to classify various types of out-of-domain (OOD) inputs (e.g., too noisy, incapable, etc.).

[0189] In certain embodiments, assistant system 140 may determine one or more modalities for presenting a response. By way of example and not limitation, the one or more modalities may include one or more of a text modality, an image modality, a video modality, an extended reality (XR) modality, a sound modality, a voice modality, or a light modality. When the one or more modalities include one or more of a video modality, a sound modality, or a voice modality, assistant system 140 may determine an audio signal enhancement threshold amount associated with the presented response. Thus, the instructions may include the determined enhanced instructions.

[0190] FIG. 12A to FIG. 12C An example scenario of contextual response in a noisy environment is shown. Fig. 12A As shown, user 1210 may be hiking. User 1210 may be wearing smart glasses 130. Then, user 1210 may speak to assistant system 140 via smart glasses 130. The utterance may be "Hey, assistant, take a photo. 1220". The environment may be noisy, such as windy. Fig. 12B As shown, assistant system 140 can generate an out-of-context (OOD) response that takes into account the noisy environment. The response can be "It's too noisy. I can't understand you. 1230". Alternatively, Fig. 12C As shown, when the assistant system 140 has a low confidence in what the user 1210 said, the assistant system 140 can perform task recovery. The assistant system 140 can provide a recovered out-of-domain (OOD) response, which can be "It's noisy, but do you want to take a photo? 1240".

[0191] Fig.13 An example method 1300 for generating a contextual response in a noisy environment is shown. The method may start at step 1310, where the assistant system 140 may access one or more audio signals corresponding to a speech input from a user from a head-mounted device associated with the user; wherein the one or more audio signals are collected by one or more microphones of the head-mounted device; wherein the speech input is associated with a first conversation session between the user and the assistant system 140. At step 1320, the assistant system 140 may determine one or more audio environment metrics corresponding to the speech input based on the one or more audio signals; wherein the one or more audio environment metrics include one or more of the following: ambient loudness, wind energy, or speech activity. At step 1330, the assistant system 140 may determine a noise level associated with the one or more audio signals based on the one or more audio environment metrics, wherein the noise level is above a threshold level. At step 1340, the assistant system 140 may generate one or more updated audio signals by combining the one or more audio environment metrics with the one or more audio signals, and provide the one or more updated audio signals to one or more modules associated with the assistant system 140. At step 1350, assistant system 140 may determine one or more noise signals associated with the one or more audio signals, and suppress the one or more noise signals from the one or more audio signals based on one or more of a noise suppression model, a filter bank process, an echo cancellation, or a beamforming algorithm. At step 1360, assistant system 140 may calculate a confidence score indicating the likelihood that the assistant system associated with the head mounted device understands the voice input; wherein the confidence score is less than a threshold score. At step 1370, assistant system 140 may generate a response based on one or more of the following: the confidence score, one or more audio environment metrics, or one or more of the processing results of one or more of the one or more modules associated with assistant system 140 on the one or more updated audio signals; wherein the response includes one or more of the following: an indication that the assistant system 140 does not understand the user, an indication that the user's environment is too noisy, or an implicit confirmation of a command associated with the voice input, the implicit confirmation generated based on a previous action taken by assistant system 140 during the first conversation session. At step 1380, assistant system 140 may determine one or more modalities for presenting the response; wherein the one or more modalities include one or more of the following: text modality, image modality, video modality, XR modality, sound modality, voice modality, or light modality. At step 1390, assistant system 140 may provide instructions to the head mounted device to present the response. Where appropriate, particular embodiments may repeat Fig.13 One or more steps of the method. Fig.13 The specific steps of the method are described and shown as being performed in a specific order, but the present disclosure contemplates Fig.13 In addition, although the present disclosure describes and illustrates a method for generating a contextual response in a noisy environment, including Fig.13 However, the present disclosure contemplates any suitable method for generating a contextual response in a noisy environment, including any suitable steps, which may include, where appropriate, Fig.13 All, some, or none of the steps in the method Fig.13 Furthermore, although the present disclosure describes and illustrates the steps of performing Fig.13 The present disclosure contemplates specific components, devices or systems for performing the specific steps of the method. Fig.13 Any suitable combination of any suitable components, devices, or systems for any suitable steps of the method.

[0192] privacy

[0193] In a particular embodiment, one or more objects (e.g., content or other types of objects) of a computing system may be associated with one or more privacy settings. One or more objects may be stored on or otherwise associated with any suitable computing system or application, such as a social networking system 160, a client system 130, an assistant system 140, a third-party system 170, a social networking application, an assistant application, a messaging application, a photo sharing application, or any other suitable computing system or application. Although the examples discussed herein are in the context of an online social network, these privacy settings may be applied to any other suitable computing system. The privacy settings (or "access settings") of an object may be stored in any suitable manner, such as by being associated with the object, by being indexed on an authorization server, by another suitable manner, or by any combination of these manners. The privacy settings of an object may specify how the object (or specific information associated with the object) may be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, executed, presented, or identified) within an online social network. When the privacy settings of an object allow a particular user or other entity to access the object, the object may be described as "visible" relative to the user or other entity. By way of example and not limitation, a user of an online social network may specify a privacy setting for a user profile page that identifies a set of users who may access work experience information on the user profile page, thereby denying other users access to that information.

[0194] In certain embodiments, the privacy settings of an object may specify a "blocked list" of users or other entities that should not be allowed to access certain information associated with the object. In certain embodiments, the blocked list may include third-party entities. The blocked list may specify one or more users or entities for which the object is not visible. As an example and not limitation, a user may specify a group of users that may not access an album associated with the user, thereby denying access to the album to these users (while also potentially allowing certain users not in the group of users to access the album). In certain embodiments, privacy settings may be associated with specific social graph elements. The privacy settings of a social graph element (e.g., a node or edge) may specify how an online social network may be used to access the social graph element, information associated with the social graph element, or an object associated with the social graph element. As an example and not limitation, a specific photo may have a privacy setting that specifies that only the user tagged in the photo and the friends of the user tagged in the photo can access the photo. In certain embodiments, privacy settings may allow users to choose to have or not have their content, information, or actions stored / recorded by social networking system 160 or assistant system 140, or shared with other systems (e.g., third-party systems 170). Although this disclosure describes using particular privacy settings in particular ways, this disclosure contemplates using any suitable privacy settings in any suitable way.

[0195] In certain embodiments, the social networking system 160 may present a "privacy wizard" to the first user (e.g., within a webpage, module, one or more dialog boxes, or any other suitable interface) to assist the first user in specifying one or more privacy settings. The privacy wizard may display instructions, appropriate privacy-related information, current privacy settings, one or more input fields for receiving one or more inputs from the first user to change or confirm the specified privacy settings, or any suitable combination thereof. In certain embodiments, the social networking system 160 may provide the first user with a "control panel" function that may display the first user's current privacy settings. The control panel function may be displayed to the first user at any suitable time (e.g., after input from the first user invoking the control panel function, after a particular event or triggering action occurs). The control panel function may allow the first user to modify one or more of the first user's current privacy settings at any time and in any suitable manner (e.g., redirecting the first user to the privacy wizard).

[0196] The privacy settings associated with an object can specify any suitable granularity at which access is allowed or denied. As an example and not limitation, access or denial of access can be specified for the following users: a specific user (e.g., only me, my roommate, and my boss), users within a specific degree of separation (e.g., friends, or friends of friends), user groups (e.g., a gaming club, my family), user networks (e.g., employees of a specific employer, students or alumni of a specific university), all users ("public"), no users ("private"), users of third-party systems 170, specific applications (e.g., third-party applications, external websites), other suitable entities, or any combination of the above users or entities. Although the present disclosure describes a specific granularity at which access is allowed or denied, the present disclosure contemplates any suitable granularity at which access is allowed or denied.

[0197] In certain embodiments, one or more servers 162 may be authorization / privacy servers for implementing privacy settings. In response to a request from a user (or other entity) for a particular object stored in a data repository 164, the social networking system 160 may send a request for the object to the data repository 164. The request may identify a user associated with the request, and the request may only be sent to the user (or the user's client system 130) if the authorization server determines that the user is authorized to access the object based on the privacy settings associated with the object. If the requesting user is not authorized to access the object, the authorization server may block the retrieval of the requested object from the data repository 164, or may block the sending of the requested object to the user. In the context of a search query, an object may be provided as a search result only if the querying user is authorized to access the object, for example, if the privacy settings of the object allow the object to be presented to the querying user, discovered by the querying user, or otherwise visible to the querying user. In certain embodiments, an object may represent content visible to a user through the user's news feed. As an example and not limitation, one or more objects may be visible to a user's "trending" page. In certain embodiments, an object may correspond to a specific user. The object may be content associated with the particular user, or may be an account of the particular user or information stored on the social networking system 160 or other computing system. By way of example and not limitation, a first user may view one or more second users of an online social network through a “people you may know” feature of the online social network or by viewing the first user’s friend list. By way of example and not limitation, a first user may specify that they do not want to see objects associated with a particular second user in their newsfeed or friend list. If the privacy settings of the object do not allow it to appear in front of, be discovered by, or be visible to the user, the object may be excluded from the search results. Although the present disclosure describes implementing privacy settings in a particular manner, the present disclosure contemplates implementing privacy settings in any suitable manner.

[0198] In certain embodiments, different objects of the same type associated with a user may have different privacy settings. Different types of objects associated with a user may have different types of privacy settings. As an example and not limitation, a first user may specify that the first user's status update is public, but any images shared by the first user are only visible to the first user's friends on an online social network. As another example and not limitation, a user may specify different privacy settings for different types of entities, such as individual users, friends of friends, followers, user groups, or corporate entities. As another example and not limitation, a first user may specify a group of users who may view a video posted by the first user while preventing the video from being visible to the first user's employer. In certain embodiments, different privacy settings may be provided for different user groups or user background information. As an example and not limitation, a first user may specify that other users at the same university as the first user may view the first user's photos, but other users who are family members of the first user may not view those same photos.

[0199] In certain embodiments, social networking system 160 may provide one or more default privacy settings for each object of a certain object type. The privacy setting for an object that is set to default may be changed by a user associated with the object. As an example and not limitation, all images posted by a first user may have a default privacy setting that is visible only to the first user's friends, and for a particular image, the first user may change the privacy setting for the image to be visible to friends and friends of friends.

[0200] For example, in certain embodiments, the privacy settings may allow the first user to specify (e.g., by opting out, by not opting in) whether the social networking system 160 or assistant system 140 may receive, collect, record, or store specific objects or information associated with the user for any purpose. In certain embodiments, the privacy settings may allow the first user to specify whether a specific application or process may access, store, or use specific objects or information associated with the user. The privacy settings may allow the user to opt in or opt out of having a specific application or process access, store, or use an object or information. The social networking system 160 or assistant system 140 may access such information to provide certain functions or services to the first user, while the social networking system 160 or assistant system 140 may not access the information for any other purpose. Prior to accessing, storing, or using such objects or information, the social networking system 160 or assistant system 140 may prompt the user to provide a privacy setting that specifies which applications or processes (if any) may access, store, or use the object or information before allowing any such action. As an example and not limitation, a first user may send a message to a second user via an application associated with an online social network (e.g., a messaging application) and may specify a privacy setting that social networking system 160 or assistant system 140 should not store such messages.

[0201] In certain embodiments, a user may specify whether certain types of objects or information associated with a first user may be accessed, stored, or used by social-networking system 160 or assistant system 140. By way of example and not limitation, a first user may specify that images sent by a first user via social-networking system 160 or assistant system 140 may not be stored by social-networking system 160 or assistant system 140. By way of another example and not limitation, a first user may specify that messages sent from a first user to a particular second user may not be stored by social-networking system 160 or assistant system 140. By way of yet another example and not limitation, a first user may specify that all objects sent via a particular application may be saved by social-networking system 160 or assistant system 140.

[0202] In certain embodiments, the privacy settings may allow the first user to specify whether a particular object or information associated with the first user may be accessed from a particular client system 130 or third-party system 170. The privacy settings may allow the first user to select whether to access an object or information from a particular device (e.g., a phone book on the user's smartphone), from a particular application (e.g., a messaging application), or from a particular system (e.g., an email server). The social networking system 160 or assistant system 140 may provide a default privacy setting for each device, system, or application, and / or may prompt the first user to specify a specific privacy setting for each context. As an example and not limitation, the first user may utilize the location services feature of the social networking system 160 or assistant system 140 to provide recommendations for restaurants or other places near the user. The first user's default privacy settings may specify that the social networking system 160 or assistant system 140 may use the location information provided from the first user's client system 130 to provide location-based services, but the social networking system 160 or assistant system 140 may not store the first user's location information or provide it to any third-party system 170. The first user may then update the privacy settings to allow the third-party image sharing application to use the location information in order to geotag photos.

[0203] In certain embodiments, privacy settings may allow a user to specify one or more geographic locations where an object may be accessed. Access or denial of access to an object may depend on the geographic location of the user attempting to access the object. As an example and not limitation, a user may share an object and specify that only users in the same city may access or view the object. As another example and not limitation, a first user may share an object and specify that the object may be visible to a second user only when the first user is in a particular location. If the first user leaves the particular location, the object is no longer visible to the second user. As another example and not limitation, the first user may specify that an object may only be visible to a second user within a threshold distance from the first user. If the user subsequently changes location, the original second user who had access to the object may lose access, while a new set of second users may gain access when they come within a threshold distance of the first user.

[0204] In certain embodiments, the social networking system 160 or assistant system 140 may have functionality that can use the user's personal information or biometric information as input for user authentication or experience personalization purposes. Users may choose to utilize these features to enhance their experience in online social networks. By way of example and not limitation, a user may provide personal information or biometric information to the social networking system 160 or assistant system 140. The user's privacy settings may specify that such information may only be used for a specific process (e.g., authentication), and may also specify that such information may not be shared with any third-party system 170 or used for other processes or applications associated with the social networking system 160 or assistant system 140. As another example and not limitation, the social networking system 160 may provide a user with the functionality to provide a voiceprint recording to an online social network. By way of example and not limitation, if a user wishes to utilize this functionality of an online social network, the user may provide a voice recording of his or her own voice to provide a status update to the online social network. The recording of the voice input may be compared to the user's voiceprint to determine what words the user said. The user's privacy settings may specify that such voice recordings may be used only for voice input purposes (e.g., authenticating the user, sending voice messages, improving voice recognition in order to use the voice-operated features of the online social network), and further specify that such voice recordings may not be shared with any third-party system 170 or used by other processes or applications associated with the social networking system 160.

[0205] System and method

[0206] Fig.14 An example computer system 1400 is shown. In particular embodiments, one or more computer systems 1400 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 1400 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 1400 performs one or more steps of one or more methods described or illustrated herein, or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 1400. In this document, references to a computer system may include a computing device, and vice versa, where appropriate. In addition, references to a computer system may include one or more computer systems, where appropriate.

[0207] The present disclosure contemplates any suitable number of computer systems 1400. The present disclosure contemplates computer systems 1400 in any suitable physical form. By way of example and not limitation, computer system 1400 may be an embedded computer system, a system on a chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or a system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive self-service terminal (kiosk), a mainframe, a network of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these systems. Where appropriate, computer system 1400 may include one or more computer systems 1400; may be single or distributed; across multiple locations; across multiple machines; across multiple data centers; or located in a cloud (which may include one or more cloud components in one or more networks). Where appropriate, one or more computer systems 1400 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not limitation, one or more computer systems 1400 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch processing mode. Where appropriate, one or more computer systems 1400 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations.

[0208] In a particular embodiment, computer system 1400 includes a processor 1402, memory 1404, storage 1406, an input / output (I / O) interface 1408, a communication interface 1410, and a bus 1412. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0209] In a particular embodiment, the processor 1402 includes hardware for executing a plurality of instructions, such as those that constitute a computer program. By way of example and not limitation, to execute a plurality of instructions, the processor 1402 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 1404, or storage 1406; decode and execute the instructions; and then write one or more results to an internal register, an internal cache, memory 1404, or storage 1406. In a particular embodiment, the processor 1402 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1402 that includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 1402 may include one or more instruction caches, one or more data caches, and one or more page table caches (translation lookaside buffers, TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1404 or storage 1406, and the instruction cache may speed up the retrieval of the instructions by the processor 1402. The data in the data cache may be a copy of the data in the memory 1404 or storage 1406 for the instruction operation executed at the processor 1402; may be the result of the previous instruction executed at the processor 1402, for access by the subsequent instruction executed at the processor 1402, or for writing to the memory 1404 or storage 1406; or may be other suitable data. The data cache may accelerate the read operation or write operation of the processor 1402. The TLB may accelerate the virtual address translation of the processor 1402. In a particular embodiment, the processor 1402 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1402 including any suitable number of any suitable internal registers. Where appropriate, the processor 1402 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 1402. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0210] In a particular embodiment, the memory 1404 includes a main memory for storing instructions for execution by the processor 1402 or data for operation by the processor 1402. As an example and not limitation, the computer system 1400 can load instructions from the memory 1406 or another source (e.g., another computer system 1400) to the memory 1404. The processor 1402 can then load these instructions from the memory 1404 to an internal register or an internal cache memory. In order to execute these instructions, the processor 1402 can retrieve these instructions from the internal register or the internal cache memory and decode them. During or after executing these instructions, the processor 1402 can write one or more results (which can be intermediate results or final results) to the internal register or the internal cache memory. The processor 1402 can then write one or more of these results to the memory 1404. In a particular embodiment, the processor 1402 executes only instructions in one or more internal registers or in one or more internal cache memories, or in memory 1404 (different from memory 1406 or other locations), and operates only on data in one or more internal registers or in internal cache memories, or in memory 1404 (different from memory 1406 or other locations). One or more memory buses (each memory bus may include an address bus and a data bus) can couple the processor 1402 to the memory 1404. As described below, the bus 1412 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) are located between the processor 1402 and the memory 1404 and facilitate access to the memory 1404 requested by the processor 1402. In a particular embodiment, the memory 1404 includes random access memory (RAM). Where appropriate, the RAM is a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1404 may include one or more memories 1404. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0211] In a particular embodiment, memory 1406 includes a large capacity memory for data or instructions. As an example and not limitation, memory 1406 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these memories. Where appropriate, memory 1406 may include removable media or non-removable (or fixed) media. Where appropriate, memory 1406 may be inside or outside of computer system 1400. In a particular embodiment, memory 1406 is a non-volatile solid-state memory. In a particular embodiment, memory 1406 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory, or a combination of two or more of these ROMs. The present disclosure contemplates mass storage 1406 taking any suitable physical form. Where appropriate, storage 1406 may include one or more storage control units that facilitate communication between processor 1402 and storage 1406. Where appropriate, storage 1406 may include one or more memories 1406. Although the present disclosure describes and illustrates particular storage, the present disclosure contemplates any suitable storage.

[0212] In a particular embodiment, the I / O interface 1408 includes such hardware, software, or both: the hardware, software, or both provide one or more interfaces for communication between the computer system 1400 and one or more I / O devices. Where appropriate, the computer system 1400 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and the computer system 1400. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a tracking ball, a camera, another suitable I / O device, or a combination of two or more of these I / O devices. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1408 for the I / O device. Where appropriate, the I / O interface 1408 may include one or more devices or software drivers that enable the processor 1402 to drive one or more of these I / O devices. Where appropriate, I / O interface 1408 may include one or more I / O interfaces 1408. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0213] In a particular embodiment, the communication interface 1410 includes such hardware, software, or both: the hardware, software, or both provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1400 and one or more other computer systems 1400 or one or more networks. As an example and not limitation, the communication interface 1410 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other line-based network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network such as a WI-FI network. The present disclosure contemplates any suitable network and any suitable communication interface 1410 for the network. As an example and not limitation, the computer system 1400 can communicate with the following networks: an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more parts of the Internet, or a combination of two or more of these networks. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1400 can communicate with the following networks: a wireless PAN (WPAN) (e.g., a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these networks. Where appropriate, the computer system 1400 may include any suitable communication interface 1410 for any of these networks. Where appropriate, the communication interface 1410 may include one or more communication interfaces 1410. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0214] In particular embodiments, bus 1412 includes hardware, software, or both that couples the various components of computer system 1400 to each other. By way of example and not limitation, bus 1412 may include: Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, front-side bus (FSB), Hypertransport (HT) interconnect, Industry Standard Architecture (ISA) bus, InfiniBAND interconnect, low-pin-count (LPC) bus, memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCIe) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association local (VLB) bus, or another suitable bus, or a combination of two or more of these buses. Where appropriate, bus 1412 may include one or more buses 1412. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0215] In this document, where appropriate, one or more computer-readable non-transitory storage media may include: one or more semiconductor-based integrated circuits (integrated circuits, ICs) or other ICs (e.g., field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives, tapes, solid-state drives (SSDs), RAM drives, secure digital cards (SECURE DIGITAL cards) or secure digital drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these storage media. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0216] Other parameters

[0217] As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Furthermore, "and" is joint and several, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

[0218] The scope of the present disclosure includes all changes, substitutions, changes, transformations and modifications to the example embodiments described or shown herein, which will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the various embodiments herein as including specific parts, elements, features, functions, operations, or steps, any embodiment in these embodiments may include any combination or arrangement of any of the parts, elements, features, functions, operations, or steps described or shown anywhere herein as will be understood by those of ordinary skill in the art. In addition, references to devices or systems that are suitable for, arranged to, capable of, configured to, enabled to, can be operated, or can be operated to perform a specific function, or components in a device or system in the appended claims include the device, system, component, regardless of whether the device, system, component, or the specific function is activated, turned on, or unlocked, as long as the device, system, or component is so suitable for, arranged to, capable of, configured to, enabled to, can be operated, or can be operated. Additionally, although this disclosure may describe or illustrate a particular embodiment as providing particular advantages, that particular embodiment may not provide these advantages, or may provide some or all of these advantages.< / domain> < / domian>

Claims

1. A method comprising: One or more computing systems: accessing, from a head mounted device associated with a user, one or more audio signals corresponding to speech input from the user; determining, based on the one or more audio signals, one or more audio environment metrics corresponding to the speech input; calculating a confidence score indicating a likelihood that an assistant system associated with the head mounted device understood the speech input; generating a response based on the confidence score and the one or more audio environment metrics; as well as Instructions are provided to the head mounted device to present the response.

2. The method according to claim 1, further comprising: A noise level associated with the one or more audio signals is determined based on the one or more audio environment metrics, wherein the noise level is above a threshold level.

3. The method according to claim 1, wherein: The one or more audio signals are collected by one or more microphones of the head mounted device.

4. The method according to claim 1, wherein: The one or more audio environment metrics include one or more of: ambient loudness, wind energy, or voice activity.

5. The method according to claim 1, further comprising: determining one or more noise signals associated with the one or more audio signals; as well as The one or more noise signals are suppressed from the one or more audio signals based on a noise suppression model.

6. The method according to claim 5, further comprising: A transcription of the speech input is generated based on the noise suppressed audio signal through an automatic speech recognition (ASR) model.

7. The method according to claim 5, wherein: Suppressing the one or more noise signals from the one or more audio signals is also based on one or both of filter bank processing or echo cancellation.

8. The method according to claim 5, wherein: Suppressing the one or more noise signals from the one or more audio signals includes applying a beamforming algorithm to the one or more audio signals.

9. The method according to claim 1, further comprising: generating one or more updated audio signals by combining the one or more audio environment metrics with the one or more audio signals; as well as providing the one or more updated audio signals to one or more modules associated with the assistant system; Wherein, generating the response is also based on the processing results of the one or more updated audio signals by one or more of the one or more modules associated with the assistant system.

10. The method according to claim 1, wherein: The confidence score is less than a threshold score.

11. The method according to claim 1, wherein: The response includes one or more of the following: The assistant system does not understand the user's indication; an indication that the user's environment is too noisy; or An implicit confirmation of a command associated with the voice input.

12. The method according to claim 1, wherein: The response includes an implicit confirmation of a command associated with the voice input, wherein the voice input is associated with a first conversation session between the user and the assistant system, and wherein the implicit confirmation is generated based on a previous action taken by the assistant system during the first conversation session.

13. The method according to claim 1, further comprising: One or more modalities for presenting the response are determined.

14. The method according to claim 13, wherein: The one or more modalities include one or more of: a text modality, an image modality, a video modality, an XR modality, a sound modality, a voice modality, or a light modality.

15. The method according to claim 1, wherein: The one or more modalities include one or more of a video modality, an audio modality, or a voice modality, wherein the method further comprises: A threshold amount of audio signal enhancement associated with the presented response is determined, wherein the instructions include instructions for the determined enhancement.

16. The method according to claim 1, wherein: The speech input is associated with a first conversation session between the user and the assistant system, and wherein generating the response is also based on one or both of: a previous conversation state associated with the first conversation session or a device context associated with the head mounted device.

17. One or more computer-readable non-transitory storage media containing software that, when executed, is operable to: accessing, from a head mounted device associated with a user, one or more audio signals corresponding to speech input from the user; determining, based on the one or more audio signals, one or more audio environment metrics corresponding to the speech input; calculating a confidence score indicating a likelihood that an assistant system associated with the head mounted device understood the speech input; generating a response based on the confidence score and the one or more audio environment metrics; as well as Instructions are provided to the head mounted device to present the response.

18. A system comprising: one or more processors; and a non-transitory memory coupled to the one or more processors, the non-transitory memory comprising instructions executable by the one or more processors, the one or more processors being operable, when executing the instructions, to: accessing, from a head mounted device associated with a user, one or more audio signals corresponding to speech input from the user; determining, based on the one or more audio signals, one or more audio environment metrics corresponding to the speech input; calculating a confidence score indicating a likelihood that an assistant system associated with the head mounted device understood the speech input; generating a response based on the confidence score and the one or more audio environment metrics; as well as Instructions are provided to the head mounted device to present the response.

Images