Rendering XR avatars based on acoustic features

Abstract

The present disclosure provides rendering an augmented reality (XR) avatar based on acoustic features. In one embodiment, a method includes receiving a speech input from a first client system associated with a first user, the speech input having a plurality of acoustic features; determining, by a machine learning model, an emotion associated with the speech input based on one or more of the plurality of acoustic features; determining a facial feature representing a first augmented reality (XR) avatar of the first user based on the emotion; and sending an instruction to a second client system associated with a second user to render a first XR avatar representing the first user, where the first XR avatar is rendered with the determined facial features.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. non-provisional patent application No. 18 / 353,693, filed on July 17, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to database and file management within a network environment, and more particularly to hardware and software for an extended reality system. Background Art

[0004] Extended reality (or artificial reality) is a form of reality that has been adjusted in some way before being presented to the user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Extended reality content may include fully generated content, or generated content combined with captured content (e.g., real-world photos). Extended reality content may include video, audio, tactile feedback, or some combination thereof, and any of the above may be presented in a single channel or multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). Extended reality may be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in an extended reality environment and / or to be used in an extended reality environment (e.g., to perform activities in an extended reality). An extended reality system that provides extended reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers. Summary of the invention

[0005] In certain embodiments, extended-reality (XR) systems can provide users with a variety of unique experiences, from exploring the real and virtual worlds (metaverse), improving productivity, and enhancing real-time collaboration with colleagues, to playing with friends and family, to new games and interactive applications. XR systems can not only fully immerse users in digital environments, but also enable users to move freely around and interact with people and objects in their physical space while enjoying virtual content, thereby unlocking a new class of AR-like experiences in VR that enhance their physical space. In certain embodiments, XR systems can mix virtual and physical worlds. XR systems can digitally reconstruct the user's physical space into a 3D space through understanding the objects and surfaces around the user. XR systems can utilize a variety of hardware and software technologies, such as high-resolution sensors and artificial intelligence (AI)-driven scene understanding. XR systems can transform a deep understanding of how users perceive the world around them and connect this perception with immersive experiences such as these: These immersive experiences inspire users' interest in the possibilities of virtual reality, augmented reality, and mixed reality. Although this disclosure describes a particular XR system in a particular manner, this disclosure contemplates any suitable XR system in any suitable manner.

[0006] In certain embodiments, an extended reality (XR) system may detect emotions in a voice conversation in real time based on acoustic signals for a variety of use cases. As an example and not limitation, an XR system may support expressive interactions, enhanced calls, and smart communications via avatars by identifying underlying emotions in the speaker's voice. In certain embodiments, the XR system may respect the privacy of all parties involved and enable authentic, inclusive, and appropriate virtual self-expression. In addition, the XR system may support action-oriented expressions because the XR system may not have to rely on a camera or a user to express emotions. Experiments on public benchmark datasets demonstrate the commitment of the embodiments disclosed herein to supporting a variety of use cases. Although the present disclosure describes detecting emotions for a particular use case in a particular manner by a particular system, the present disclosure contemplates detecting emotions for any suitable use case in any suitable manner by any suitable system (e.g., an assistant system).

[0007] In certain embodiments, the XR system may receive a voice input from a first client system associated with a first user, the voice input having a plurality of acoustic features. The XR system may then determine, through one or more machine learning models, one or more emotions associated with the voice input based on one or more of the plurality of acoustic features. The XR system may then determine one or more facial features of a first extended reality (XR) avatar representing the first user based on the one or more emotions. In certain embodiments, the XR system may also send instructions to a second client system associated with a second user to render a first XR avatar representing the first user. The first XR avatar may be rendered using the determined one or more facial features.

[0008] There are certain technical challenges for providing real-time avatar expressivity. One technical challenge may include limiting the computation, model size, and latency of implementation on the device, and improving the user experience. A solution to this challenge presented by the embodiments disclosed herein may be to use an end-to-end emotion detection model that acquires user input and outputs emotion classification in one shot, which may help reduce computation, model size, and latency. Another technical challenge may include effectively detecting emotions in speech. A solution to this challenge presented by the embodiments disclosed herein may be to use one or more acoustic models (e.g., an acoustic model including an acoustic model based on a Mel-scale spectrogram of an audio signal), because the one or more acoustic models can be used to exploit patterns of prosody, intonation, and stress in speech for emotion detection.

[0009] Certain embodiments disclosed herein may provide one or more technical advantages. A technical advantage of an embodiment may include improving the accuracy of emotion detection, because the assistant system 140 can use only an acoustic model to directly detect emotions from an audio signal without using an automatic speech recognition (ASR) model to avoid compound errors introduced by these ASR models when generating inaccurate transcriptions. Another technical advantage of an embodiment may include improving the user experience, because the assistant system can use the emotions detected in the user's voice to adjust the assistant service to reflect such emotions. Certain embodiments disclosed herein may not provide the above technical advantages, or may provide some or all of the above technical advantages. In view of the drawings, description, and claims of the present disclosure, one or more other technical advantages may be apparent to those skilled in the art.

[0010] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to these embodiments. A specific embodiment may include all, some, or not include the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. The attached claims relating to methods, storage media, systems, and computer program products disclose embodiments according to the present invention, in particular, wherein any feature mentioned in one claim category (e.g., method) may also be protected in another claim category (e.g., system). The dependencies or backreferences in the attached claims are selected for formal reasons only. However, any subject matter generated from an intentional reference to any previous claim (especially multiple dependent claims) may also be claimed, so that any combination of multiple claims and multiple features thereof are disclosed and may be claimed regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features stated in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any features in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An example network environment associated with an assistant system is shown.

[0012] Figure 2 An example augmented reality system is shown.

[0013] Figure 3 An example of a virtual reality (VR) system worn by a user is shown.

[0014] Figure 4 Example trends of emotion detection accuracy with respect to the length of the audio segments used for inference are shown.

[0015] Figure 5 An example user session represented by an avatar is shown.

[0016] Figure 6 Example methods for providing real-time avatar expression are shown.

[0017] Figure 7 An example architecture for an assistant system is shown.

[0018] Figure 8 An example flow diagram of an assistant system is shown.

[0019] Fig. 9 An example task-centric flow chart for processing user input is shown.

[0020] Fig.10 An example artificial neural network is shown.

[0021] Fig.11 An example computer system is shown. DETAILED DESCRIPTION

[0022] System Overview

[0023] Figure 1 An example network environment 100 associated with an extended reality system is shown. The network environment 100 includes a client system 130, an assistant system 140, an extended reality (XR) system 160, and a third-party system 170 connected to each other via a network 110. Figure 1 A particular arrangement of client system 130, assistant system 140, extended reality system 160, third-party system 170, and network 110 is shown, but the present disclosure contemplates any suitable arrangement of client system 130, assistant system 140, extended reality system 160, third-party system 170, and network 110. By way of example and not limitation, two or more of client system 130, extended reality system 160, assistant system 140, and third-party system 170 may be directly connected to each other, bypassing network 110. As another example, two or more of client system 130, assistant system 140, extended reality system 160, and third-party system 170 may be physically or logically co-located in whole or in part with each other. Furthermore, although Figure 1 A particular number of client systems 130, assistant systems 140, extended reality systems 160, third-party systems 170, and networks 110 are shown, but the present disclosure contemplates any suitable number of client systems 130, assistant systems 140, extended reality systems 160, third-party systems 170, and networks 110. By way of example and not limitation, network environment 100 may include multiple client systems 130, multiple assistant systems 140, multiple extended reality systems 160, multiple third-party systems 170, and multiple networks 110.

[0024] The present disclosure contemplates any suitable network 110. By way of example and not limitation, one or more portions of network 110 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a Public Switched Telephone Network (PSTN), a network based on cellular technology, a network based on satellite communication technology, another network 110, or a combination of two or more such networks 110.

[0025] A plurality of links 150 may connect the client system 130, the assistant system 140, the extended reality system 160, and the third-party system 170 to the communication network 110, or to each other. The present disclosure contemplates any suitable links 150. In particular embodiments, the one or more links 150 include one or more wired (e.g., Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)) links, one or more wireless (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or one or more optical (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In certain embodiments, one or more links 150 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 150, or a combination of two or more such links 150. The multiple links 150 need not all be identical throughout the network environment 100. In one or more aspects, the one or more first links 150 can be different from the one or more second links 150.

[0026] In certain embodiments, client system 130 may be any suitable electronic device including hardware, software, or embedded logic components or a combination of two or more such components, and may be capable of performing the functions implemented or supported by client system 130. By way of example and not limitation, client system 130 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a global positioning system (GPS) device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a smart speaker, a smart watch, smart glasses, augmented reality (AR) smart glasses, virtual reality (VR) head-mounted devices (headsets), other suitable electronic devices, or any suitable combination thereof. In certain embodiments, client system 130 may be a smart assistant device. More information about smart assistant devices can be found in the following documents: U.S. Patent Application No. 15 / 949011 filed on April 9, 2018, U.S. Patent Application No. 16 / 153574 filed on October 5, 2018, U.S. Design Patent Application No. 29 / 631910 filed on January 3, 2018, U.S. Design Patent Application No. 29 / 631747 filed on January 2, 2018, U.S. Design Patent Application No. 29 / 631913 filed on January 3, 2018, and U.S. Design Patent Application No. 29 / 631914 filed on January 3, 2018, each of which is incorporated by reference. The present disclosure contemplates any suitable client system 130. In a particular embodiment, the client system 130 can enable a network user at the client system 130 to access the network 110. The client system 130 can also enable the user to communicate with other users at other client systems 130.

[0027] In certain embodiments, the client system 130 may include a web browser 132 and may have one or more add-ons, plug-ins, or other extensions. A user at the client system 130 may enter a Uniform Resource Locator (URL) or other address that directs the web browser 132 to a specific server (e.g., server 162, or a server associated with a third-party system 170), and the web browser 132 may generate a Hyper Text Transfer Protocol (HTTP) request and transmit the HTTP request to the server. The server may receive the HTTP request and transmit one or more Hyper Text Markup Language (HTML) files to the client system 130 in response to the HTTP request. The client system 130 may render a web interface (e.g., a web page) based on an HTML file from a server for presentation to a user. The present disclosure contemplates any suitable source file. As an example and not limitation, a web interface may be rendered according to an HTML file, an Extensible Hyper Text Markup Language (XHTML) file, or an Extensible Markup Language (XML) file, depending on specific needs. Such an interface may also execute scripts, a combination of markup languages ​​and scripts, etc. In this document, where appropriate, a reference to a web interface includes one or more corresponding source files (which a browser may use to render the web interface), and vice versa.

[0028] In certain embodiments, client system 130 may include social networking application 134 installed on client system 130. A user at client system 130 may use social networking application 134 to access an online social network. A user at client system 130 may use social networking application 134 to communicate with the user's social connections (e.g., friends, followers, followed accounts, contacts, etc.). A user at client system 130 may also use social networking application 134 to interact with a plurality of content objects (e.g., posts, news articles, ephemeral content, etc.) on an online social network. By way of example and not limitation, a user may use social networking application 134 to browse trending topics and breaking news.

[0029] In a particular embodiment, the client system 130 may include an assistant application 136. A user at the client system 130 may use the assistant application 136 to interact with the assistant system 140. In a particular embodiment, the assistant application 136 may include an assistant xbot function, which is used as a front-end interface to interact with the user of the client system 130 (including receiving user input and presenting output). In a particular embodiment, the assistant xbot may be the type of a chatbot. The assistant xbot may include a programmable service channel, which may be a software code, logic or routine used as a user personal assistant. The assistant xbot may serve as a user portal of the assistant system 140. Therefore, the assistant xbot may be considered as a dialogue agent. In a particular embodiment, the assistant application 136 may include an independent application. In a particular embodiment, the assistant application 136 may be integrated into a social network application 134 or another suitable application (e.g., a messaging application). In a particular embodiment, the assistant application 136 may also be integrated into the client system 130, an assistant hardware device, or any other suitable hardware device. In a particular embodiment, the assistant application 136 may also be a part of the assistant system 140. In certain embodiments, the assistant application 136 may be accessed through a web browser 132. In certain embodiments, a user may interact with the assistant system 140 in the following manner: user input is provided to the assistant application 136 through various modalities (e.g., audio, voice, text, vision, image, video, gesture, motion, activity, position, orientation). The assistant application 136 may transmit the user input to the assistant system 140 (e.g., through the assistant xbot). The assistant system 140 may generate a response based on the user input. The assistant system 140 may send the generated response to the assistant application 136. The assistant application 136 may then present the response to the user at the client system 130 through various modalities (e.g., audio, text, image, and video). As an example and not limitation, a user may interact with the assistant system 140 by providing user input (e.g., a verbal request for information about the current state of nearby vehicle traffic) to the assistant xbot via a microphone of the client system 130. The assistant application 136 may then transmit the user input to the assistant system 140 through the network 110. Assistant system 140 may analyze the user input accordingly, generate a response based on the analysis of the user input (e.g., vehicle traffic information obtained from a third-party source), and transmit the generated response back to assistant application 136. Assistant application 136 may then present the generated response to the user in any suitable manner (e.g., display on a display of client system 130: a text-based push notification and / or one or more images of a local map showing nearby vehicle traffic).

[0030] In certain embodiments, client system 130 may implement wake-up word detection techniques to allow a user to conveniently activate assistant system 140 using one or more wake-up words associated with assistant system 140. By way of example and not limitation, a system audio API on client system 130 may continuously monitor user input including audio data (e.g., voice data frames) received at client system 130. In this example, the wake-up word associated with assistant system 140 may be a voice phrase "Hey Assistant". In this example, when the system audio API on client system 130 detects the voice phrase "Hey Assistant" in the monitored audio data, assistant system 140 may be activated for subsequent interaction with the user. In alternative embodiments, similar detection techniques may be implemented to activate assistant system 140 using specific non-audio user input associated with assistant system 140. For example, the non-audio user input may be a specific visual signal detected by a low-power sensor (e.g., a camera) of client system 130. By way of example and not limitation, the visual signal may be a static image (e.g., a bar code, a quick response (QR) code, a universal product code (UPC)), the user's location (e.g., the user's gaze at client system 130), user motion (e.g., the user pointing at an object), or any other suitable visual signal.

[0031] In certain embodiments, client system 130 may include rendering device 137 and, optionally, companion device 138. Rendering device 137 may be configured to render output generated by assistant system 140 to a user. Companion device 138 may be configured to perform computations locally on companion device 138 (i.e., on the device) in certain circumstances, such as when rendering device 137 is unable to perform computations associated with a particular task (e.g., communication with assistant system 140). In certain embodiments, client system 130, rendering device 137, and / or companion device 138 may each be a suitable electronic device including hardware, software, or embedded logic components or a combination of two or more such components, and client system 130, rendering device 137, and / or companion device 138 may be capable of independently or collaboratively performing the functionality implemented or supported by client system 130 as described herein. By way of example and not limitation, the client system 130, the rendering device 137, and / or the companion device 138 may each include a computer system, such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a smart speaker, a virtual reality (VR) head-mounted device, an augmented reality (AR) smart glasses, other suitable electronic devices, or any suitable combination thereof. In a particular embodiment, one or more of the client system 130, the rendering device 137, and the companion device 138 may operate as a smart assistant device. By way of example and not limitation, the rendering device 137 may include smart glasses, and the companion device 138 may include a smart phone. By way of another example and not limitation, the rendering device 137 may include a smart watch, and the companion device 138 may include a smart phone. By way of yet another example and not limitation, the rendering device 137 may include smart glasses, and the companion device 138 may include a smart remote control for the smart glasses. By way of yet another example and not limitation, the rendering device 137 may include a VR / AR head-mounted device, and the companion device 138 may include a smart phone.

[0032] In certain embodiments, a user may use rendering device 137 or companion device 138, either alone or in combination, to interact with assistant system 140. In certain embodiments, one or more of client system 130, rendering device 137, and companion device 138 may implement a multi-stage wake-up word detection model to enable a user to conveniently activate assistant system 140 by continuously monitoring one or more wake-up words associated with assistant system 140. In the first stage of the wake-up word detection model, rendering device 137 may receive audio user input (e.g., a voice data frame). If a wireless connection between rendering device 137 and companion device 138 is available, an application on rendering device 137 may transmit the received audio user input to a companion application on companion device 138 via a wireless connection. In the second stage of the wake-up word detection model, a companion application on companion device 138 may process the received audio user input to detect a wake-up word associated with assistant system 140. Then, the companion application on companion device 138 may transmit the detected wake-up word to a server associated with assistant system 140 via wireless network 110. In the third stage of the wake-up word detection model, a server associated with assistant system 140 may perform keyword verification on the detected wake-up word to verify whether the user intends to activate assistant system 140 and receive help from assistant system 140. In alternative embodiments, any of the processing, detection, or keyword verification may be performed by rendering device 137 and / or companion device 138. In particular embodiments, when assistant system 140 has been activated by a user, an application on rendering device 137 may be configured to receive user input from the user, and a companion application on companion device 138 may be configured to process user input (e.g., user request) received by the application on rendering device 137. In particular embodiments, rendering device 137 and companion device 138 may be associated (i.e., paired) with each other via one or more wireless communication protocols (e.g., Bluetooth).

[0033] The following example workflow illustrates how the rendering device 137 and the companion device 138 may process user input provided by a user. In this example, an application on the rendering device 137 may receive user input including a user request directed to the rendering device 137. The application on the rendering device 137 may then determine the state of the wireless connection between the rendering device 137 and the companion device 138 (i.e., the tethering state). If the wireless connection between the rendering device 137 and the companion device 138 is not available, the application on the rendering device 137 may transmit the user request (optionally including additional data and / or contextual information available to the rendering device 137) to the assistant system 140 via the network 110. The assistant system 140 may then generate a response to the user request and transmit the generated response back to the rendering device 137. The rendering device 137 may then present the response to the user in any suitable manner. Alternatively, if a wireless connection between rendering device 137 and companion device 138 is available, the application on rendering device 137 can transmit the user request (optionally including additional data and / or context information available to rendering device 137) to the companion application on companion device 138 via the wireless connection. The companion application on companion device 138 can then transmit the user request (optionally including additional data and / or context information available to companion device 138) to assistant system 140 via network 110. Assistant system 140 can then generate a response to the user request and transmit the generated response back to companion device 138. The companion application on companion device 138 can then transmit the generated response to the application on rendering device 137. Rendering device 137 can then present the response to the user in any suitable manner. In the previous example workflow, rendering device 137 and companion device 138 can each perform one or more calculations and / or processing at each corresponding step of the workflow. In certain embodiments, the performance of computations and / or processing disclosed herein may be adaptively switched between the rendering device 137 and the companion device 138 based at least in part on the device state of the rendering device 137 and / or the companion device 138, the task associated with the user input, and / or one or more additional factors. By way of example and not limitation, one factor may be the signal strength of the wireless connection between the rendering device 137 and the companion device 138. For example, if the signal strength of the wireless connection between the rendering device 137 and the companion device 138 is strong, the computations and processing may be adaptively switched to be substantially performed by the companion device 138, e.g., to benefit from the greater processing power of the central processing unit (CPU) of the companion device 138. Alternatively, if the signal strength of the wireless connection between the rendering device 137 and the companion device 138 is weak, the computations and processing may be adaptively switched to be substantially performed by the rendering device 137 in an independent manner.In certain embodiments, if the client system 130 does not include the companion device 138, the aforementioned calculations and processing may be performed solely by the rendering device 137 in an independent manner.

[0034] In certain embodiments, assistant system 140 can assist a user in performing various assistant-related tasks. Assistant system 140 can interact with augmented reality system 160 and / or third-party system 170 when performing these assistant-related tasks.

[0035] In certain embodiments, the extended reality system 160 may be a network addressable computing system that may host an online social network. The extended reality system 160 may generate, store, receive, and send social network data, such as user profile data, concept profile data, social graph information, or other suitable data related to an online social network. The extended reality system 160 may be accessed directly by other components in the network environment 100 or via the network 110. As an example and not limitation, the client system 130 may use a web browser 132 or a local application associated with the extended reality system 160 (e.g., a mobile social network application, a messaging application, another suitable application, or any combination thereof) to access the extended reality system 160 directly or via the network 110. In certain embodiments, the extended reality system 160 may include one or more servers 162. Each server 162 may be a single server or a distributed server across multiple computers or multiple data centers. As an example and not limitation, each server 162 may be a network server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In a particular embodiment, each server 162 may include hardware, software, or embedded logic components, or a combination of two or more such components, for performing the appropriate functions implemented or supported by the server 162. In a particular embodiment, the extended reality system 160 may include one or more data repositories 164. The data repository 164 may be used to store various types of information. In a particular embodiment, the information stored in the data repository 164 may be organized according to a particular data structure. In a particular embodiment, each data repository 164 may be a relational database, a columnar database, a correlation database, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. Particular embodiments may provide an interface that enables client system 130 , extended reality system 160 , assistant system 140 , or third-party system 170 to manage, retrieve, modify, add, or delete information stored in data repository 164 .

[0036] In certain embodiments, the extended reality system 160 may store one or more social graphs in one or more data repositories 164. In certain embodiments, the social graph may include a plurality of nodes—these nodes may include a plurality of user nodes (each user node corresponds to a particular user) or a plurality of concept nodes (each concept node corresponds to a particular concept)—and a plurality of edges connecting these nodes. The extended reality system 160 may provide users of an online social network with the ability to communicate and interact with other users. In certain embodiments, a user may join an online social network via the extended reality system 160 and may then add connections (e.g., relationships) with a plurality of other users of the extended reality system 160 to whom the user would like to establish connections. As used herein, the term "friend" may refer to any other user of the extended reality system 160 with whom the user has formed a connection, association, or relationship via the extended reality system 160.

[0037] In certain embodiments, extended reality system 160 may provide users with the ability to take actions on various types of items or objects supported by extended reality system 160. By way of example and not limitation, these items and objects may include groups or social networks to which a user of extended reality system 160 may belong, events or calendar entries that may be of interest to the user, computer-based applications that the user may use, transactions that allow the user to buy or sell items via a service, interactions with advertisements that the user may perform, or other suitable items or objects. The user may interact with anything that can be represented in extended reality system 160 or represented by an external system of third-party system 170 that is separate from extended reality system 160 and coupled to extended reality system 160 via network 110.

[0038] In certain embodiments, extended reality system 160 may be able to link various entities. By way of example and not limitation, extended reality system 160 may enable users to interact with each other and receive content from third-party systems 170 or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0039] In certain embodiments, third-party system 170 may include one or more types of servers, one or more data repositories, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable component with which, for example, a server can communicate. Third-party system 170 may be operated by an entity different from the entity operating extended reality system 160. However, in certain embodiments, extended reality system 160 and third-party system 170 may operate in conjunction with each other to provide social networking services to users of extended reality system 160 or third-party system 170. In this sense, extended reality system 160 may provide a platform or backbone that other systems (such as third-party system 170) can use to provide social networking services and functions to users over the Internet.

[0040] In certain embodiments, third-party system 170 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be transmitted to client system 130. By way of example and not limitation, a content object may include information about things or activities of interest to a user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not limitation, a content object may include an incentive content object, such as a coupon, discount ticket, gift certificate, or other suitable incentive object. In certain embodiments, a third-party content provider may use one or more third-party agents to provide content objects and / or services. A third-party agent may be an implementation hosted and executed on third-party system 170.

[0041] In certain embodiments, the extended reality system 160 also includes user-generated content objects that can enhance the user's interaction with the extended reality system 160. User-generated content can include any content that a user can add, upload, send, or "post" to the extended reality system 160. By way of example and not limitation, a user transmits a post from the client system 130 to the extended reality system 160. Posts can include data such as status updates or other text data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to the extended reality system 160 by third parties through "communication channels" (e.g., news feeds or streams).

[0042] In a particular embodiment, the extended reality system 160 may include various servers, subsystems, programs, modules, logs, and data repositories. In a particular embodiment, the extended reality system 160 may include one or more of the following: a network server, an action recorder, an API request server, a correlation and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object public log, an inference module, an authorization / privacy server, a search module, an advertising positioning module, a user interface module, a user profile repository, a connection repository, a third-party content repository, or a location repository. The extended reality system 160 may also include suitable components, such as a network interface, a security mechanism, a load balancer, a failover server, a management and network operation console, other suitable components, or any suitable combination thereof. In a particular embodiment, the extended reality system 160 may include one or more user profile repositories for storing user profiles. A user profile may include, for example, biometric information, personal background information, behavioral information, social information, or other types of descriptive information, such as work experience, educational history, hobbies or preferences, interests, in-laws, or addresses. Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not limitation, if a user "likes" an item about a shoe brand, the category may be that brand, or may be a general category of "shoes" or "clothing". A connection repository may be used to store connection information about a user. The connection information may indicate users with similar or common work experience, group membership, hobbies, educational history, or users who are related or share common attributes in any way. The connection information may also include user-defined connections between different users and content (both internal and external). A network server may be used to link the extended reality system 160 to one or more client systems 130 or one or more third-party systems 170 via a network 110. The network server may include a mail server or other messaging functionality for receiving and routing messages between the extended reality system 160 and one or more client systems 130. The API request server may allow, for example, an assistant system 140 or a third-party system 170 to access information from the extended reality system 160 by calling one or more APIs. An action recorder may be used to receive communications from a network server related to a user's actions to log in or out of the extended reality system 160. Together with the action log, a third-party content object log may be saved for user contact with a third-party content object. The notification controller can provide information about the content object to the client system 130. The information can be pushed to the client system 130 as a notification, or can be pulled from the client system 130 in response to user input including a user request received from the client system 130.The authorization server can be used to enforce one or more privacy settings for users of the extended reality system 160. The privacy settings of the user can determine how specific information associated with the user can be shared. The authorization server can allow the user to opt in or out of having their actions recorded by the extended reality system 160, or sharing their actions with other systems (e.g., third-party systems 170), for example, by setting appropriate privacy settings. A third-party content object repository can be used to store content objects received from a third party (e.g., third-party system 170). A location repository can be used to store location information received from a client system 130 associated with a user. The advertising pricing module can combine social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0043] Augmented reality system

[0044] Figure 2An example augmented reality (AR) system 200 is shown. In certain embodiments, the augmented reality system 200 may perform one or more processes described herein. The augmented reality system 200 may include a head-mounted AR display device 210 (e.g., glasses) that includes a frame 212, one or more displays 214, and a computing system 216 (the computing system may be, for example, a companion device 138). The display 214 may be transparent or translucent, allowing a user wearing the AR display device 210 to see through the display 214 to see the real world environment, and simultaneously display visual artificial reality content to the user (e.g., AR content objects may be rendered on the display 214 as if overlaying the real world environment). The AR display device 210 may include an audio device that may provide audio content to the user. The AR display device 210 may include one or more cameras that may capture images and videos of the real world environment. The AR display device 210 may include an eye tracking system for tracking the vergence movement of a user wearing the AR display device 210. The AR display device 210 may include a microphone for collecting voice input from the user. The augmented reality system 200 may also include a controller including a touchpad and one or more buttons. The controller may receive input from the user and relay these inputs to the computing system 216. The controller may also provide tactile feedback to the user. The computing system 216 may be connected to the AR display device 210 and the controller via a wired or wireless connection. The computing system 216 may control the AR display device 210 and the controller to provide augmented reality content to the user and receive input from the user. The computing system 216 may be an independent host computer device, an onboard computer device integrated with the AR display device 210, a mobile device, or any other hardware platform capable of providing augmented reality content to the user and receiving input from the user.

[0045] Object tracking in the image domain is a known technique. For example, a stationary camera may capture video of a moving object, and a computing system may compute, for each frame, the 3D position of an object of interest or one of a plurality of observable features of the object of interest relative to the camera. When the camera is stationary, any changes in the position of the object can only be attributed to the movement of the object and / or jitter caused by the tracking algorithm. In this case, the motion of the tracked object may be smoothed in time by simply applying a suitable averaging algorithm (e.g., averaging with exponential time decay) to the current estimated position of the object and one or more previous estimated positions of the object.

[0046] In the context of augmented reality, motion smoothing becomes more complicated. For augmented reality systems, an outward-facing camera is usually mounted on the HMD, so that the outward-facing camera can capture video of another moving object while moving with the user's head. When such a non-stationary camera is used to track a moving object, the position change of the tracked object may be due not only to the movement of the object, but also to the movement of the camera. Therefore, the aforementioned methods for smoothing the position of the tracked object in time will no longer work.

[0047] Virtual reality system

[0048] Figure 3 An example of a virtual reality (VR) system 300 worn by a user 302 is shown. In certain embodiments, the VR system 300 may include a head-mounted VR display device 304, a controller 306, and one or more computing systems 308 (the one or more computing systems may be, for example, the companion device 138). The VR display device 304 may be worn on the user's eyes and provide visual content to the user 302 through an internal display (not shown). The VR display device 304 may have two independent internal displays, one for each eye of the user 302 (a single display device is also possible). In certain embodiments, the VR display device 304 may include one or more outward-facing cameras, such as two forward-facing cameras 305A and 305B, which may capture images and videos of the real-world environment. The VR system 300 may also include one or more computing systems 308. The one or more computing systems 308 may be independent units physically separated from the VR display device 304, or the one or more computing systems 308 may be integrated with the VR display device 304. In an embodiment where the one or more computing systems 308 are independent units, the one or more computing systems 308 may be communicatively coupled to the VR display device 304 via a wireless link or a wired link. The one or more computing systems 308 may be high-performance devices (e.g., desktop computers or laptop computers) or resource-constrained devices (e.g., mobile phones). High-performance devices may have a dedicated graphics processor (GPU) and a high-capacity or constant power supply. In other words, resource-constrained devices may not have a GPU and may have limited battery capacity. Therefore, the algorithms that the VR system 300 can actually use depend on the capabilities of the one or more computing systems 308 of the VR system.

[0049] Rendering XR avatars based on acoustic signatures

[0050] In certain embodiments, the XR system 160 may detect emotions in voice conversations in real time based on acoustic signals for a variety of use cases. As an example and not limitation, the XR system 160 may support expressive interactions, enhanced calls, and intelligent communications via avatars by identifying potential emotions in the speaker's voice. In certain embodiments, the XR system 160 may respect the privacy of the parties involved and enable real, inclusive, and appropriate virtual self-expression. In addition, the XR system 160 may support action-oriented expressions because the XR system may not have to rely on cameras or users who want to express emotions. Experiments on public benchmark datasets demonstrate the commitment of the embodiments disclosed herein to supporting a variety of use cases. Although the present disclosure describes detecting emotions for a particular use case in a particular manner by a particular system, the present disclosure contemplates detecting emotions for any suitable use case in any suitable manner by any suitable system (e.g., assistant system 140).

[0051] In certain embodiments, the XR system 160 may receive a voice input from a first client system 130 associated with a first user, the voice input having a plurality of acoustic features. The XR system 160 may then determine one or more emotions associated with the voice input based on one or more of the plurality of acoustic features through one or more machine learning models. The XR system 160 may then determine one or more facial features of a first extended reality (XR) avatar representing the first user based on the one or more emotions. In certain embodiments, the XR system 160 may also send instructions to a second client system 130 associated with a second user to render a first XR avatar representing the first user. The first XR avatar may be rendered using the determined one or more facial features. For example, the first client system 130 and the second client system 130 may be an AR system 200 or a VR system 300. In certain embodiments, the XR system 160 may also render a more sophisticated animation of the first XR avatar based on the detected emotions. By way of example and not limitation, if the first user says "congratulations," the XR system 160 may also render confetti and a pre-recorded congratulatory animation to create a more enjoyable experience.

[0052] Regardless of whether the modality of communication is audio or video, there may be a potential conversation. A conversation can be a user monologue, a binary exchange, or a multi-party conversation. A conversation may span multiple dialog sessions between multiple people in the same group, and may even be intertwined. Some conversations may be topic-centric, while others may be general chats. In addition, a conversation may be audio-only, text-only, or multimodal, which includes audio and video of the speaker. (However, note that the embodiments disclosed herein focus on conversations that include at least an audio component). For a given conversation, the XR system 160 as a voice interaction system may be able to detect the user's potential emotions based on a predetermined list of emotions (e.g., happy, sad, angry, surprised, disgusted, etc.). The detected emotions can also be used for a variety of use cases. In addition to detecting emotions, the XR system 160 can also detect the intensity of such emotions to appropriately fine-tune the rendering.

[0053] The following are some example use cases for which emotion detection by the XR system 160 can be used. The primary use case can be avatar expression. In a particular embodiment, a first user can have a conversation with a second user in an XR environment that includes a first XR avatar. Avatars may be particularly relevant to virtual reality (VR) devices. When users are immersed in a virtual world using a virtual reality device, the avatar represents these users. The XR system 160 can provide detected emotion information through effective emotion detection to promote vivid and expressive avatars, making interactions in the virtual world enjoyable, interesting, and inclusive. Note that the emotion detection of the XR system 160 can be equally applicable to avatars in augmented reality (AR) use cases in addition to VR use cases.

[0054] VR and AR devices may support facial tracking. Thus, expressive avatars may be implemented using facial tracking with a camera facing the user. However, in some cases, facial tracking may not be feasible or comprehensive, limiting its application to certain groups or situations. By way of example and not limitation, these situations may include more limited devices, the user's face being obscured (e.g., the user is wearing a mask, the user has a beard, etc.), some emotions that may not be effectively captured by facial analysis (because these emotions rely more on body language), etc. In these scenarios, the XR system 160 may use audio features of the user's input to determine the emotion.

[0055] In certain embodiments, the high-level steps for detecting emotions expressed by an avatar can be summarized as follows. First, the XR system 160 can receive user input. Generally, assuming that there are no visual signals of the user's facial features, the user input can be a voice input. Optionally, the XR system 160 can also access supplementary signals, such as the user's facial features (e.g., obtained from a camera facing the user), posture (e.g., obtained from an electromyography (EMG) wristband or smart watch). In certain embodiments, the XR system 160 can receive a visual input corresponding to the voice input. The visual input can have multiple visual signals. Therefore, determining one or more emotions can also be based on one or more of the multiple visual signals. As an example and not limitation, the visual input can reflect the user's facial expression, which can also be used to improve emotion detection in addition to the voice input. In certain embodiments, the XR system 160 can receive a motion input corresponding to the voice input. The motion input can include multiple motion signals. Therefore, determining the one or more emotions can also be based on one or more of the multiple motion signals. By way of example and not limitation, motion input may indicate that a user's hands are moving quickly, which may be used along with voice input to determine that the user is excited.

[0056] Next, XR system 160 can use the emotion detection model to perform feature transformation based on the speech input and analyze the emotion based on these features. In certain embodiments, one or more machine learning models can include an acoustic model. Therefore, embodiments disclosed herein can have a technical advantage of improving the accuracy of emotion detection because XR system 160 can use only the acoustic model to directly detect emotions from the audio signal without using ASR models to avoid compound errors introduced by these ASR models when generating inaccurate transcriptions.

[0057] In certain embodiments, the acoustic model may use a spectrogram. A spectrogram is a visual representation of the frequency spectrum of the frequencies that make up the input waveform over time. The digital sample data may be divided into blocks in the time domain, which may often overlap. A Fourier transform may be applied to calculate the spectral amplitude of each block. Each block may then correspond to a vertical line in the image. In the case of a speech signal, the spectrogram may be further divided into Mel filterbanks, thereby producing a Mel spectrogram, which are logarithmically scaled frequency bands that are used to represent the range of human hearing.

[0058] In certain embodiments, XR system 160 may convert speech input into a mel-scale spectrogram via an acoustic model. Thus, determining one or more emotions may also be based on the mel-scale spectrogram. As an example and not limitation, an emotion detection model may use an acoustic model to obtain user input, convert the audio signal into a mel-scale spectrogram, and analyze only the audio signal features to derive emotions from the audio pattern analysis. In certain embodiments, XR system 160 may apply a convolutional neural network (CNN) followed by a multi-layer perceptron (MLP) on a two-dimensional (2D) spectrogram and a mel-scale spectrogram as input features.

[0059] In a particular embodiment, the acoustic model may use cepstral coefficients derived from a spectrogram. Cepstral coefficients are the amplitude of the spectrum of a discrete cosine transform performed at each frequency and considered as a list of logarithmic powers of an input signal. Cepstral coefficients of Mel-spaced frequency may be referred to as Mel-frequency cepstral coefficients (MFCC). Cepstral coefficients of linearly spaced frequencies are linear frequency cepstral coefficients (LFCC). MFCC is commonly used for speech recognition. MFCC has been used as an input to a shallow CNN before MLP or MLP for identifying emotions from speech. Using one or more acoustic models (e.g., an acoustic model including an acoustic model based on a Mel-scaled spectrogram of an audio signal) may be an effective solution to the technical challenge of effectively detecting emotions in speech, because these acoustic models can help to detect emotions using patterns of prosody, intonation, and stress in speech.

[0060] In an alternative embodiment, the emotion detection model may obtain speech input and transcribe the speech input into text (via an ASR module), and derive emotions based on the transcribed text using a language model within a natural-language understanding (NLU) module. In a particular embodiment, the XR system 160 may generate a transcription of the speech input. Thus, determining one or more emotions may also be based on an analysis of the transcription. By way of example and not limitation, the transcription may be “…this is unacceptable!…”, which may indicate that the user is angry. In a particular embodiment, the language model may utilize different types of context to improve detection performance, which are implemented as masked attention to previous utterances. By way of example and not limitation, these types of context may include global attention (all previous turns), speaker attention (turns belonging to the current speaker), and listener attention (turns belonging to the current listener). In a particular embodiment, such a conversational context may also be applied to the acoustic model.

[0061] In addition to these above-described methods that include intermediate steps, the XR system 160 may also have only an end-to-end model that obtains user input and outputs an emotion classification in one go without these intermediate steps. The XR system 160 may use an end-to-end model that goes directly from user input to emotions rendered on an avatar. Using an end-to-end emotion detection model may be an effective solution to the technical challenges of limiting computation, model size, and latency for implementation on a device and improving user experience, because the end-to-end model can obtain user input and output an emotion classification in one go, which may help reduce computation, model size, and latency. In a particular embodiment, the XR system 160 may merge the acoustic model and the language model to simultaneously learn from the prosody, intonation, and stress of the speech, as well as from the content of the speech. This higher accuracy model may be beneficial for asynchronous tasks that use emotions as context.

[0062] In certain embodiments, the acoustic model may be based on a deep learning model. The output layer of the deep learning model (i.e., the classification head) may include a multi-layer perceptron (MLP). The distinguishing factor may include an input representation to the classification head. The acoustic model may be based on the following variants.

[0063] In certain embodiments, the acoustic model may use the original audio signal. The original waveform may be used directly for classification without any explicit feature transformation. Representation learning may occur inside the model. As an example and not limitation, the popular M5 model for waveform classification may be used, which consists of 5 convolutional layers culminating in a fully connected layer for output. The convolutional layer closest to the input may have the widest kernel size. The kernel size may gradually decrease toward the output layer.

[0064] In certain embodiments, the acoustic model may use encodings from a pre-trained speech model. By way of example and not limitation, Wav2Vec2 is a pre-trained speech model. XR system 160 may use Wav2Vec2 as an encoder on the input waveform. Then, an MLP may be applied to the encoding for classification.

[0065] In certain embodiments, an avatar expression application may need to deploy an emotion detection model on the device to enable real-time rendering and also respect privacy considerations. Therefore, the XR system 160 can use those model architectures that are fast, small, and able to work with lightweight solutions to run deep learning models on the device (e.g., TensorFlow Lite interpreter): These model architectures may be fast, small, and able to work with lightweight solutions. All models described below (except the pre-trained Wav2Vec2 encoder) have a deployment size of less than 6MB.

[0066] The CNN models described below all have multiple layers of "convolutional blocks", each of which includes (in order) convolution, pooling, batch normalization (batch-norm), and linear rectifier (ReLU) activation, followed by several layers of "dense blocks", each of which includes a fully connected linear layer (with dropout) and ReLU activation. The optimal number of convolutional blocks and dense blocks is determined by hyperparameter tuning.

[0067] Table 2 depicts the performance comparison of multiple models and feature transformation variants on the Ryerson Audiovisual Database of Emotional Speech and Song (RAVDESS) dataset for 4 emotion categories (happy, sad, surprised, and neutral). Although the present disclosure describes a specific method for performing emotion detection using a specific model in a specific manner, the present disclosure contemplates any suitable method for performing emotion detection using any suitable model in any suitable manner.

[0068]

[0069]

[0070] Table 2. Performance comparison of various models and feature transformation variants.

[0071] In addition to audio signals, the XR system 160 may also use previous emotional states from previous inputs to further improve accuracy. In particular embodiments, previous emotional states may be stored in a conversation state. In particular embodiments, the XR system 160 may access previous emotional states from a conversation state associated with a first user. The previous emotional state may be determined based on one or more of the following: previous user input from the first user or previous user input from the second user. Thus, determining one or more emotions may also be based on previous emotional states. As an example and not limitation, the XR system 160 may infer a user's emotional state (e.g., being in a happy emotional state) by tracking the utterances of previous participants in a conversation, and storing these previous emotional states in a conversation state. The XR system 160 may also use the stored emotional states (e.g., happy) and current voice input to determine that the user may still be happy.

[0072] In certain embodiments, the output of the emotion detection model may include multiple emotion classifications. In addition, each of the one or more emotions may be associated with one or more of a range, an intensity, or a confidence score. In certain embodiments, the one or more emotions may be associated with one or more confidence scores, respectively. The XR system 160 may also rank the one or more emotions based on their confidence scores. Thus, one or more facial features are determined based on the highest ranked emotion. In other words, the XR system 160 may adopt the emotion classification with the highest confidence score and may render the avatar expression based on the emotion.

[0073] In a particular embodiment, the one or more emotions may be associated with one or more confidence scores, respectively. The XR system 160 may also rank the one or more emotions based on their confidence scores, and select two or more of the ranked emotions based on the ranking of the one or more emotions. Thus, determining the one or more facial features may be based on the two or more selected emotions. In a particular embodiment, the XR system 160 may generate a mixed emotion based on the two or more selected emotions. Thus, determining the one or more facial features may be based on a mixed emotion. In other words, the XR system 160 may obtain the top N emotions and render a mixed emotion (i.e., a mixture of these top N emotions) for the avatar. As an example and not limitation, the top two emotions may include disappointment and sadness. The XR system 160 may mix the two emotions and render the avatar accordingly.

[0074] In certain embodiments, XR system 160 may also map the emotions to facial features (expressions, actions, etc.) of the avatar. In certain embodiments, determining the one or more facial features may include identifying a plurality of facial landmarks of the first XR avatar. Determining the one or more facial features may also include mapping the one or more emotions to one or more predefined expression shapes for the face of the first XR avatar based on the plurality of facial landmarks. Thus, an animated avatar may be derived.

[0075] In certain embodiments, XR system 160 may also have a lip sync model that renders the avatar's lip movements based on emotions. XR system 160 may use the lip sync model to generate one or more lip movements based on the determined one or more emotions. Thus, the first XR avatar may also be rendered based on the one or more lip movements.

[0076] In certain embodiments, XR system 160 may detect an emotion outside of the coverage area. XR system 160 may make a prediction based on a library of predefined emotions. If XR system 160 cannot make a determination based on the library of predefined emotions, then XR system 160 may predict the emotion as neutral.

[0077] In certain embodiments, there may be animation constraints on the avatar's expression. Therefore, the prediction range may be redefined based on the animation constraints and expression dynamics. In certain embodiments, XR system 160 may determine conflicting emotions. In this case, XR system 160 may report the identified emotion along with a confidence score, which may be mitigated based on the specific use case.

[0078] In certain embodiments, XR system 160 may predict emotions within the shortest possible relevant range (e.g., a word, phrase, sentence, or even a paragraph). In certain embodiments, XR system 160 may limit the detection and expression of emotions based on social norms and topic sensitivity. In certain embodiments, XR system 160 may perform early detection of emotions because people may have emotions before they speak. XR system 160 may utilize contextual information for early detection. By way of example and not limitation, contextual information may indicate that a user is shaking his head slowly. XR system 160 may then perform early detection of disappointment.

[0079] In certain embodiments, XR system 160 can detect implicit emotions. Humans may express emotions without speaking based on the speech of another party. Therefore, XR system 160 can model the emotion detection model based on the emotions of all parties in the conversation. As an example and not limitation, the speech of the other party may be aggressive, for which XR system 160 can determine that the user may be unhappy or annoyed.

[0080] In certain embodiments, there may be latency in the rendering of avatar emotions. In certain embodiments, the XR system 160 may determine whether the latency for rendering the first XR avatar is greater than a threshold time. The XR system 160 may delay the transmission of one or more audio signals corresponding to the voice input to the second client system 130 with a delay time. In certain embodiments, the delay time may be determined based on the latency. As an example and not a limitation, users typically do not notice changes less than 18 milliseconds (threshold time). Therefore, if the latency is less than 18 milliseconds, it may not be a problem. However, if the latency is greater than 18 milliseconds, the XR system 160 may deliver audio to one or more other session participants with a slight delay while the emotion is being analyzed / rendered. In certain embodiments, the XR system 160 may use an end-to-end model that acquires user input and outputs an emotion classification in one go, which may help reduce latency.

[0081] In certain embodiments, providing avatar expressions may require accurate emotion detection after the speaker expresses the emotion, while limiting the delay of emotion detection. Longer audio clips may include more emotional information and may make detection more accurate. However, beyond a certain duration, other unrelated signals may drown out the emotional signal in the audio. In addition, longer audio clips may require longer inference times and may use relatively less recent audio. In addition, if the emotion is not immediately expressed on the avatar following the audio representing the emotion, the expression may appear to be out of sync with the actual audio.

[0082] To understand this tradeoff, embodiments disclosed herein conducted experiments to determine the best tradeoff for classification performance. Figure 4 An example trend of emotion detection accuracy with respect to the length of the audio segment used for inference is shown. Results can be generated based on an example model based on a CNN classifier on a 2D Mel-spectrogram of the audio input. Figure 4 As shown, although higher accuracy can be achieved between 1500 milliseconds (ms) and 2500 milliseconds, the resulting delay in expression rendering on the head-mounted device is unacceptable when we try to use these models for expressions. These current expression lag evaluations are qualitative based on our manual judgment and discussion. In this disclosure, we chose to detect emotions on 1 second audio clips and perform such inference every 150 milliseconds as the "emotion refresh rate" for rendering expressions.

[0083] In a particular embodiment, the XR system 160 may implement the emotion detection model as a zero-shot classifier using textual entailment based on a language model pre-trained on a public dataset. As an example and not limitation, the language model may be a bidirectional autoregressive transformer (BART), a bidirectional encoder representation based on a transformer (BERT), a robustly optimized BERT (RoBERT), a generative pre-trained transformer (GPT), or a large language model meta-platform AI (LLaMA). Emotion detection may be implemented using a language (text-based) model. Emotion detection may also be implemented using an acoustic model only, which is faster and more efficient in terms of computing resources. An acoustic model only may not require analysis of the text output by the ASR module, but may only analyze the acoustic features of the input to determine the speaker's emotions. Comprehensive experiments were conducted on multiple publicly available datasets for quantitative analysis. Table 3 shows example predictions for certain utterances without any task-specific fine-tuning. The experimental results are as expected. The emotion detection model can also be extended beyond zero-shot classification to incorporate longer contexts using the following XLMR classifier and state-of-the-art methods for multi-party conversations: The XLMR classifier is trained for this specific task.

[0084]

[0085]

[0086] Table 3. Example predictions for some utterances without any task-specific fine-tuning.

[0087] In addition to avatar expressions, the emotion detection model can also be applied to the following use cases. One use case can be augmented avatar calls. Augmented avatar calls are an alternative when holographic calls are not possible due to the lack of visual effects (for example, when the user turns off their camera because of discomfort). Although lip movements based on lip shape can be achieved based on audio, realistic expressions in avatar calls may only be achieved by understanding emotions from the conversation. Another use case can be ultra-low conflict emotional reactions. The XR system 160 can improve the emotional reaction features to accurately suggest quick emotional confirmations to incoming messages.

[0088] Another use case may be smart communications, such as smart writing, smart replies, message summaries, notification summaries, automatic collection, and smart keyboards. Understanding the underlying emotions in the conversation can lead to better natural language generation, emoji recommendations, and sticker / content recommendations. In a particular embodiment, the XR system 160 may generate a recommendation for communication content for the first user based on the determined one or more emotions. The XR system 160 may also send an instruction to the first client system 130 to present the recommendation for the communication content.

[0089] Another use case could be rich responses from non-playing characters (NPCs) in games. XR system 160 can provide developers with a software development kit (SDK) so that these developers can design their games to provide customized responses through NPCs in real time based on the user's emotions.

[0090] Another use case may be for the XR system 160 itself. The assistant system 140 may act as a contextual intelligent assistant that gives people "superpowers" in a digitally augmented world. In conversational artificial intelligence (AI) tasks involving natural language understanding, natural language generation, questions and answers, proactive relevance, and summarization, emotions may be important contextual personalization features. In particular embodiments, the first client system 130 may send the determined one or more emotions to the XR system 160 executing on the first client system 130. The client system 130 may also receive from the XR system 160 the results of the execution of one or more tasks determined based on the one or more emotions. Therefore, the embodiments disclosed herein may have the technical advantage of an improved user experience because the XR system 160 can use the emotions detected in the user's voice to adjust the assistant service to reflect such emotions.

[0091] Figure 5 An example user conversation represented by avatars is shown. User 510 may be wearing VR headset 130a. User 520 may be wearing VR headset 130b. The two users may be having a conversation in a virtual world. As can be seen, each of the two users can see an avatar representing the other user. More specifically, user 510 can see avatar 525 representing user 520. User 520 can see avatar 515 representing user 510. XR system 160 can detect the emotions of each user based on the voice input of each of these users. XR system 160 can also render the expression of the avatar accordingly. As an example and not limitation, XR system 160 may detect that user 510 is unhappy. Therefore, XR system 160 can cause avatar 515 to frown. As another example and not limitation, XR system 160 may detect that user 520 is happy. Therefore, XR system 160 can cause avatar 525 to smile.

[0092] Figure 6An example method 600 for providing real-time avatar expression is shown. The method may begin at step 605, where the XR system 160 may receive voice input from a first client system 130 associated with a first user, the voice input having a plurality of acoustic features, wherein the first user is having a conversation with a second user in an extended reality (XR) environment including a first XR avatar representing the first user. At step 610, the XR system 160 may receive visual input corresponding to the voice input, wherein the visual input has a plurality of visual signals. At step 615, the XR system 160 may receive motion input corresponding to the voice input, wherein the motion input includes a plurality of motion signals. At step 620, the XR system 160 may generate a transcription for the voice input. At step 625, the XR system 160 may access a previous emotional state from a conversation state associated with the first user, wherein the previous emotional state is determined based on one or more of: a previous user input from the first user or a previous user input from the second user. At step 630, the XR system 160 may determine one or more emotions associated with the speech input based on one or more of the following: acoustic features, visual signals, motion signals, analysis of transcription, or previous emotional states through one or more machine learning models, wherein the one or more machine learning models include an acoustic model, and wherein each of the one or more emotions is associated with one or more of a range, an intensity, or a confidence score. At step 635, the XR system 160 may determine one or more facial features of the first XR avatar based on the one or more emotions, wherein determining the one or more facial features includes: identifying a plurality of facial landmarks of the first XR avatar, and mapping the one or more emotions to one or more predefined expression shapes of the face of the first XR avatar based on the plurality of facial landmarks. At step 640, the XR system 160 may send instructions to a second client system 130 associated with a second user to render a first XR avatar representing the first user, wherein the first XR avatar is rendered using the determined one or more facial features. At step 645, the XR system 160 may generate a recommendation for communication content for the first user based on the determined one or more emotions. At step 650, XR system 160 may send instructions to first client system 130 for presenting a recommendation for the communication content. Where appropriate, certain embodiments may repeat Figure 6 One or more steps in a method. Figure 6 Specific steps in the method are described and shown as occurring in a specific order, but the present disclosure contemplates Figure 6 Any suitable steps in the method may occur in any suitable order. In addition, although the present disclosure describes and illustrates methods for providing real-time avatar expression, including Figure 6The present disclosure contemplates any suitable method for providing real-time avatar expression including any suitable steps, which, where appropriate, may include: Figure 6 All, some, or none of the steps in the method Figure 6 Furthermore, although the present disclosure describes and illustrates the steps in the method of Figure 6 specific components, devices or systems for specific steps in the method, but the present disclosure contemplates performing Figure 6 Any suitable combination of any suitable components, devices, or systems for any suitable steps in the method.

[0093] Assistant System

[0094] Figure 7 An example architecture 700 of an assistant system is shown. In certain embodiments, the assistant system 140 can help a user obtain information or services. The assistant system 140 can enable a user to interact with the assistant system 140 through user input in various modalities (e.g., audio, voice, text, vision, image, video, gesture, motion, activity, position, orientation) in a stateful and multi-turn conversation to receive help from the assistant system 140. As an example and not limitation, the user input can include audio input based on user voice (e.g., verbal commands), which can be processed by a system audio application programming interface (API) on the client system 130. The system audio API can perform the following techniques to generate audio data that can be easily processed by the assistant system 140: these techniques include echo cancellation, noise removal, beamforming, self-user voice activation, speaker recognition, voice activity detection (VAD), and / or any other suitable acoustic technology. In particular embodiments, assistant system 140 may support unimodal input (e.g., voice input only), multimodal input (e.g., voice input and text input), hybrid / multimodal input, or any combination thereof. In particular embodiments, user input may be user-generated input that is sent to assistant system 140 in a single turn. User input provided by a user may be associated with a particular assistant-related task and may include, for example, a user request (e.g., a verbal request for information or to perform an action), a user interaction with an assistant application 136 associated with assistant system 140 (e.g., selecting a UI element via touch or gesture), or any other type of suitable user input that may be detected and understood by assistant system 140 (e.g., user movement detected by the user's client system 130).

[0095] In a particular embodiment, the assistant system 140 may create and store such a user profile: the user profile includes both personal information and contextual information associated with the user. In a particular embodiment, the assistant system 140 may use natural language understanding (NLU) technology to analyze user input. The analysis may be based at least in part on the user's user profile for a more personalized and context-aware understanding. The assistant system 140 may parse entities associated with the user input based on the analysis. In a particular embodiment, the assistant system 140 may interact with different agents to obtain information or services associated with the parsed entity. The assistant system 140 may generate responses about information or services for the user by using natural language generation (NLG). Through interaction with the user, the assistant system 140 may use dialogue management technology to manage and forward conversation flows with the user. In a particular embodiment, the assistant system 140 may further help the user to effectively and efficiently understand the information obtained by summarizing the information. The assistant system 140 may also help users to participate more in online social networks by providing tools that help users interact with online social networks (e.g., create posts, comments, messages). The assistant system 140 may also help the user manage different tasks, such as keeping track of events. In certain embodiments, the assistant system 140 may proactively perform pre-authorized tasks related to the user's interests and preferences based on the user profile at a time relevant to the user, without user input. In certain embodiments, the assistant system 140 may check privacy settings to ensure that access to the user's profile or other user information and performance of different tasks are allowed based on the user's privacy settings. More information about helping users based on privacy settings can be found in the following documents: U.S. patent application Ser. No. 16 / 182,542, filed on Nov. 6, 2018, which is incorporated by reference.

[0096] In certain embodiments, assistant system 140 can assist users through an architecture built on client-side processes and server-side processes that can operate in various modes of operation. Figure 7, the client-side process is shown above the dashed line 702, while the server-side process is shown below the dashed line 702. The first operating mode (i.e., on-device mode) can be a workflow in which the assistant system 140 processes user input and provides assistance to the user by primarily or exclusively executing the client-side process locally on the client system 130. For example, if the client system 130 is not connected to the network 110 (i.e., when the client system 130 is offline), the assistant system 140 can process user input in the first operating mode using only the client-side process. The second operating mode (i.e., cloud mode) can be a workflow in which the assistant system 140 processes user input and provides assistance to the user by primarily or exclusively executing the server-side process on one or more remote servers (e.g., servers associated with the assistant system 140). As Figure 7 As shown, the third operation mode (i.e., hybrid mode) can be a parallel workflow in which the assistant system 140 processes user input and provides assistance to the user by executing client-side processes locally on the client system 130 in combination with executing server-side processes on one or more remote servers (e.g., servers associated with the assistant system 140). For example, both the client system 130 and the server associated with the assistant system 140 can perform automatic speech recognition (ASR) processes and natural language understanding (NLU) processes, but the client system 130 can delegate the dialog, agent, and natural language generation (NLG) processes to be performed by the server associated with the assistant system 140.

[0097] In certain embodiments, the selection of an operating mode may be based at least in part on the device state, the task associated with the user input, and / or one or more additional factors. By way of example and not limitation, as described above, one factor may be the network connection state of client system 130. For example, if client system 130 is not connected to network 110 (i.e., when client system 130 is offline), assistant system 140 may process the user input in a first operating mode (i.e., on-device mode). By way of another example and not limitation, another factor may be based on a measurement of the available battery power (i.e., battery state) of client system 130. For example, if client system 130 needs to conserve battery power (e.g., when client system 130 has a minimum available battery power or the user has indicated a desire to conserve the battery power of client system 130), assistant system 140 may process the user input in a second operating mode (i.e., cloud mode) or a third operating mode (i.e., hybrid mode) to perform less power-consuming operations on client system 130. By way of yet another example and not limitation, another factor may be one or more privacy constraints (e.g., specified privacy settings, applicable privacy policies). For example, if one or more privacy constraints restrict or prevent certain data from being sent to a remote server (e.g., a server associated with assistant system 140), assistant system 140 may process user input in a first operating mode (i.e., on-device mode) to protect user privacy. As yet another example and not limitation, another factor may be unsynchronized context data between client system 130 and a remote server (e.g., a server associated with assistant system 140). For example, it may be determined that client system 130 and a server associated with assistant system 140 have inconsistent, missing, and / or inconsistent context data, and assistant system 140 may process user input in a third operating mode (i.e., hybrid mode) to reduce the likelihood of inadequate analysis associated with the user input. As yet another example and not limitation, another factor may be a measurement of the latency of a connection between client system 130 and a remote server (e.g., a server associated with assistant system 140). For example, if the task associated with the user input may significantly benefit from and / or require timely or immediate execution (e.g., a photo collection task), assistant system 140 may process the user input in a first operating mode (i.e., on-device mode) to ensure timely execution of the task. As yet another example and not limitation, another factor may be whether, for a feature related to the task associated with the user input, the feature is supported only by a remote server (e.g., a server associated with assistant system 140).For example, if the relevant feature requires advanced technical functions (e.g., high-performance processing capabilities, fast update cycles), and the advanced technical functions are only supported by a server associated with the assistant system 140 and not by the client system 130 at the time of user input, the assistant system 140 can process the user input in a second operating mode (i.e., cloud mode) or a third operating mode (i.e., hybrid mode) in order to benefit from the relevant feature.

[0098] In certain embodiments, the on-device coordinator 706 on the client system 130 can coordinate the receipt of user input and can determine, at one or more decision points in the example workflow, which of the above-described operating modes should be used to process or continue processing the user input. As described above, the selection of an operating mode can be based at least in part on the device state, the task associated with the user input, and / or one or more additional factors. By way of example and not limitation, reference is made to Figure 7 In the workflow architecture shown in , after receiving user input from the user, the on-device coordinator 706 can determine at decision point (D0) 705 whether to start processing the user input in a first operating mode (i.e., on-device mode), a second operating mode (i.e., cloud mode), or a third operating mode (i.e., hybrid mode). For example, if the client system 130 is not connected to the network 110 (i.e., when the client system 130 is offline), if one or more privacy constraints explicitly require on-device processing (e.g., adding or removing another person to a private call between multiple users), or if the user input is associated with a task that does not require or benefit from a server-side process (e.g., setting a reminder or calling another user), then at decision point (D0) 705, the on-device coordinator 706 can select the first operating mode (i.e., on-device mode). As another example, if the client system 130 needs to conserve battery power (e.g., when the client system 130 has a minimum available battery power or the user has expressed a desire to conserve battery power of the client system 130) or needs to limit additional utilization of computing resources (e.g., when other processes operating on the client system 130 (e.g., SMS messaging applications) require high CPU utilization), then at decision point (D0) 705, the on-device coordinator 706 may select the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode).

[0099] In a specific embodiment, if Figure 7As shown, if the on-device coordinator 706 determines at decision point (D0) 705 that the user input should be processed using the first operating mode (i.e., on-device mode) or the third operating mode (i.e., hybrid mode), the client process can continue. By way of example and not limitation, if the user input includes voice data, the voice data can be received at a local automatic speech recognition (ASR) module 708a on the client system 130. The ASR module 708a can allow the user to dictate and transcribe the speech into written text, synthesize a document into an audio stream, or issue a command that is recognized by the system accordingly.

[0100] In certain embodiments, the output of the ASR module 708a can be sent to a local natural language understanding (NLU) module 710a. The NLU module 710a can perform named entity resolution (NER), or named entity resolution can be performed by an entity resolution module 712a, as described below. In certain embodiments, one or more of an intent, a slot, or a domain can be the output of the NLU module 710a.

[0101] In a particular embodiment, user input may include non-voice data, which may be received at a local context engine 720a. As an example and not limitation, non-voice data may include location, vision, touch, posture, world update, social update, context information, information related to people, activity data, and / or any other suitable type of non-voice data. Non-voice data may also include sensory data received by a sensor (e.g., microphone, camera) of the client system 130, which may be accessed in accordance with privacy constraints and may be further analyzed by computer vision techniques. In a particular embodiment, computer vision techniques may include object detection, scene recognition, hand tracking, eye tracking, and / or any other suitable computer vision techniques. In a particular embodiment, non-voice data may be subject to geometric constructions, which may include using any suitable type of data collected by the client system 130 to construct objects around the user. As an example and not limitation, the user may be wearing AR glasses, and geometric constructions may be used to determine the spatial position of surfaces and items (e.g., floors, walls, user's hands). In certain embodiments, the non-voice data may be inertial data collected by AR glasses or VR head-mounted devices, and the inertial data may be data associated with linear and angular motion (e.g., measurements associated with user body movement). In certain embodiments, the context engine 720a may determine various types of events and contexts based on the non-voice data.

[0102] In a particular embodiment, the output of the NLU module 710a and / or the context engine 720a may be sent to an entity resolution module 712a. The entity resolution module 712a may resolve entities associated with one or more slots output by the NLU module 710a. In a particular embodiment, each resolved entity may be associated with one or more entity identifiers. As an example and not limitation, an identifier may include a unique user identifier (ID) corresponding to a particular user (e.g., a unique user name or user ID number of the extended reality system 160). In a particular embodiment, each resolved entity may also be associated with a confidence score. More information on resolving entities may be found in the following documents: U.S. Patent No. 10,803,050 filed on July 27, 2018, and U.S. Patent Application No. 16 / 048072 filed on July 27, 2018, each of which is incorporated by reference.

[0103] In certain embodiments, at decision point (D0) 705, the on-device coordinator 706 may determine that the user input should be processed in the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode). In these operating modes, the user input may be processed by certain server-side modules in a manner similar to the client-side process described above.

[0104] In certain embodiments, if the user input includes voice data, the user input voice data may be received at a remote automatic speech recognition (ASR) module 708b on a remote server, such as a server associated with assistant system 140. ASR module 708b may allow the user to dictate and transcribe the speech into written text, synthesize a document into an audio stream, or issue commands that are recognized by the system.

[0105] In certain embodiments, the output of the ASR module 708b may be sent to a remote natural language understanding (NLU) module 710b. In certain embodiments, the NLU module 710b may perform named entity resolution (NER), or named entity resolution may be performed by an entity resolution module 712b of a dialog manager 716b as described below. In certain embodiments, one or more of an intent, a slot, or a domain may be the output of the NLU module 710b.

[0106] In certain embodiments, the user input may include non-speech data, which may be received at a remote context engine 720b. In certain embodiments, the remote context engine 720b may determine various types of events and contexts based on the non-speech data. In certain embodiments, the output of the NLU module 710b and / or the context engine 720b may be sent to a remote dialog manager 716b.

[0107] In certain embodiments, as described above, the on-device coordinator 706 on the client system 130 can coordinate receiving the user input and can determine, at one or more decision points in the example workflow, which of the above-described operating modes should be used to process or continue processing the user input. As further described above, the selection of the operating mode can be based at least in part on the device state, the task associated with the user input, and / or one or more additional factors. By way of example and not limitation, continue to refer to Figure 7 In the workflow architecture shown in , after the entity resolution module 712a generates an output or an empty output, the device coordinator 706 can determine at decision point (D1) 715 whether to continue processing the user input in the first operating mode (i.e., the device mode), in the second operating mode (i.e., the cloud mode), or in the third operating mode (i.e., the hybrid mode). For example, if the identified intent is associated with a latency-sensitive processing task (e.g., taking a photo, pausing a timer), then at decision point (D1) 715, the device coordinator 706 can select the first operating mode (i.e., the device mode). As another example and not limitation, if the on-device processing on the client system 130 does not support messaging tasks, the device coordinator 706 can select the third operating mode (i.e., the hybrid mode) to process user input associated with messaging requests. As yet another example, if the task being processed requires access to a social graph, knowledge graph, or concept graph that is not stored on the client system 130, then at decision point (D1) 715, the on-device coordinator 706 may select the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode). Alternatively, if a sufficient version of the information graph (e.g., a small version and / or a bootstrapped version of the knowledge graph) exists on the client system 130 (the information graph includes the necessary information for the task), then the on-device coordinator 706 may alternatively select the first operating mode (i.e., on-device mode).

[0108] In a specific embodiment, if Figure 7As shown, if the on-device coordinator 706 determines at decision point (D1) 715 that the processing should continue using the first operating mode (i.e., on-device mode) or the third operating mode (i.e., hybrid mode), the client process can continue. As an example and not limitation, the output from the entity resolution module 712a can be sent to the on-device dialogue manager 716a. In a particular embodiment, the on-device dialogue manager 716a may include a dialogue state tracker 718a and an action selector 722a. The on-device dialogue manager 716a may have complex dialogue logic and product-related business logic to manage the dialogue state and conversation flow between the user and the assistant system 140. The on-device dialogue manager 716a may include full functionality for end-to-end integration and multi-round support (e.g., confirmation, disambiguation). The on-device dialogue manager 716a may also be lightweight in terms of computing constraints and resources (including memory, computing (CPU), and binary size constraints). The on-device dialogue manager 716a may also be extensible to improve the developer experience. In particular embodiments, on-device dialog manager 716a can benefit assistant system 140 by, for example, providing offline support to reduce network connectivity issues (e.g., unstable or unavailable network connections), using client-side processes to prevent privacy-sensitive information from being transmitted outside of client system 130, and providing a stable user experience in high-latency sensitive scenarios.

[0109] In certain embodiments, the on-device dialogue manager 716a may also perform false trigger mitigation. Implementations of false trigger mitigation may detect and prevent false triggers (e.g., accidental wake-up words) from user input that would also invoke the assistant system 140, and may further prevent the assistant system 140 from generating data records based on false triggers that may be inaccurate and / or comply with privacy constraints. As an example and not limitation, if a user is in a voice call, the user's conversation during the voice call may be considered private, and false trigger mitigation may limit the detection of the wake-up word to audio user input received locally by the user's client system 130. In certain embodiments, the on-device dialogue manager 716a may implement false trigger mitigation based on a nonsense detector. If the nonsense detector determines with high confidence that the wake-up word received from the user is logically and / or contextually unreasonable at the time point when the wake-up word is received from the user, the on-device dialogue manager 716a may determine that the user did not intend to invoke the assistant system 140.

[0110] In certain embodiments, due to the limited computing power of the client system 130, the on-device dialog manager 716a may perform on-device learning based on a learning algorithm specifically customized for the client system 130. As an example and not a limitation, the on-device dialog manager 716a may implement a federated learning technique. Federated learning is a specific class of distributed machine learning techniques that can train machine learning models using decentralized data stored on terminal devices (e.g., mobile phones). In certain embodiments, the on-device dialog manager 716a may use a joint user representation learning model to extend existing neural network personalization techniques to implement federated learning of the on-device dialog manager 716a. Joint user representation learning can personalize the federated learning model by learning task-specific user representations (i.e., embeddings) and / or by personalizing model weights. Joint user representation learning is simple, scalable, privacy-preserving, and resource-efficient. Joint user representation learning can separate model parameters into joint parameters and privacy parameters. Privacy parameters (e.g., private user embeddings) can be trained locally on the client system 130 instead of being transmitted to a remote server (e.g., a server associated with the assistant system 140) or evenly distributed by the remote server. In contrast, the federated parameters can be trained remotely on the server. In certain embodiments, the on-device dialog manager 716a can use an active federated learning model that can transfer a global model trained on a remote server to the client system 130 and compute gradients locally on the client system 130. Active federated learning can enable the on-device dialog manager 716a to minimize the transmission costs associated with downloading the model and uploading the gradients. For active federated learning, in each round, the client system 130 can be selected in a semi-random manner based at least in part on a probability conditioned on the current model and the data on the client system 130 to optimize the training efficiency of the federated learning model.

[0111] In certain embodiments, as the user interacts with the world and assistant system 140 interacts with the user, conversation state tracker 718a can track state changes over time. By way of example and not limitation, conversation state tracker 718a can track, for example, what the user is talking about, who the user is with, where the user is, what task is currently being performed, and where the user is looking, subject to applicable privacy policies.

[0112] In certain embodiments, at decision point (D1) 715, the on-device coordinator 706 may determine to forward the user input to a server in a second operating mode (i.e., cloud mode) or a third operating mode (i.e., hybrid mode). As an example and not limitation, if a particular function or process (e.g., messaging) is not supported on the client system 130, the on-device coordinator 706 may determine to use the third operating mode (i.e., hybrid mode) at decision point (D1) 715. In certain embodiments, the on-device coordinator 706 may cause the output from the NLU module 710a, the context engine 720a, and the entity resolution module 712a to be forwarded to the entity resolution module 712b of the remote dialog manager 716b through the dialog manager proxy 724 to continue the process. The dialog manager proxy 724 may be a communication channel for exchanging information / events between the client system 130 and the server. In certain embodiments, the dialog manager 716b may also include a remote arbitrator 726b, a remote dialog state tracker 718b, and a remote action selector 722b. In certain embodiments, at decision point (D0) 705, assistant system 140 may have already begun processing user input in a second mode of operation (i.e., cloud mode), and at decision point (D1) 715, on-device coordinator 706 may determine to continue processing the user input based on the second mode of operation (i.e., cloud mode). Thus, outputs from NLU module 710b and context engine 720b may be received at remote entity resolution module 712b. Remote entity resolution module 712b may have similar functionality to local entity resolution module 712a, which may include resolving entities associated with slots. In certain embodiments, entity resolution module 712b may access one or more of a social graph, a knowledge graph, or a concept graph when resolving entities. Outputs from entity resolution module 712b may be received at arbitrator 726b.

[0113] In certain embodiments, the remote arbitrator 726b may be responsible for selecting between client-side upstream results and server-side upstream results (e.g., results from NLU modules 710a / 710b, results from entity resolution modules 712a / 712b, and results from context engines 720a / 720b). The arbitrator 726b may send the selected upstream results to the remote dialog state tracker 718b. In certain embodiments, similar to the local dialog state tracker 718a, the remote dialog state tracker 718b may convert the upstream results into candidate tasks using task specifications and resolve arguments using entity resolution.

[0114] In a particular embodiment, at decision point (D2) 725, the on-device coordinator 706 may determine whether to continue to process the user input based on the first operating mode (i.e., the on-device mode) or to forward the user input to the server in a third operating mode (i.e., the hybrid mode). The decision may depend on, for example, whether the client process can successfully resolve the task and slot, whether there is an effective task policy with specific feature support, and / or context differences between the client process and the server-side process. In a particular embodiment, the decision made at decision point (D2) 725 may be for a multi-round scenario. In a particular embodiment, there may be at least two possible scenarios. In the first scenario, the assistant system 140 may have begun to process the user input in the first operating mode (i.e., the on-device mode) using the client-side dialog state. If the assistant system 140 decides at some point to switch to having the remote server process the user input, the assistant system 140 may create a programmed / predefined task with the current task state and forward the programmed / predefined task to the remote server. For subsequent rounds, the assistant system 140 may continue to process in the third operating mode (i.e., the hybrid mode) using the server-side dialog state. In another scenario, assistant system 140 may have already begun processing user input in the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode), and assistant system 140 may rely substantially on the server-side dialog state for all subsequent turns. If on-device coordinator 706 determines to continue processing user input based on the first operating mode (i.e., on-device mode), output from dialog state tracker 718a may be received at action selector 722a.

[0115] In certain embodiments, at decision point (D2) 725, the on-device coordinator 706 may determine to forward the user input to the remote server and continue to process the user input in the second operating mode (i.e., cloud mode) or the third operating mode (i.e., hybrid mode). The assistant system 140 may create a programmed / predefined task using the current task state and forward the programmed / predefined task to the server, which may be received at the action selector 722b. In certain embodiments, the assistant system 140 may have already begun processing the user input in the second operating mode (i.e., cloud mode), and the on-device coordinator 706 may determine at decision point (D2) 725 to continue processing the user input in the second operating mode (i.e., cloud mode). Thus, the output from the dialog state tracker 718b may be received at the action selector 722b.

[0116] In a particular embodiment, the action selector 722a / 722b can perform interaction management. The action selector 722a / 722b can determine and trigger a set of general executable actions. These actions can be performed on the client system 130 or at a remote server. As an example and not a limitation, these actions can include providing information or suggestions to the user. In a particular embodiment, these actions can interact with the agent 728a / 728b, the user, and / or the assistant system 140 itself. These actions can include such actions: the action includes one or more of slot requests, confirmations, disambiguation or agent execution. These actions can be independent of the bottom layer implementation of the action selector 722a / 722b. For more complex scenarios (such as multi-round tasks or tasks with complex business logic), the local action selector 722a can call one or more local agents 728a, and the remote action selector 722b can call one or more remote agents 728b to perform these actions. Agent 728a / 728b can be called by task ID, and any action can be routed to the correct agent 728a / 728b using the task ID. In a specific embodiment, agent 728a / 728b can be configured to act as a broker between multiple content providers of a domain. A content provider can be an entity responsible for performing an action associated with an intent or completing a task associated with the intent. In a specific embodiment, agent 728a / 728b can provide several functions for assistant system 140, including, for example, local template generation, task-specific business logic, and querying external APIs. Agent 728a / 728b can use the context from dialog state tracker 718a / 718b when performing the action of a task, and can also update dialog state tracker 718a / 718b. In a specific embodiment, agent 728a / 728b can also generate a partial payload from a dialog action.

[0117] In a particular embodiment, the local agent 728a may have different implementations to be edited / registered for different platforms (e.g., smart glasses and VR head-mounted devices). In a particular embodiment, multiple device-specific implementations (e.g., real-time calls to a client system 130 or a messaging application on the client system 130) may be handled internally by a single agent 728a. Alternatively, multiple device-specific implementations may be handled by multiple agents 728a associated with multiple domains. As an example and not limitation, calling an agent 728a on smart glasses may be implemented in a different manner than calling an agent 728a on a smartphone. Different platforms may also use different numbers of agents 728a. Agent 728a may also be cross-platform (i.e., different operating systems on the client system 130). In addition, agent 728a may have a minimized startup time or binary size impact. The local agent 728a may be suitable for specific use cases. As an example and not limitation, one use case may be an emergency call to the client system 130. As another example and not limitation, another use case may be responding to user input without a network connection. As yet another example and not limitation, another use case may be that certain domains / tasks may be privacy sensitive and sending user input to a remote server may be prohibited.

[0118] In a particular embodiment, the local action selector 722a can call the local delivery system 730a to perform the action, and the remote action selector 722b can call the remote delivery system 730b to perform the action. Once the delivery system 730a / 730b receives the trigger signal from the conversation state tracker 718a / 718b, it can deliver the predefined event by performing the corresponding action. The delivery system 730a / 730b can ensure that the event is delivered to the host through a real-time connection. As an example and not limitation, the delivery system 730a / 730b can broadcast to all online devices belonging to a user. As another example and not limitation, the delivery system 730a / 730b can deliver the event to a target-specific device. The delivery system 730a / 730b can also use the latest device context to render the payload.

[0119] In a particular embodiment, the on-device dialog manager 716a may also include a separate local action execution module, and the remote dialog manager 716b may also include a separate remote action execution module. The local action execution module and the remote action execution module may have similar functions. In a particular embodiment, the action execution module may call an agent 728a / 728b to perform a task. The action execution module may also execute a set of general executable actions determined by the action selector 722a / 722b. The set of executable actions may interact with the agent 728a / 728b, the user, and the assistant system 140 itself through the delivery system 730a / 730b.

[0120] In a particular embodiment, if the user input is processed using the first operating mode (i.e., on-device mode), the results from the agent 728a and / or the delivery system 730a may be returned to the device dialog manager 716a. Then, the on-device dialog manager 716a may instruct the local arbitrator 726a to generate a final response based on these results. The arbitrator 726a may aggregate these results and evaluate them. As an example and not limitation, the arbitrator 726a may sort and select the best result to respond to the user input. If the user request is processed in the second operating mode (i.e., cloud mode), the results from the agent 728b and / or the delivery system 730b may be returned to the remote dialog manager 716b. The remote dialog manager 716b may instruct the arbitrator 726a to generate a final response based on these results through the dialog manager agent 724. Similarly, the arbitrator 726a may analyze these results and select the best result to provide to the user. If the user input is processed based on the third mode of operation (i.e., hybrid mode), the client-side results and server-side results (e.g., from the agent 728a / 728b and / or the delivery system 730a / 730b) can be provided to the arbitrator 726a through the on-device dialog manager 716a and the remote dialog manager 716b, respectively. The arbitrator 726a can then select between the client-side results and the server-side results to determine the final result to be presented to the user. In a particular embodiment, the logic for deciding between these results can depend on the particular use case.

[0121] In a particular embodiment, the local arbitrator 726a may generate a response based on the final result and send the response to the rendering output module 732. The rendering output module 732 may determine how to render the output in a manner suitable for the client system 130. As an example and not limitation, for a VR head-mounted device or AR smart glasses, the rendering output module 732 may determine to render the output using a visual-based modality (e.g., an image or video clip), which can be displayed by the VR head-mounted device or AR smart glasses. As another example, the response may be rendered as audio signals, which may be played by the user through the VR head-mounted device or AR smart glasses. As yet another example, the response may be rendered as augmented reality data for enhancing the user experience.

[0122] In certain embodiments, in addition to determining the operating mode for processing user input, the on-device coordinator 706 may also determine whether to process the user input on the rendering device 137, process the user input on the companion device 138, or process the user request on a remote server. The rendering device 137 and / or the companion device 138 may each use an assistant stack to process the user input in a manner similar to that disclosed above. As an example and not limitation, the on-device coordinator 706 may determine that a portion of the processing should be completed on the rendering device 137, a portion of the processing should be completed on the companion device 138, and the remainder of the processing should be completed on the remote server.

[0123] In certain embodiments, assistant system 140 may have various capabilities including audio cognition, visual cognition, signal intelligence, reasoning, and memory. In certain embodiments, audio cognition capabilities may enable assistant system 140 to, for example, understand user input associated with various domains of different languages, understand and summarize conversations, perform on-device audio cognition for complex commands, recognize users by voice, extract topics from conversations and auto-tagged portions of the conversation, enable audio interaction without a wake word, filter and amplify user voice from ambient noise and conversations, and / or understand which client system 130 a user is talking to when multiple client systems 130 are nearby.

[0124] In particular embodiments, visual recognition capabilities may enable assistant system 140 to: for example, identify interesting objects in the world through a combination of existing machine learning models and one-shot learning, identify interesting moments and automatically capture them, achieve semantic understanding across multiple visual frames spanning different time segments, provide platform support for additional place recognition or object recognition capabilities, recognize a full range of settings and micro-locations including personalized locations, recognize complex activities, recognize complex gestures for controlling client system 130, process images / videos from egocentric cameras (e.g., with motion, capture angles, resolution) to achieve similar accuracy and speed levels associated with images with lower resolution, perform one-shot registration and recognition of places and objects, and / or perform visual recognition on client system 130.

[0125] In certain embodiments, the assistant system 140 may utilize computer vision techniques to achieve visual cognition. In addition to computer vision techniques, the assistant system 140 may also explore options that can supplement these techniques to expand the recognition of objects. In certain embodiments, the assistant system 140 may use supplementary signals, such as optical character recognition (OCR) of object labels, GPS signals for location recognition, and / or signals from the user's client system 130 to identify the user. In certain embodiments, the assistant system 140 may perform general scene recognition (e.g., home space, work space, public space) to set the user's context and narrow the computer vision search space to identify possible objects or people. In certain embodiments, the assistant system 140 may guide users to train the assistant system 140. For example, crowdsourcing may be used to allow users to tag objects and help the assistant system 140 recognize more objects over time. As another example, when using the assistant system 140, users may register their personal objects as part of the initial settings. The assistant system 140 may also allow users to provide positive / negative signals for the objects they interact with to train and improve the personalized models of these users.

[0126] In certain embodiments, signals intelligence capabilities may enable assistant system 140 to, for example, determine user location, understand date / time, determine home location, understand the user's schedule and future destinations, integrate richer voice understanding to identify settings / context by voice alone, and / or build signals intelligence models at runtime that can be personalized for a user's individual routines.

[0127] In certain embodiments, reasoning capabilities may enable assistant system 140 to, for example, extract previous conversation threads at any point in the future, synthesize all signals to understand micro-context and personalized context, learn interaction patterns and preferences based on a user's historical behavior and accurately suggest interactions that they may value, generate highly predictive proactive suggestions based on micro-context understanding, understand what content a user may want to watch at what time of day, and / or understand changes in context and how those changes may affect what a user wants.

[0128] In certain embodiments, memory capabilities may enable assistant system 140 to, for example, remember social connections that a user has previously visited or interacted with, write to memory and query memory at will (i.e., open dictation and automatic tagging), extract richer preferences based on previous interactions and long-term learning, remember the user's life history, extract rich information from egocentric data streams and automatic catalogs, and / or write to memory in a structured form to form rich short-term memory, episodic memory, and long-term memory.

[0129] Figure 8 An example flow chart 800 of an assistant system 140 is shown. In certain embodiments, the assistant service module 805 may access a request manager 810 upon receiving user input. In certain embodiments, the request manager 810 may include a context extractor 812 and a conversational understanding (CU) object generator 814. The context extractor 812 may extract context information associated with the user input. The context extractor 812 may also update the context information based on the assistant application 136 executed on the client system 130. As an example and not limitation, the update to the context information may include displaying a content item on the client system 130. As another example and not limitation, the update to the context information may include whether a reminder is set on the client system 130. As another example and not limitation, the update to the context information may include whether a song is being played on the client system 130. The CU object generator 814 may generate a specific CU object related to the user input. The CU object may include conversational session data and features associated with the user input, which may be shared with all modules of the assistant system 140. In particular embodiments, request manager 810 may store the context information and the generated CU objects in data store 820 , which is a particular data store implemented in assistant system 140 .

[0130] In a particular embodiment, the request manager 810 may send the generated CU object to the NLU module 710. The NLU module 710 may perform multiple steps to process the CU object. The NLU module 710 may first traverse the CU object in the allow list / block list 830. In a particular embodiment, the allow list / block list 830 may include interpretation data that matches the user input. Then, the NLU module 710 may perform characterization 832 of the CU object. Then, the NLU module 710 may perform domain classification / selection 834 on the user input based on the features generated by the characterization 832 to classify the user input into a predefined domain. In a particular embodiment, the domain may represent a social context of interaction (e.g., education), or a namespace of a set of intents (e.g., music). The domain classification / selection results may be further processed based on two related processes. In one process, the NLU module 710 may use a meta-intent classifier 836a to process the domain classification / selection results. The meta-intent classifier 836a may determine a category that describes the user intent. An intent may be an element in a predefined semantic intent classification, which may indicate the purpose of a user's interaction with the assistant system 140. The NLU module 710 may classify the user input as a member of a predefined classification. For example, the user input may be "play Beethoven's Fifth," and the NLU module 710 may classify the input as having the intent [IN:play_music]. In a particular embodiment, intents common to multiple domains may be processed by a meta-intent classifier 836a. As an example and not limitation, the meta-intent classifier 836a may be based on a machine learning model that may take a domain classification / selection result as input and calculate the probability that the input is associated with a specific predefined meta-intent. The NLU module 710 may then use a meta-slot annotator 838a to annotate one or more meta-slots for the classification result from the meta-intent classifier 836a. A slot may be a named substring corresponding to a string representing a basic semantic entity within the user input. For example, the slot for "pizza" may be [SL:dish]. In a particular embodiment, a set of valid or expected named slots may be conditional on the classification intent. As an example and not limitation, for the intent [IN:play_music], a valid slot may be [SL:song_name]. In particular embodiments, the meta-slot tagger 838a may tag a generic slot (e.g., a reference to an item (e.g., first)), the type of slot, the value of the slot, etc. In particular embodiments, the NLU module 710 may use an intent classifier 836b to process domain classification / selection results. The intent classifier 836b may determine a user intent associated with the user input. In particular embodiments, for each domain, there may be one intent classifier 836b to determine the most likely intent in a given domain.As an example and not limitation, the intent classifier 836b can be based on a machine learning model that can take the domain classification / selection result as input and calculate the probability that the input is associated with a specific predefined intent. Then, the NLU module 710 can use the slot tagger 838b to tag one or more slots associated with the user input. In a specific embodiment, the slot tagger 838b can tag one or more slots for the n-gram of the user input. As an example and not limitation, the user input can include "convert 500 US dollars in my account to Japanese yen". The intent classifier 836b can take the user input as input and represent it as a vector. Then, the intent classifier 836b can calculate the probability that the user input is associated with different predefined intents based on a vector comparison between the vector representing the user input and the vector representing different predefined intents. In a similar manner, the slot tagger 838b can take the user input as input and represent each word as a vector. Then, the slot tagger 838b can calculate the probability that each word is associated with different predefined slots based on a vector comparison between the vector representing the word and the vector representing different predefined slots. The user's intent may be classified as "Exchange currency". The user's input slots may include "500", "US dollars", "account", and "Japanese yen". The user's meta-intent may be classified as "Financial services". The meta-slot may include "Finance".

[0131] In certain embodiments, the natural language understanding (NLU) module 710 may also extract information from one or more of a social graph, a knowledge graph, or a concept graph, and may retrieve a user profile stored locally on the client system 130. The NLU module 710 may also consider contextual information when analyzing user input. The NLU module 710 may further process information from these different sources by identifying and aggregating information, annotating n-grams of user input, sorting n-grams using confidence scores based on the aggregated information, and representing the sorted n-grams as features that can be used by the NLU module 710 to understand the user input. In certain embodiments, the NLU module 710 may identify one or more of a domain, an intent, or a slot from the user input in a personalized and context-aware manner. As an example and not limitation, the user input may include "tell me how to get to a coffee shop." The NLU module 710 may identify a specific coffee shop that the user wants to go to based on the user's personal information and associated contextual information. In certain embodiments, the NLU module 710 may include a dictionary, a parser, and grammar rules for a specific language to divide the sentence into internal representations. The NLU module 710 may also include one or more programs that perform naive semantic or random semantic analysis, and may also use pragmatics to understand user input. In a particular embodiment, the parser may be based on such a deep learning architecture: the deep learning architecture includes multiple long-short term memory (LSTM) networks. As an example and not limitation, the parser may be based on a recurrent neural network grammar (RNNG) model, which is a recursive and cyclic LSTM algorithm. More information about natural language understanding (NLU) can be found in the following documents: U.S. Patent Application No. 16 / 011062 filed on June 18, 2018, U.S. Patent Application No. 16 / 025317 filed on July 2, 2018, and U.S. Patent Application No. 16 / 038120 filed on July 17, 2018, each of which is incorporated by reference.

[0132] In a particular embodiment, the output of the NLU module 710 may be sent to an entity resolution module 712 to resolve related entities. An entity may include, for example, a unique user or concept, each of which may have a unique identifier (ID). These entities may include one or more of the following: a real-world entity (from a general knowledge base), a user entity (from a user memory), a context entity (device context / conversation context), or a value resolution (number, date time, etc.). In a particular embodiment, the entity resolution module 712 may include a domain entity resolution 840 and a general entity resolution 842. The entity resolution module 712 may perform general entity resolution and domain-specific entity resolution. General entity resolution 842 may resolve entities by classifying slots and meta-slots into different general topics. Domain entity resolution 840 may resolve entities by classifying slots and meta-slots into different domains. As an example and not limitation, in response to the input of a query about the advantages of a particular brand of electric vehicles, general entity resolution 842 may resolve the mentioned brand of electric vehicles as vehicles, and domain entity resolution 840 may resolve the mentioned brand of electric vehicles as electric vehicles.

[0133] In a particular embodiment, entities can be parsed based on knowledge 850 about the world and the user. The assistant system 140 can extract ontology data from the graph 852. As an example and not limitation, the graph 852 may include one or more of a knowledge graph, a social graph, or a concept graph. The ontology data may include structural relationships between different slots / meta-slots and domains. The ontology data may also include information such as how slots / meta-slots can be grouped, how slots / meta-slots can be related within a hierarchy (wherein a higher level includes a domain), and how slots / meta-slots can be subdivided according to similarities and differences. For example, a knowledge graph may include multiple entities. Each entity may include a single record associated with one or more attribute values. A specific record may be associated with a unique entity identifier. Each record may have different values ​​for an attribute of an entity. Each attribute value may be associated with a confidence probability and / or a semantic weight. The confidence probability of an attribute value indicates the probability that the value is accurate for a given attribute. The semantic weight of an attribute value may indicate how semantically appropriate the value is for a given attribute, given all available information. For example, a knowledge graph may include an entity for a book named "BookName", which may include information extracted from multiple content sources (e.g., online social networks, online encyclopedias, book review sources, media databases, and entertainment content sources), which may be deduplicated, parsed, and fused to generate a single unique record of the knowledge graph. In this example, the entity named "BookName" may be associated with the "Fantasy" attribute value of the "Genre" entity attribute. More information about knowledge graphs can be found in the following documents: U.S. Patent Application No. 16 / 048049, filed on July 27, 2018, and U.S. Patent Application No. 16 / 048101, filed on July 27, 2018, each of which is incorporated by reference.

[0134] In certain embodiments, assistant user memory (AUM) 854 may include user context memory that helps determine how to more effectively assist users. AUM 854 may be a central location for storing user data, retrieving user data, indexing user data, and searching user data. By way of example and not limitation, AUM 854 may store information such as contacts, photos, reminders, and the like. In addition, AUM 854 may automatically synchronize data to servers and other devices (only for non-sensitive data). By way of example and not limitation, if a user sets a nickname for a contact on one device, all devices may synchronize and obtain the nickname based on AUM 854. In certain embodiments, AUM 854 may first prepare events, user status, reminders, and trigger status for storage in a data repository. A memory node identifier (ID) may be created to store an entry object in AUM 854, where an entry may be a piece of information about a user (e.g., photos, reminders, etc.). By way of example and not limitation, the first few bits of a memory node ID may indicate that this is a memory node ID type, the last few bits may be a user ID, and the last few bits may be a creation time. AUM 854 can then index this data so that it can be retrieved as needed. For this purpose, an index ID can be created. In a specific embodiment, given an "index keyword" (e.g., PHOTO_LOCATION) and an "index value" (e.g., "San Francisco"), AUM 854 can obtain a list of memory IDs with this attribute (e.g., photos in San Francisco). As an example and not limitation, the first few bits can indicate that this is an index ID type, the second few bits can be a user ID, and the third few bits can encode the "index keyword" and "index value". AUM 854 can also use a flexible query language for information retrieval. For this purpose, a relationship index ID can be created. In a specific embodiment, given a source memory node and edge type, AUM 854 can obtain the memory IDs of all the following target nodes: these target nodes have edges of this type derived from the source. As an example and not limitation, the first few bits can indicate that this is a relationship index ID type, the second few bits can be a user ID, and the third few bits can be the source node ID and edge type. In certain embodiments, AUM 854 can help detect concurrent updates of different events. More information about episodic memory can be found in the following documents: U.S. Patent Application No. 16 / 552,559, filed on August 27, 2019, which is incorporated by reference.

[0135] In certain embodiments, the entity resolution module 712 may use different techniques to resolve different types of entities. For real-world entities, the entity resolution module 712 may use a knowledge graph to resolve the coverage of entities, such as "music track", "movie", etc. For user entities, the entity resolution module 712 may use user memory or some proxy to resolve the coverage of user-specific entities, such as "contact", "reminder", or "relationship". For contextual entities, the entity resolution module 712 may perform coreference based on information from the context engine 720 to resolve references to entities in the context, such as "he", "she", "first", or "last". In certain embodiments, for coreference, the entity resolution module 712 may create references for entities determined by the NLU module 710. The entity resolution module 712 may then accurately resolve these references. As an example and not limitation, the user input may include "find the nearest grocery store for me and direct me there". Based on the coreference, the entity resolution module 712 may interpret "there" as "the nearest grocery store". In certain embodiments, coreference may depend on information from context engine 720 and dialog manager 716 to interpret references with improved accuracy. In certain embodiments, entity resolution module 712 may also resolve entities under context (device context or dialog context), such as entities shown on the screen or entities from the last dialog history. For value resolution, entity resolution module 712 may resolve the mentioned value into a precise value in a standard form, such as a numeric value, date time, address, etc.

[0136] In a particular embodiment, the entity resolution module 712 may first perform a check on applicable privacy constraints to ensure that performing entity resolution does not violate any applicable privacy policies. As an example and not limitation, the entity to be resolved may be another user who specifies in its privacy settings that its identity should not be searchable on an online social network. In this case, the entity resolution module 712 may not return an entity identifier for the user in response to the user input. By utilizing the described information obtained from the social graph, knowledge graph, concept graph, and user profile, and by complying with any applicable privacy policies, the entity resolution module 712 can resolve entities associated with the user input in a personalized, context-aware, and privacy-preserving manner.

[0137] In a particular embodiment, the entity resolution module 712 can work with the ASR module 708 to perform entity resolution. The following example shows how the entity resolution module 712 can resolve entity names. The entity resolution module 712 can first expand multiple names associated with a user into the normalized text form of each of these names as a phonetic consonant representation that can be phonetically transcribed using a double metaphone algorithm. Then, the entity resolution module 712 can determine n best candidate transcription sets and perform a parallel understanding process on all phonetic transcriptions in the n best candidate transcription sets. In a particular embodiment, each transcription that is parsed to the same intent can then be transformed (collapse) into a single intent. Then, each intent can be assigned a score corresponding to the highest scoring candidate transcription of the intent. During the transition, the entity resolution module 712 can identify various possible text transcriptions associated with each slot, which are associated by the boundary timing offset associated with the transcription of the slot. Then, the entity resolution module 712 can extract a subset of possible candidate transcriptions for each slot from multiple (e.g., 1000) candidate transcriptions, regardless of whether they are classified into the same intent. In this way, slots and intents can be a scored list of phrases. In certain embodiments, a new task or running task that can handle the intent (e.g., a message composition task for an intent to send a message to another user) can be identified and associated with the intent. Figure 1Then, the identified task can trigger the entity resolution module 712 by providing the entity resolution module 712 with a score list of phrases associated with one of its slots, and the category to be resolved. As an example and not limitation, if the entity attribute is specified as "friend", the entity resolution module 712 can traverse each candidate list of items in the same extension that can be run when the matcher is compiled. Each candidate extension of the item can be matched in a pre-compiled trie matching structure. A function based at least in part on the transcription input, the matching form and the friend's name can be used to score the match. As another example and not limitation, if the entity attribute is specified as "celebrity / prominent person", the entity resolution module 712 can perform a parallel search on the knowledge graph for each candidate item set of the slot output from the ASR module 708. The entity resolution module 712 can score the match based on the popularity of the person matched and the score signal provided by the ASR. In a specific embodiment, when the memory category is specified, the entity resolution module 712 can perform the same search for the user memory. The entity resolution module 712 can crawl backwards in the user's memory and try to match each memory (e.g., a person recently mentioned in a conversation, or seen and recognized through visual signals, etc.). For each entity, the entity resolution module 712 can use a match similar to how it matches friends (i.e., phonetic symbols). In a particular embodiment, the score can include a time decay factor associated with the most recent time the name was previously mentioned. The entity resolution module 712 can also combine, sort, and deduplicate all matches. In a particular embodiment, a task can receive a set of candidates. When there are multiple high-scoring candidates, the entity resolution module 712 can perform user-facilitated disambiguation (e.g., obtaining real-time user feedback on these candidates from the user).

[0138] In a particular embodiment, the context engine 720 can help the entity resolution module 712 improve entity resolution. The context engine 720 may include an offline aggregator and an online reasoning service. The offline aggregator may process multiple data associated with a user collected from a previous time window. As an example and not limitation, these data may include news push posts / comments collected within a predetermined time frame (e.g., from a previous 90-day window), interactions with news push posts / comments, search history, etc. The processing results may be stored in the context engine 720 as part of a user profile. The user profile of a user may include user profile data, which includes demographic information, social information, and contextual information associated with the user. The user profile data may also include the user's interests and preferences for multiple topics, which are aggregated through conversations on news push, search logs, messaging platforms, etc. The use of the user profile may be subject to privacy constraints to ensure that the user's information can only be used for his / her benefit and cannot be shared with anyone else. More information about user profiles can be found in the following documents: U.S. Patent Application No. 15 / 967,239 filed on April 30, 2018, which is incorporated by reference. In a particular embodiment, the online reasoning service can analyze the conversation data associated with the user received by the assistant system 140 at the current time. The analysis results can also be stored in the context engine 720 as part of the user profile. In a particular embodiment, both the offline aggregator and the online reasoning service can extract personalized features from multiple data. Other modules of the assistant system 140 can use the extracted personalized features to better understand the user input. In a particular embodiment, the entity resolution module 712 can process the information (e.g., user profile) from the context engine 720 in the following steps based on natural language processing (NLP). In a particular embodiment, the entity resolution module 712 can tokenize the text by text normalization based on NLP, extract syntactic features from the text, and extract semantic features from the text. The entity resolution module 712 can also extract features from context information, which is taken from the conversation history between the user and the assistant system 140. The entity resolution module 712 can further perform global word embedding, domain-specific embedding, and / or dynamic embedding based on the context information. The processing results can be annotated with entities by the entity tagger. Based on the annotations, the entity resolution module 712 can generate a dictionary. In a specific embodiment, the dictionary can include global dictionary features that can be dynamically updated offline. The entity resolution module 712 can sort the entities marked by the entity tagger.In certain embodiments, the entity resolution module 712 may communicate with a different graph 852, including one or more of a social graph, a knowledge graph, or a concept graph, to extract ontology data related to the information retrieved from the context engine 720. In certain embodiments, the entity resolution module 712 may further resolve entities based on the user profile, the ranked entities, and the information from the graph 852.

[0139] In a particular embodiment, the entity resolution module 712 can be driven by a task (corresponding to the agent 728). This reversal of the processing order can make it possible for the domain knowledge present in the task to be applied to pre-filter or bias the set of resolution targets when it is obvious and appropriate to do so. As an example and not limitation, for the expression "Who is John?", there is no explicit category implied in the expression. Therefore, the entity resolution module 712 can resolve "John" for all content. As another example and not limitation, for the expression "Send a message to John", the entity resolution module 712 can easily determine that "John" refers to a person who can send and receive messages. Therefore, the entity resolution module 712 can bias the resolution towards friends. As another example and not limitation, for the expression "What is John's most famous album?", in order to resolve "John", the entity resolution module 712 can first determine the task corresponding to the expression, that is, to find a music album. The entity resolution module 712 can determine that entities related to music albums include singers, producers, and recording studios. Therefore, the entity resolution module 712 can search in these types of entities in the music domain to resolve "John".

[0140] In a particular embodiment, the output of the entity resolution module 712 can be sent to the dialogue manager 716 to advance the flow of dialogue with the user. The dialogue manager 716 can be an asynchronous state machine that repeatedly updates the state and selects an action based on the new state. The dialogue manager 716 can also store previous dialogues between the user and the assistant system 140. In a particular embodiment, the dialogue manager 716 can perform dialogue optimization. Dialogue optimization involves the following challenges: understanding and identifying the most likely branch options in a dialogue with the user. As an example and not a limitation, the assistant system 140 can implement dialogue optimization techniques to avoid the need to confirm who the user wants to call, because the assistant system 140 can determine the high confidence that the person inferred based on the context and available data is the intended recipient. In a particular embodiment, the dialogue manager 716 can implement a reinforcement learning framework to improve dialogue optimization. The dialogue manager 716 can include a dialogue intent parsing 856, a dialogue state tracker 718, and an action selector 722. In a particular embodiment, the dialogue manager 716 can perform the selected action and then call the dialogue state tracker 718 again until the selected action requires a user response or there are no other actions to be performed. Each action selected may depend on the results of the execution of the previous action. In a particular embodiment, the conversation intent parsing 856 may parse the user intent associated with the current conversation session based on the conversation history between the user and the assistant system 140. The conversation intent parsing 856 may map the intent determined by the NLU module 710 to different conversation intents. The conversation intent parsing 856 may further sort the conversation intents based on signals from the NLU module 710, the entity resolution module 712, and the conversation history between the user and the assistant system 140.

[0141] In a particular embodiment, the dialog state tracker 718 can use a set of operators to track the dialog state. The operator can include the data and logic required to update the dialog state. Each operator can act as a delta of the dialog state after processing the received user input. In a particular embodiment, the dialog state tracker 718 can include a task tracker, which can be based on task specifications and different rules. The dialog state tracker 718 can also include a slot tracker and a co-referencing component, which can be based on rules and / or based on recent time. The co-referencing component can help the entity resolution module 712 to resolve entities. In an alternative embodiment, using the co-referencing component, the dialog state tracker 718 can replace the entity resolution module 712, and can resolve any reference / mention and keep tracking of the state. In a particular embodiment, the dialog state tracker 718 can use the task specification to convert the upstream results into candidate tasks, and use entity resolution to resolve the actual parameters. Both user status (e.g., the user's current activity) and task status (e.g., trigger conditions) can be tracked. Given the current state, the dialog state tracker 718 can generate candidate tasks that the assistant system 140 can process and perform for the user. By way of example and not limitation, the candidate tasks can include "show suggestions," "get weather information," or "take a photo." In particular embodiments, the dialog state tracker 718 can generate candidate tasks based on available data from, for example, a knowledge graph, user memory, and user task history. In particular embodiments, the dialog state tracker 718 can then use the parsed arguments to parse the trigger object. By way of example and not limitation, the user input "remind me to call mom when she's online and I'm home tonight" can be converted from NLU output to trigger representation by the dialog state tracker 718:

[0142]

[0143]

[0144] Table 1: Example transformation from NLU output to trigger representation

[0145] In the example above, "mom", "home", and "tonight" are represented by their respective entities: personEntity, locationEntity, datetimeEntity.

[0146] In certain embodiments, the dialog manager 716 may map events determined by the context engine 720 to actions. By way of example and not limitation, an action may be a natural language generation (NLG) action, a display or overlay, a device action, or a retrieval action. The dialog manager 716 may also perform context tracking and interaction management. Context tracking may include aggregating real-time event streams into a unified user state. Interaction management may include selecting the best action in each state. In certain embodiments, the dialog state tracker 718 may perform context tracking (i.e., tracking events related to the user). To support processing of event streams, the dialog state tracker 718 may use event processors (e.g., for disambiguation, confirmation, request) that may consume various types of events and update internal assistant states. Each event type may have one or more processors. Each event processor may be modifying a certain segment of the assistant state. In certain embodiments, multiple event processors may be operating on disjoint subsets of states (i.e., only one processor may have write access to a specific field in the state). In certain embodiments, all event processors may have the opportunity to process a given event. By way of example and not limitation, the dialog state tracker 718 may run all event handlers in parallel for each event, and may then merge the state updates proposed by the various event handlers (eg, most handlers may return NULL updates for each event).

[0147] In certain embodiments, the dialog state tracker 718 can be used as any programming processor (logic) that needs to be versioned. In certain embodiments, instead of directly changing the dialog state, the dialog state tracker 718 can be a marginal effect-independent component and can generate the n best candidates for the dialog state update operator (which proposes an update to the dialog state). The dialog state tracker 718 can include intent parsers that contain logic that handles different types of NLU intents based on the dialog state and generates operators. In certain embodiments, the logic can be organized by an intent processor, such as a disambiguation intent processor for processing intents when the assistant system 140 requests disambiguation, a confirmation intent processor including the logic for processing confirmation, etc. The intent parser can combine the turn intent with the dialog state to generate a contextual update of the dialog with the user. Then, the slot resolution component can recursively resolve the slots in the update operator using a resolution provider including a knowledge graph and a domain agent. In certain embodiments, the dialog state tracker 718 can update / sort the dialog state of the current dialog session. As an example and not limitation, if the dialog session ends, the dialog state tracker 718 can update the dialog state to “complete.” As another example and not limitation, the dialog state tracker 718 can sort the dialog state based on the priority associated with the dialog state.

[0148] In a particular embodiment, the dialogue state tracker 718 can communicate with the action selector 722 about the dialogue intent and the associated content object. In a particular embodiment, the action selector 722 can sort different dialogue hypotheses for different dialogue intents. The action selector 722 can use candidate dialogue state operators and query the dialogue strategy 860 to determine what action should be performed. In a particular embodiment, the dialogue strategy 860 can be a tree-based strategy, which is a pre-built dialogue plan. Based on the current dialogue state, the dialogue strategy 860 can select nodes to be executed and generate corresponding actions. As an example and not limitation, the tree-based strategy can include topic grouping nodes and dialogue action (leaf) nodes. In a particular embodiment, the dialogue strategy 860 can also include a data structure that describes the execution plan of the agent 728 for the action. The dialogue strategy 860 can further include multiple goals that are related to each other through logical operators. In a particular embodiment, the goal can be the result of a part of the dialogue strategy, and the goal can be constructed by the dialogue manager 716. A goal may be represented by an identifier (e.g., a string) with one or more named arguments that parameterize the goal. By way of example and not limitation, a goal with its associated goal arguments may be represented as {confirm_artist, args: {artist: "Madonna"}}. In particular embodiments, goals may be mapped to leaves of a tree of a tree structure representation of dialog policies 860.

[0149] In certain embodiments, assistant system 140 may use hierarchical dialog strategies 860, where generic strategies 862 handle cross-domain business logic and task strategies 864 handle task / domain specific logic. Generic strategies 862 may be used for actions that are not specific to individual tasks. Generic strategies 862 may be used to determine task stacking and switching, active tasks, notifications, etc. Generic strategies 862 may include: handling low confidence intents, internal errors, unacceptable user responses with retries, and / or skipping or inserting confirmations based on ASR or NLU confidence scores. Generic strategies 862 may also include logic that sorts dialog state update candidates output from dialog state tracker 718 and selects dialog state update candidates to be updated (e.g., selects the top ranked task intents). In certain embodiments, assistant system 140 may have a specific interface for generic strategies 862 that allows decentralized cross-domain policies / business rules (particularly those found in dialog state tracker 718) to be merged into the functionality of action selector 722. The interface for the generic policy 862 may also allow the creation of independent sub-policy units that can be bound to specific situations or clients (e.g., policy functionality that can be easily turned on or off based on the client, situation). The interface for the generic policy 862 may also allow for the provision of policy hierarchy (i.e., multiple policy units) with back-off, where highly specialized policy units that handle specific situations are backed up by more generic policies 862 that apply to a wider range of situations. In this context, the generic policy 862 may optionally include intent or task-specific policies.

[0150] In certain embodiments, task policies 864 may include logic for action selectors 722 based on the task and the current state. Task policies 864 may be dynamic and self-organizing. In certain embodiments, types of task policies 864 may include one or more of the following: (1) manually created tree-based dialogue plans; (2) coded policies that directly implement an interface for generating actions; (3) slot-filling tasks specified by a configurator; or (4) policies based on machine learning models learned from data. In certain embodiments, assistant system 140 may use rule-based logic to bootstrap new domains and then use machine learning models to refine task policies 864. In certain embodiments, general policies 862 may select an operator from candidate operators to update the dialogue state, and then task policies 864 may select a user-facing action. Once a task is active in a dialogue state, the corresponding task policy 864 may be queried to select the correct action.

[0151] In certain embodiments, the action selector 722 may select an action based on one or more of the following: an event determined by the context engine 720, a dialog intent and state, an associated content object, and guidance from a dialog strategy 860. Each dialog strategy 860 may subscribe to a specific condition on a state field. After the event is processed and the state is updated, the action selector 722 may run a fast search algorithm (e.g., similar to Boolean satisfiability) to identify which strategies should be triggered based on the current state. In certain embodiments, if multiple strategies are triggered, the action selector 722 may use a tie-breaker mechanism to select a specific strategy. Alternatively, the action selector 722 may use a more sophisticated approach as follows: the approach may preview each strategy and then select a specific strategy that may be determined to have a high probability of success. In certain embodiments, mapping events to actions may bring several technical advantages to the assistant system 140. One technical advantage may include that each event may be a state update from the user or the user's physical / digital environment, which may or may not trigger an action from the assistant system 140. Another technical advantage may include the possibility to handle rapid event bursts (e.g., a user enters a new building and sees many people) by first updating the state with all events and then triggering one or more actions based on the final state. Another technical advantage may include translating all events into a single global assistant state.

[0152] In certain embodiments, action selector 722 may take as part of the input a dialog state update operator to select a dialog action. The execution of a dialog action may generate a set of expectations to instruct dialog state tracker 718 to process future turns. In certain embodiments, the expectations may be used to provide context to dialog state tracker 718 when processing user input from the next turn. As an example and not limitation, a slot request dialog action may have an expectation to verify the value of the requested slot. In certain embodiments, both dialog state tracker 718 and action selector 722 may not change the dialog state until the selected action is executed. This may allow assistant system 140 to execute dialog state tracker 718 and action selector 722 for processing inferred ASR results and performing n-best ranking using previews.

[0153] In a particular embodiment, the action selector 722 may call different agents 728 to perform tasks. At the same time, the dialogue manager 716 may receive instructions to update the dialogue state. As an example and not a limitation, the update may include waiting for a response from the agent 728. The agent 728 may be selected from the registered content providers to complete the action. The data structure may be constructed by the dialogue manager 716 based on the intent and one or more slots associated with the intent. In a particular embodiment, the plurality of agents 728 may include first-party agents and third-party agents. In a particular embodiment, the first-party agent may include an internal agent (e.g., an agent associated with a service provided by an online social network (e.g., a messaging service or a photo sharing service)) that can be accessed and controlled by the assistant system 140. In a particular embodiment, the third-party agent may include an external agent (e.g., a third-party online music application agent, a ticketing agent) that the assistant system 140 cannot control. The first-party agent may be associated with a first-party provider that provides content objects and / or services hosted by the extended reality system 160. The third-party agent may be associated with a third-party provider that provides content objects and / or services hosted by a third-party system 170. In certain embodiments, each of the first-party agent or the third-party agent may be designated for a specific domain. By way of example and not limitation, the domain may include weather, traffic, music, shopping, social, video, photo, event, location, and / or work. In certain embodiments, the assistant system 140 may collaboratively use multiple agents 728 to respond to user input. By way of example and not limitation, the user input may include “direct me to my next meeting.” The assistant system 140 may use a calendar agent to retrieve the location of the next meeting. The assistant system 140 may then use a navigation agent to direct the user to the next meeting.

[0154] In a particular embodiment, the dialogue manager 716 can support multiple rounds of compositional parsing of slot mentions. For compositional parsing from the NLU module 710, the parser can recursively parse nested slots. The dialogue manager 716 can also support disambiguation of nested slots. As an example and not limitation, the user input can be "remind me to call Alex". The parser may need to know which Alex to call before creating an actionable reminder to-do entity. When further user clarification is required for a particular slot, the parser can pause parsing and set the parsing state. The general strategy 862 can check the parsing state and create corresponding dialogue actions for user clarification. The dialogue manager 716 can update the nested slots in the dialogue state tracker 718 based on the user input and the most recent dialogue actions. This functionality can allow the assistant system 140 to interact with the user so that it can not only collect missing slot values, but also reduce the ambiguity of more complex / ambiguous expressions to complete the task. In certain embodiments, the dialog manager 716 may further support requesting missing slots in nested intents and multi-intent user inputs (e.g., "Take this photo and send it to Dad"). In certain embodiments, the dialog manager 716 may support machine learning models for a more robust dialog experience. As an example and not limitation, the dialog state tracker 718 may use a neural network-based model (or any other suitable machine learning model) to model beliefs over task assumptions. As another example and not limitation, for the action selector 722, the highest priority policy unit may include whitelist / blacklist overrides that may have to be done by design; the medium priority unit may include a machine learning model designed for action selection; and the lower priority unit may include a rule-based fallback when the machine learning model chooses not to handle a situation. In certain embodiments, a general policy unit based on a machine learning model can help the assistant system 140 reduce redundant disambiguation or confirmation steps, thereby reducing the number of turns to execute user input.

[0155] In certain embodiments, the action determined by the action selector 722 may be sent to a delivery system 730. The delivery system 730 may include a CU editor 870, a response generation component 880, a dialog state writing component 882, and a text-to-speech (TTS) component 890. Specifically, the output of the action selector 722 may be received at the CU editor 870. In certain embodiments, the output from the action selector 722 may be represented as<k,c,u,d> tuple, where k indicates the knowledge source, c indicates the communication target, u indicates the user model, and d indicates the discourse model.

[0156] In a particular embodiment, the CU editor 870 may use a natural language generation (NLG) component 872 to generate communication content for a user. In a particular embodiment, the NLG component 872 may use different language models and / or language templates to generate natural language output. The generation of natural language output may be application-specific. The generation of natural language output may also be personalized for each user. In a particular embodiment, the NLG component 872 may include a content determination component, a sentence planner, and a surface implementation component. The content determination component may determine the communication content based on knowledge sources, communication goals, and user expectations. As an example and not limitation, the determination may be based on description logic. Description logic may include, for example, three basic notions, which are individuals (representing objects in a domain), concepts (describing a collection of individuals), and roles (representing binary relationships between individuals or concepts). Description logic may be characterized by a set of constructors that allow a natural language generator to build complex concepts / roles from atomic concepts / roles. In a particular embodiment, the content determination component may perform the following tasks to determine the communication content. The first task may include a conversion task, in which the input of the NLG component 872 may be converted into multiple concepts. The second task may include a selection task in which relevant concepts may be selected from the concepts generated from the conversion task based on the user model. The third task may include a verification task in which the coherence of the selected concepts may be verified. The fourth task may include an instantiation task in which the verified concepts may be instantiated into executable files that can be processed by the NLG component 872. The sentence planner may determine the organization of the communication content so that it can be understood by humans. The surface implementation component may determine the specific words to be used, the order of sentences, and the style of the communication content.

[0157] In a particular embodiment, the CU editor 870 may also use a UI payload generator 874 to determine the modality of the generated communication content. Since the generated communication content can be considered as a response to user input, the CU editor 870 may also use a response sorter 876 to sort the generated communication content. As an example and not a limitation, the sorting may indicate the priority of the response. In a particular embodiment, the CU editor 870 may include a natural language synthesis (natural-language synthesis, NLS) component, which may be separated from the NLG component 872. The NLS component may specify the properties (including gender, volume, rhythm, style, or range) of the synthesized speech generated by the CU editor 870 to customize the response for a specific user, task, or agent. The NLS component may adjust the language synthesis without participating in the implementation of the associated task. In a particular embodiment, the CU editor 870 may check the privacy constraints associated with the user to ensure that the generation of the communication content follows the privacy policy. More information about customized natural language generation (NLG) can be found in the following documents: U.S. patent application No. 15 / 967,279 filed on April 30, 2018, and U.S. patent application No. 15 / 966,455 filed on April 30, 2018, which are incorporated by reference.

[0158] In certain embodiments, the delivery system 730 may perform different tasks based on the output of the CU editor 870. These tasks may include: writing (i.e., storing / updating) the dialog state into the data repository 820 using the dialog state writing component 882, and generating responses using the response generation component 880. In certain embodiments, if the modality of the determined communication content is audio, the output of the CU editor 870 may also be sent to the TTS component 890. In certain embodiments, the output from the delivery system 730 may then be sent back to the dialog manager 716, the output including one or more of the following: the generated response, the communication content, or the speech generated by the TTS component 890.

[0159] In certain embodiments, the coordinator 706 may determine whether to process the user input on the client system 130, on the server, or in a third mode of operation (i.e., a hybrid mode) based on the output of the entity resolution module 712. In addition to determining how to process the user input, the coordinator 706 may also receive results from the agent 728 and / or from the delivery system 730 provided by the dialog manager 716. The coordinator 706 may then forward these results to the arbitrator 726. The arbitrator 726 may aggregate these results, analyze them, select the best results, and provide the selected results to the rendering output module 732. In certain embodiments, the arbitrator 726 may query the dialog policy 860 for guidance when analyzing these results. In certain embodiments, the rendering output module 732 may generate a response suitable for the client system 130.

[0160] Fig. 9An example task-centric flowchart 900 for processing user input is shown. In certain embodiments, the assistant system 140 can help users not only with voice-initiated experiences, but also with more proactive multimodal experiences initiated when understanding the user context. In certain embodiments, the assistant system 140 can rely on assistant tasks for such purposes. Assistant tasks can be such central concepts that are shared throughout the assistant stack to understand user intent, interact with the user and the world, and complete the correct task for the user. In certain embodiments, assistant tasks can be the original unit of assistant functions. Assistant functions can include data extraction, updating a certain state, executing a certain command, or complex tasks composed of a set of smaller tasks. Correctly and successfully completing tasks to pass values ​​to users may be the goal of optimizing the assistant system 140. In certain embodiments, assistant tasks can be defined as functions or features. If multiple product surfaces have exactly the same requirements, assistant tasks can be shared across the multiple product surfaces, so the assistant tasks can be easily tracked. Assistant tasks can also be passed from one device to another and are easily obtained by another device midway through the task because the original units are consistent. In addition, the consistent format of assistant tasks can allow developers working on different modules in the assistant stack to design around it more easily. In addition, it can also allow task sharing. As an example and not limitation, if a user is listening to music on smart glasses, the user can say "play this music on my phone." In the case where the phone has not yet been awakened or there is a task to perform, the smart glasses can formulate a task provided to the phone, which can then be executed by the phone to start playing music. In a specific embodiment, if each surface has different expected behaviors, each surface can retain assistant tasks separately. In a specific embodiment, the assistant system 140 can identify the correct task based on user input or other signals in different modalities on the server or local product surface, conduct a dialogue to collect all necessary information, and use an internally or externally implemented action selector 722 to complete the task. In a specific embodiment, the assistant stack may include a set of processing components that start from awakening, identifying user input, understanding user intent, reasoning about the task, and completing the task to generate a natural language response with voice.

[0161] In certain embodiments, the user input may include a voice input. The voice input may be received at the ASR module 708 to extract a text transcription from the voice input. The ASR module 708 may use a statistical model to determine the most likely sequence of words corresponding to a given portion of speech received as an audio input by the assistant system 140. These models may include one or more of a hidden Markov model, a neural network, a deep learning model, or any combination thereof. The received audio input may be encoded as digital data at a specific sampling rate (e.g., 16 kilohertz (kHz), 44.1kHz, or 96kHz) and using a specific number of bits (e.g., 8 bits, 16 bits out of 24 bits) to represent each sample.

[0162] In certain embodiments, the ASR module 708 may include one or more of the following: a grapheme-to-phoneme (G2P) model, a pronunciation learning model, a personalized acoustic model, a personalized language model (PLM), or an end-pointing model. In certain embodiments, the grapheme-to-phoneme (G2P) model may be used to determine a user's grapheme-to-phoneme style (i.e., what it might sound like when a particular user says a particular word). In certain embodiments, a personalized acoustic model may be a model of the relationship between an audio signal and the sound of a speech unit in a language. Thus, such a personalized acoustic model may recognize how a user's voice sounds. A personalized acoustic model may be generated using training data (e.g., a training speech received as an audio input and a corresponding speech unit corresponding to the speech). A personalized acoustic model may be trained or refined using the voice of a particular user to recognize the user's voice. In certain embodiments, the personalized language model may then determine the most likely phrase corresponding to the speech unit recognized for a particular audio input. A personalized language model may be a model of the probability that various word sequences may appear in the language. A personalized language model can be used to match the sound of a speech unit in an audio input with a sequence of words, and a greater weight can be assigned to a sequence of words that is more likely to be a phrase in the language. Then, the sequence of words with the highest weight can be selected as the text corresponding to the audio input. In a specific embodiment, the personalized language model can also be used to predict what words a user is most likely to say in a given context. In a specific embodiment, an end-pointing model can detect when the end of an expression is reached. In a specific embodiment, during the client process, the assistant system 140 can optimize the personalized language model at runtime based at least in part on the limited computing power of the client system 130. As an example and not a limitation, the assistant system 140 can pre-calculate multiple personalized language models for multiple possible topics that a user may talk about. When the user input is associated with a request for help, the assistant system 140 can quickly switch between the pre-calculated language models and optimize these language models locally based on user activity at runtime. Therefore, the assistant system 140 can retain computing resources while efficiently identifying topics associated with the user input. In a specific embodiment, the assistant system 140 can also dynamically re-learn user pronunciation at runtime.

[0163] In a particular embodiment, the user input may include non-speech input. Non-speech input may be received at the context engine 720 to determine events and contexts based on the non-speech input. The context engine 720 may determine multimodal events, which include speech / text intent, location updates, visual events, touch, gaze, gesture, action, device / application events, and / or any other suitable type of event. The speech / text intent may depend on the ASR module 708 and the NLU module 710. Location updates may be used by the dialog manager 716 to support various active / passive scenarios. Visual events may be based on people or objects that appear in the user's field of view. These events may be used by the dialog manager 716 and recorded in the temporary user state to support visual co-reference (e.g., parsing out "that shirt how much?" in "that" and parsing out "send him my contact information" in "him"). Gaze, gesture, and activity may cause a tag to be set under the temporary user state (e.g., the user is running), which may adjust the action selector 722. For device / application events, if the application updates the device state, it can be published to the assistant system 140 so that the dialogue manager 716 can use the context (the context currently displayed to the user) to handle passive and active scenarios. As an example and not limitation, the context engine 720 can cause a push notification message to be displayed on the display screen of the user's client system 130. The user can interact with the push notification message, which can initiate a multimodal event (e.g., an event workflow for replying to a message received from another user). Other example multimodal events can include seeing a friend, seeing a landmark, being at home, running, starting a call with touch, taking a photo with touch, opening an application, etc. In a specific embodiment, the context engine 720 can also determine world events / social events based on world updates / social updates (e.g., weather changes, friends online). Social updates can include events that the user subscribes to (e.g., friends' birthdays, posts, comments, other notifications). The dialogue manager 716 can use these updates to trigger active actions based on context (e.g., suggesting that the user call a friend on his birthday, but only when the user is not paying attention to other things). By way of example and not limitation, receiving a message may be a social event that may trigger a task to read the message to the user.

[0164] In a particular embodiment, the text transcription from the ASR module 708 can be sent to the NLU module 710. The NLU module 710 can process the text transcription and extract the user intent (i.e., multiple intents), and parse the slot or parse results based on the language ontology. In a particular embodiment, the intent and slot from the NLU module 710, and / or the event and context from the context engine 720 can be sent to the entity resolution module 712. In a particular embodiment, the entity resolution module 712 can resolve the entity associated with the user input based on the output from the NLU module 710 and / or the context engine 720. The entity resolution module 712 can use different techniques to resolve the entity, including accessing the user memory from the assistant user memory (AUM) 854. In a particular embodiment, the AUM 854 may include a user context memory that helps the entity resolution module 712 to resolve the entity. The AUM854 can be a central location for storing user data, retrieving user data, indexing user data, and searching user data.

[0165] In certain embodiments, the entity resolution module 712 may provide one or more of intent, slot, entity, event, context, or user memory to the dialog state tracker 718. The dialog state tracker 718 may accordingly identify a set of state candidates for the task, interact with the user to collect the information required to fill the state, and call the action selector 722 to complete the task. In certain embodiments, the dialog state tracker 718 may include a task tracker 910. The task tracker 910 may track the task state associated with the assistant task. In certain embodiments, the task state may be a data structure that persists across multiple interaction rounds and is updated in real time to collect the task state during the entire interaction. The task state may include all current information related to the task execution state, such as actual parameters, confirmation status, confidence scores, etc. Any errors or outdated information in the task state may cause the task execution to fail or error. The task state may also be used as a set of context information for many other components (e.g., ASR module 708, NLU module 710, etc.).

[0166] In a particular embodiment, the task tracker 910 may include an intent processor 911, a task candidate ranking module 914, a task candidate generation module 916, and a merging layer 919. In a particular embodiment, a task may be identified by its ID name. If the task ID is not explicitly set in the task specification (e.g., dialogue strategy 860, agent execution, NLG dialogue action, etc.), the task ID may be used to associate the corresponding component resource (asset). Therefore, the output from the entity resolution module 712 may be received by the task ID resolution component 917 of the task candidate generation module 916 to resolve the task ID of the corresponding task. In a particular embodiment, the task ID resolution component 917 may call the task specification manager API 930 to access the trigger specification and deployment specification for resolving the task ID. Given these specifications, the task ID resolution component 917 may use intent, slot, dialogue state, context, and user memory to resolve the task ID.

[0167] In a particular embodiment, the technical specification of a task can be defined by a task specification. The assistant system 140 can use the task specification to trigger a task, conduct a dialogue session, and find the correct execution module (e.g., agent 728) to perform the task. The task specification can be the implementation of a product requirement document. The task specification can be used as a requirement for a general contract and all component agreements. The task specification can be considered as the assembly specification of a product, and all development partners deliver these modules based on the specification. In a particular embodiment, the assistant task can be defined in the implementation by specification. As an example and not a limitation, the task rule can be defined as the following categories. One category can be a basic task scheme, which includes basic identification information such as schemes for ID, name, and input parameters. Another category can be a trigger specification on how to trigger a task, such as an intention, an event message ID, etc. Another category can be a dialogue specification for a dialogue manager 716 to conduct a dialogue with a user and a system. Another category can be an execution specification on how to execute and complete a task. Another category can be a deployment specification on how to deploy a function to certain surfaces, locals, and user groups.

[0168] In a particular embodiment, the task specification manager API 930 can be an API for accessing the task specification manager. The task specification manager can be a module in the runtime stack that is used to load specifications based on all tasks and provide an interface to access all task specifications to obtain detailed information or generate task candidates. In a particular embodiment, the task specification manager can access all components in the runtime stack through the task specification manager API 930. The task specification manager can include a set of static utility functions to utilize the task specification manager to manage tasks, such as filtering task candidates through the platform. Before the task specifications are landed, the assistant system 140 can also dynamically load multiple task specifications to support end-to-end development during the development phase.

[0169] In certain embodiments, task specifications may be grouped by domain and stored in runtime configuration 935. The runtime stack may load all task specifications from runtime configuration 935 during build time. In certain embodiments, in runtime configuration 935, for a domain, there may be a cconf file and a cinc file (e.g., sidechef_task.cconf and sidechef_task.inc). By way of example and not limitation, <domain>_tasks.cconf can include all the details of the task specification. As another example and not limitation, if this feature is not yet supported, <domain>_tasks.cinc can provide a way to override the generated specification.

[0170] In a particular embodiment, task execution may require a group of actual parameters to execute.Therefore, actual parameter parsing component 918 can use the actual parameter specification of the task ID being parsed to parse the actual parameter name.These actual parameters can be parsed based on NLU output (for example, slot [slot: contact person] ([SL:contact])), dialogue state (for example, short-term call history), user memory (for example, user preference, position, long-term call history, etc.) or device context (for example, timer state, screen content, etc.).In a particular embodiment, the actual parameter modality can be text, audio, image or other structured data.The mapping of slot to actual parameters can be defined by filling strategy and / or language ontology.In a particular embodiment, in the case of a given task trigger specification, task candidate generation module 916 can be based on the task ID being parsed and actual parameters, to find the task list to be triggered as a task candidate.

[0171] In a particular embodiment, the generated task candidates can be sent to the task candidate sorting module 914 for further sorting. The task candidate sorting module 914 can use a rule-based sorter 915 to sort them. In a particular embodiment, the rule-based sorter 915 may include a set of heuristics to favor certain domain tasks. The sorting logic can be described below according to the principle of context priority. In a particular embodiment, the priority of the user-specified task can be higher than the foreground task. When the intent is a meta-intent, the priority of the foreground task can be higher than the device domain task. The priority of the device domain task can be higher than the task that triggers the intent domain. As an example and not limitation, if the task domain is mentioned or specified in the expression (e.g., "create a timer in the timer application"), the sorting can pick out the task. As another example and not limitation, when the timer application is in the foreground and there is an active timer, if the task domain is in the foreground or active state, the sorting can pick out the task (e.g., "stop timer" to stop the timer). As another example and not limitation, if the intent is a general meta-intent and the task is device control when there are no other active applications or active states, the sorting can pick out the task. As another example and not limitation, if the task is the same as the intent domain, the sorting can pick out the task. In a particular embodiment, the task candidate sorting module 914 can customize some more logic to check for matches of intent / slot / entity types. The sorted task candidates can be sent to the merging layer 919.

[0172] In a particular embodiment, the output from the entity resolution module 712 may be sent to a task ID resolution component 912 of the intent processor 911. Similar to the task ID resolution component 917, the task ID resolution component 912 may resolve the task ID of the corresponding task. In a particular embodiment, the intent processor 911 may also include an actual parameter resolution component 913. Similar to the actual parameter resolution component 918, the actual parameter resolution component 913 may resolve the actual parameter name using the actual parameter specification of the resolved task ID. In a particular embodiment, the intent processor 911 may process features that are not related to the task and may not be expressed in a task-specific task specification. The intent processor 911 may output state candidates other than task candidates, such as actual parameter updates, confirmation updates, disambiguation updates, etc. In a particular embodiment, some tasks may require very complex trigger conditions or very complex actual parameter filling logic, which may not be reused by other tasks (e.g., voice commands in a call, media tasks via [INTENT: PLAY_MEDIA], etc.), even if they are supported in the task specification. The intent processor 911 may also be adapted for this type of task. In certain embodiments, the results from the intent processor 911 may take precedence over the results from the task candidate ranking module 914. The results from the intent processor 911 may also be sent to the merging layer 919.

[0173] In certain embodiments, the merging layer 919 may combine the results from the intent processor 911 and the results from the task candidate ranking module 914. The dialog state tracker 718 may suggest each task as a new state for selection by the dialog policy 860, thereby generating a list of state candidates. The merged results may be further sent to a conversational understanding reinforcement engine (CURE) tracker 920. In certain embodiments, the CURE tracker 920 may be a personalized learning process to use real-time user feedback to improve the dialog state tracker 718's determination of state candidates in different contexts. For more information on the dialog understanding reinforcement engine, see the following document: U.S. Patent Application No. 17 / 186,459, filed on February 26, 2021, which is incorporated by reference.

[0174] In certain embodiments, the candidate states generated by CURE tracker 920 may be sent to action selector 722. Action selector 722 may query task policy 864, which may be generated from an execution specification accessed through task specification manager API 930. In certain embodiments, the execution specification may describe how the task should be performed and what actions action selector 722 may need to take to complete the task.

[0175] In certain embodiments, action selector 722 may determine an action associated with a system. Such an action may require agent 728 to perform. Therefore, action selector 722 may send system actions to agent 728, and agent 728 may return the results of the execution of these actions. In certain embodiments, action selector may determine an action associated with a user or device. Such an action may require execution by delivery system 730. Therefore, action selector 722 may send user / device actions to delivery system 730, and delivery system 730 may return the results of the execution of these actions.

[0176] The embodiments disclosed herein may include an artificial reality system, or be implemented in combination with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to the user, and the artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with collected content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of the above may be presented in a single channel or multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality, and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0177] Artificial Neural Networks

[0178] Fig.10 An example artificial neural network ("ANN") 1000 is shown. In particular embodiments, an ANN may refer to a computational model that includes one or more nodes. Example ANN 1000 may include an input layer 1010, hidden layers 1020, 1030, 1040, and an output layer 1050. Each layer of ANN 1000 may include one or more nodes, such as node 1005 or node 1015. In particular embodiments, each node of the ANN may be connected to another node of the ANN. As an example and not limitation, each node of the input layer 1010 may be connected to one or more nodes of the hidden layer 1020. In particular embodiments, one or more nodes may be bias nodes (e.g., nodes in a layer that are not connected to any node in the previous layer and do not receive input from any node in the previous layer). In particular embodiments, each node in each layer may be connected to one or more nodes in the previous or next layer. Although Fig.10 A particular ANN is depicted having a particular number of layers, a particular number of nodes, and particular connections between those nodes, but this disclosure contemplates any suitable ANN having any suitable number of layers, any suitable number of nodes, and any suitable connections between those nodes. By way of example and not limitation, although Fig.10 Connections between each node of the input layer 1010 and each node of the hidden layer 1020 are depicted, but one or more nodes of the input layer 1010 may not be connected to one or more nodes of the hidden layer 1020 .

[0179] In certain embodiments, the ANN may be a feedforward ANN (e.g., an ANN without cycles or loops, where communication between nodes flows in one direction, starting from the input layer and continuing to subsequent layers). As an example and not limitation, the input to each node of the hidden layer 1020 may include the output of one or more nodes of the input layer 1010. As another example and not limitation, the input to each node of the output layer 1050 may include the output of one or more nodes of the hidden layer 1040. In certain embodiments, the ANN may be a deep neural network (e.g., a neural network including at least two hidden layers). In certain embodiments, the ANN may be a deep residual network. The deep residual network may be a feedforward ANN including hidden layers organized as residual blocks. The input to each residual block after the first residual block may be a function of the output of the previous residual block and the input of the previous residual block. As an example and not limitation, the input into residual block N may be F(x)+x, where F(x) may be the output of residual block N-1, and x may be the input into residual block N-1. Although this disclosure describes a particular ANN, this disclosure contemplates any suitable ANN.

[0180] In certain embodiments, an activation function may correspond to each node of the ANN. For a given input, the activation function of a node may define the output of the node. In certain embodiments, the input to a node may include a set of inputs. As an example and not limitation, the activation function may be an identity function, a binary step function, a logistic function, or any other suitable function. As another example and not limitation, the activation function of node k may be a sigmoid function Hyperbolic tangent function Rectifier F k (s k )=max(0,s k ) or any other suitable function F k (s k ), where s k can be an effective input to node k. In certain embodiments, the inputs to the activation functions corresponding to the nodes can be weighted. Each node can generate an output based on the weighted inputs using the corresponding activation function. In certain embodiments, each connection between nodes can be associated with a weight. As an example and not limitation, the connection 1025 between node 1005 and node 1015 can have a weighting factor of 0.4, which can indicate that the output of node 1005 is multiplied by 0.4 to be used as the input of node 1015. As another example and not limitation, the output y of node k is k It can be k =F k (s k ), where F k can be the activation function corresponding to node k, s k =∑ j (w jk x j ) can be a valid input to node k, x j can be the output of node j connected to node k, and w jk It can be a weighted coefficient between node j and node k. In a specific embodiment, the input to each node of the input layer can be based on a vector representing an object. Although the present disclosure describes specific inputs to nodes and specific outputs of nodes, the present disclosure contemplates any suitable inputs to nodes and any suitable outputs of nodes. In addition, although the present disclosure may describe specific connections and weights between nodes, the present disclosure contemplates any suitable connections and weights between nodes.

[0181] In certain embodiments, the ANN may be trained using training data. As an example and not limitation, the training data may include inputs and expected outputs to the ANN 1000. As another example and not limitation, the training data may include multiple vectors each representing a training object and an expected label for each training object. In certain embodiments, training the ANN may include modifying weights associated with connections between nodes of the ANN by optimizing an objective function. As an example and not limitation, a training method (e.g., conjugate gradient method, gradient descent method, stochastic gradient descent method) may be used to back propagate the sum-of-squares error measured as the distance between the vectors representing the training objects (e.g., using a cost function that minimizes the sum-of-squares error). In certain embodiments, the ANN may be trained using a dropout technique. As an example and not limitation, one or more nodes may be temporarily omitted during training (e.g., the one or more nodes do not receive input and do not generate output). For each training object, one or more nodes of the ANN may have a certain possibility of being omitted. The nodes omitted for a particular training object may be different from the nodes omitted for other training objects (e.g., these nodes may be temporarily omitted on an object-by-object basis). Although this disclosure describes training the ANN in a particular manner, this disclosure contemplates training the ANN in any suitable manner.

[0182] privacy

[0183] In a particular embodiment, one or more objects (e.g., content or other types of objects) of a computing system may be associated with one or more privacy settings. The one or more objects may be stored on or otherwise associated with any suitable computing system or application, such as an extended reality system 160, a client system 130, an assistant system 140, a third-party system 170, a social network application, an assistant application, a messaging application, a photo sharing application, or any other suitable computing system or application. Although the examples discussed herein are in the context of an online social network, these privacy settings may be applied to any other suitable computing system. The privacy settings (or "access settings") of an object may be stored in any suitable manner (e.g., associated with the object, in an index on an authorization server, in another suitable manner, or in any suitable combination of these manners). The privacy settings of an object may specify how the object (or specific information associated with the object) may be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, executed, presented, or identified) within an online social network. When the privacy settings of an object allow a particular user or other entity to access the object, the object may be described as "visible" relative to the user or other entity. By way of example and not limitation, users of an online social network may specify privacy settings for user profile pages that identify a group of users who may access work experience information on the user profile pages, thereby denying other users access to that information.

[0184] In certain embodiments, the privacy settings of an object may specify a "blocked list" of users or other entities that should not be allowed to access certain information associated with the object. In certain embodiments, the blocked list may include third-party entities. The blocked list may specify one or more users or entities for which the object is not visible. As an example and not limitation, a user may specify a group of users that may not access an album associated with the user, thereby blocking these users from accessing the album (while also allowing certain users who are not in the specified group of users to access the album). In certain embodiments, privacy settings may be associated with specific social graph elements. The privacy settings of a social graph element (e.g., a node or edge) may specify how an online social network may be used to access the social graph element, information associated with the social graph element, or an object associated with the social graph element. As an example and not limitation, a specific photo may have a privacy setting that specifies that only the user tagged in the photo and the friends of the user tagged in the photo can access the photo. In certain embodiments, privacy settings may allow users to opt-in or opt-out of having their content, information, or actions stored / recorded by extended reality system 160 or assistant system 140, or to opt-in or opt-out of having their content, information, or actions shared with other systems (e.g., third-party systems 170). Although this disclosure describes using particular privacy settings in particular ways, this disclosure contemplates using any suitable privacy settings in any suitable way.

[0185] In certain embodiments, the extended reality system 160 may present a "privacy wizard" to the first user (e.g., within a webpage, module, one or more dialog boxes, or any other suitable interface) to assist the first user in specifying one or more privacy settings. The privacy wizard may display instructions, appropriate privacy-related information, current privacy settings, one or more input fields for receiving one or more inputs from the first user to change or confirm the specified privacy settings, or any suitable combination thereof. In certain embodiments, the extended reality system 160 may provide the first user with a "dashboard" function that may display the first user's current privacy settings to the first user. The dashboard function may be displayed to the first user at any suitable time (e.g., after input from the first user invoking the dashboard function, after a particular event or triggering action occurs). The dashboard function may allow the first user to modify one or more of the first user's current privacy settings at any time and in any suitable manner (e.g., redirecting the first user to the privacy wizard).

[0186] The privacy settings associated with an object can specify any suitable granularity at which access is allowed or denied. As an example and not limitation, access or denial of access can be specified for the following items: specific users (e.g., only me, my roommate, my boss), users within a specific degree of separation (e.g., friends, friends of friends), user groups (e.g., a gaming club, my family), user networks (e.g., employees of a specific employer, students or alumni of a specific university), all users ("public"), no users ("private"), users of third-party systems 170, specific applications (e.g., third-party applications, external websites), other suitable entities, or any suitable combination of the above items. Although the present disclosure describes a specific granularity at which access is allowed or denied, the present disclosure contemplates any suitable granularity at which access is allowed or denied.

[0187] In certain embodiments, one or more servers 162 may be authorization / privacy servers for implementing privacy settings. In response to a request from a user (or other entity) for a particular object stored in a data repository 164, the extended reality system 160 may send a request for the object to the data repository 164. The request may identify a user associated with the request, and the request may only be sent to the user (or the user's client system 130) if the authorization server determines that the user is authorized to access the object based on the privacy settings associated with the object. If the requesting user is not authorized to access the object, the authorization server may block the retrieval of the requested object from the data repository 164, or may block the sending of the requested object to the user. In the context of a search query, the object may be provided as a search result only if the querying user is authorized to access the object, for example, if the privacy settings of the object allow the object to be presented to the querying user, discovered by the querying user, or otherwise visible to the querying user. In certain embodiments, the object may represent content visible to the user through the user's news feed. As an example and not limitation, one or more objects may be visible to the user's "Trending" page. In a particular embodiment, an object may correspond to a particular user. The object may be content associated with the particular user, or may be an account of the particular user or information stored on the extended reality system 160 or other computing system. As an example and not limitation, a first user may view one or more second users of an online social network through a "people you may know" feature of the online social network or by viewing the first user's friend list. As an example and not limitation, a first user may specify that he or she does not want to see an object associated with a particular second user in his or her news feed or friend list. If the privacy settings of the object do not allow it to be presented to the user, discovered by the user, or visible to the user, the object may be excluded from the search results. Although the present disclosure describes performing privacy settings in a particular manner, the present disclosure contemplates performing privacy settings in any suitable manner.

[0188] In certain embodiments, different objects of the same type associated with a user may have different privacy settings. Different types of objects associated with a user may have different types of privacy settings. As an example and not limitation, a first user may specify that the first user's status update is public, but any images shared by the first user are only visible to the first user's friends on an online social network. As another example and not limitation, a user may specify different privacy settings for different types of entities (e.g., individual users, friends of friends, followers, user groups, or business entities). As another example and not limitation, a first user may specify a group of users who may view videos posted by the first user while preventing these videos from being visible to the first user's employer. In certain embodiments, different privacy settings may be provided for different user groups or user visitors. As an example and not limitation, a first user may specify that other users at the same university as the first user may view the first user's photos, but other users who are family members of the first user may not view these same photos.

[0189] In certain embodiments, the extended reality system 160 may provide one or more default privacy settings for each object of a particular object type. The privacy setting for an object that is set to default may be changed by a user associated with the object. As an example and not limitation, all images posted by a first user may have a default privacy setting that is visible only to the first user's friends, and for a particular image, the first user may change the privacy setting for the image to be visible to friends and friends of friends.

[0190] In certain embodiments, privacy settings may allow a first user to specify (e.g., by opting out, by not opting in) whether the extended reality system 160 or assistant system 140 may receive, collect, record, or store specific objects or information associated with the user for any purpose. In certain embodiments, privacy settings may allow a first user to specify whether a specific application or process may access, store, or use specific objects or information associated with the user. Privacy settings may allow a first user to opt in or opt out of having an object or information accessed, stored, or used by a specific application or process. The extended reality system 160 or assistant system 140 may access such information to provide specific functionality or services to the first user, while the extended reality system 160 or assistant system 140 may not access the information for any other purpose. Prior to accessing, storing, or using these objects or information, the extended reality system 160 or assistant system 140 may prompt the user to provide privacy settings, specifying which applications or processes (if any) may access, store, or use the object or information, before allowing any such actions. By way of example and not limitation, a first user may send a message to a second user via an application associated with an online social network (e.g., a messaging application) and may specify a privacy setting that extended reality system 160 or assistant system 140 should not store such messages.

[0191] In certain embodiments, the first user may specify whether certain types of objects or information associated with the first user may be accessed, stored, or used by the extended reality system 160 or the assistant system 140. As an example and not limitation, the first user may specify that images sent by the first user through the extended reality system 160 or the assistant system 140 may not be stored by the extended reality system 160 or the assistant system 140. As another example and not limitation, the first user may specify that messages sent from the first user to a particular second user may not be stored by the extended reality system 160 or the assistant system 140. As yet another example and not limitation, the first user may specify that all objects sent via a particular application may be saved by the extended reality system 160 or the assistant system 140.

[0192] In certain embodiments, the privacy settings may allow the first user to specify whether a particular object or information associated with the first user can be accessed from a particular client system 130 or third-party system 170. The privacy settings may allow the first user to opt-in or opt-out of having objects or information accessed from a particular device (e.g., a phone book on the user's smartphone), from a particular application (e.g., a messaging application), or from a particular system (e.g., an email server). The extended reality system 160 or assistant system 140 may provide default privacy settings for each device, system, or application, and / or may prompt the first user to specify a specific privacy setting for each context. As an example and not limitation, the first user may utilize the location service feature of the extended reality system 160 or assistant system 140 to provide recommendations for restaurants or other places near the user. The first user's default privacy settings may specify that the extended reality system 160 or assistant system 140 may use the location information provided from the first user's client system 130 to provide location-based services, but the extended reality system 160 or assistant system 140 may not store the first user's location information or provide it to any third-party system 170. The first user may then update the privacy settings to allow the third-party image sharing application to use the location information in order to geotag the photos.

[0193] In certain embodiments, privacy settings may allow a user to specify one or more geographic locations where an object may be accessed. Access or denial of access to an object may depend on the geographic location of the user who is attempting to access the object. As an example and not limitation, a user may share an object and specify that only users in the same city may access or view the object. As another example and not limitation, a first user may share an object and specify that the object is visible to a second user only when the first user is in a particular location. If the first user leaves the particular location, the object is no longer visible to the second user. As another example and not limitation, the first user may specify that the object is visible only to second users within a threshold distance from the first user. If the first user subsequently changes location, the original second user who had access to the object may lose access, while a new set of second users may gain access when they come within a threshold distance of the first user.

[0194] In certain embodiments, the extended reality system 160 or assistant system 140 may have a function that can use the user's personal information or biometric information as input for user authentication or experience personalization purposes. Users can choose to use these functions to enhance their experience in online social networks. As an example and not a limitation, the user can provide personal information or biometric information to the extended reality system 160 or assistant system 140. The user's privacy settings can specify that such information can only be used for a specific process (e.g., authentication), and also specify that such information cannot be shared with any third-party system 170 or used for other processes or applications associated with the extended reality system 160 or assistant system 140. As another example and not a limitation, the extended reality system 160 can provide the user with the function of providing a voiceprint recording to an online social network. As an example and not a limitation, if the user wishes to utilize this function of the online social network, the user can provide a voice recording of his or her own voice to provide a status update on the online social network. The recording of the voice input can be compared with the user's voiceprint to determine what words the user said. The user's privacy settings may specify that such voice recordings may be used only for voice input purposes (e.g., to authenticate the user, send voice messages, improve voice recognition, in order to use the voice-operated features of an online social network), and may also specify that such voice recordings may not be shared with any third-party systems 170 or used by other processes or applications associated with the extended reality system 160.

[0195] System and method

[0196] Fig.11 An example computer system 1100 is shown. In particular embodiments, one or more computer systems 1100 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 1100 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 1100 performs one or more steps of one or more methods described or illustrated herein, or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 1100. In this document, references to computer systems may encompass computing devices, and vice versa, where appropriate. In addition, references to computer systems may encompass one or more computer systems, where appropriate.

[0197] The present disclosure contemplates any suitable number of computer systems 1100. The present disclosure contemplates computer systems 1100 in any suitable physical form. By way of example and not limitation, computer system 1100 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or a system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive self-service terminal (kiosk), a mainframe, a network of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these systems. Where appropriate, computer system 1100 may include one or more computer systems 1100; may be single or distributed; across multiple locations; across multiple machines; across multiple data centers; or located in a cloud (which may include one or more cloud components in one or more networks). Where appropriate, one or more computer systems 1100 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 1100 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch processing mode. Where appropriate, one or more computer systems 1100 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations.

[0198] In a particular embodiment, computer system 1100 includes a processor 1102, memory 1104, storage 1106, an input / output (I / O) interface 1108, a communication interface 1110, and a bus 1112. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0199] In a particular embodiment, the processor 1102 includes hardware for executing a plurality of instructions, such as those that constitute a computer program. By way of example and not limitation, to execute a plurality of instructions, the processor 1102 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 1104, or storage 1106; decode and execute the instructions; and then write one or more results to an internal register, an internal cache, memory 1104, or storage 1106. In a particular embodiment, the processor 1102 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1102 that includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 1102 may include one or more instruction caches, one or more data caches, and one or more page table caches (translation lookaside buffers, TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1104 or storage 1106, and the instruction cache may speed up the retrieval of the instructions by the processor 1102. The data in the data cache may be a copy of the data in the memory 1104 or storage 1106 for the instruction operation executed at the processor 1102; may be the result of the previous instruction executed at the processor 1102, for access by the subsequent instruction executed at the processor 1102, or for writing to the memory 1104 or storage 1106; or may be other suitable data. The data cache may accelerate the read operation or write operation of the processor 1102. The TLB may accelerate the virtual address translation of the processor 1102. In a particular embodiment, the processor 1102 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1102 including any suitable number of any suitable internal registers. Where appropriate, the processor 1102 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 1102. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0200] In certain embodiments, the memory 1104 includes a main memory for storing instructions for execution by the processor 1102 or data for operation by the processor 1102. As an example and not limitation, the computer system 1100 can load instructions from the memory 1106 or another source (e.g., another computer system 1100) to the memory 1104. The processor 1102 can then load these instructions from the memory 1104 to an internal register or an internal cache memory. In order to execute these instructions, the processor 1102 can retrieve these instructions from the internal register or the internal cache memory and decode them. During or after executing these instructions, the processor 1102 can write one or more results (which can be intermediate results or final results) to the internal register or the internal cache memory. The processor 1102 can then write one or more of these results to the memory 1104. In a particular embodiment, the processor 1102 executes only instructions in one or more internal registers or in one or more internal cache memories, or in memory 1104 (different from memory 1106 or other locations), and operates only on data in one or more internal registers or in internal cache memories, or in memory 1104 (different from memory 1106 or other locations). One or more memory buses (each memory bus may include an address bus and a data bus) can couple the processor 1102 to the memory 1104. As described below, the bus 1112 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) are located between the processor 1102 and the memory 1104 and facilitate access to the memory 1104 requested by the processor 1102. In a particular embodiment, the memory 1104 includes random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1104 may include one or more memories 1104. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0201] In a particular embodiment, the memory 1106 includes a large capacity memory for data or instructions. As an example and not limitation, the memory 1106 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these memories. Where appropriate, the memory 1106 may include a removable medium or a non-removable (or fixed) medium. Where appropriate, the memory 1106 may be inside or outside the computer system 1100. In a particular embodiment, the memory 1106 is a non-volatile solid-state memory. In a particular embodiment, the memory 1106 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory, or a combination of two or more of these ROMs. The present disclosure contemplates mass storage 1106 taking any suitable physical form. Where appropriate, storage 1106 may include one or more storage control units that facilitate communication between processor 1102 and storage 1106. Where appropriate, storage 1106 may include one or more memories 1106. Although the present disclosure describes and illustrates particular storage, the present disclosure contemplates any suitable storage.

[0202] In a particular embodiment, the I / O interface 1108 includes such hardware, software, or both: the hardware, software, or both provide one or more interfaces for communication between the computer system 1100 and one or more I / O devices. Where appropriate, the computer system 1100 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and the computer system 1100. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a tracking ball, a camera, another suitable I / O device, or a combination of two or more of these I / O devices. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1108 for these I / O devices. Where appropriate, the I / O interface 1108 may include one or more devices or software drivers that enable the processor 1102 to drive one or more of these I / O devices. Where appropriate, I / O interface 1108 may include one or more I / O interfaces 1108. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0203] In a particular embodiment, the communication interface 1110 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1100 and one or more other computer systems 1100 or one or more networks. By way of example and not limitation, the communication interface 1110 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other line-based network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network such as a WI-FI network. The present disclosure contemplates any suitable network and any suitable communication interface 1110 for the network. By way of example and not limitation, the computer system 1100 may communicate with a network such as an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more parts of the Internet, or a combination of two or more of these networks. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1100 can communicate with the following networks: a wireless PAN (WPAN) (e.g., a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these networks. Where appropriate, the computer system 1100 may include any suitable communication interface 1110 for any of these networks. Where appropriate, the communication interface 1110 may include one or more communication interfaces 1110. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0204] In particular embodiments, bus 1112 includes hardware, software, or both that couples the various components of computer system 1100 to each other. By way of example and not limitation, bus 1112 may include: Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, front-side bus (FSB), Hypertransport (HT) interconnect, Industry Standard Architecture (ISA) bus, InfiniBAND interconnect, low-pin-count (LPC) bus, memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCIe) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association local (VLB) bus, or another suitable bus, or a combination of two or more of these buses. Where appropriate, bus 1112 may include one or more buses 1112. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0205] In this document, where appropriate, one or more computer-readable non-transitory storage media may include: one or more semiconductor-based integrated circuits (integrated circuits, ICs) or other ICs (e.g., field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards (SECUREDIGITAL cards) or secure digital drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these storage media. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0206] other

[0207] As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Furthermore, "and" is joint and several, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

[0208] The scope of the present disclosure includes all changes, substitutions, variations, transformations and modifications to the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the various embodiments herein as including specific parts, elements, features, functions, operations, or steps, any embodiment in these embodiments may include any combination or arrangement of any of the parts, elements, features, functions, operations, or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, references to devices or systems that are suitable for, arranged to, capable of, configured to, enabled to, can be operated, or can be operated to perform a specific function, or components in a device or system in the appended claims include the device, system, component, regardless of whether the device, system, component, or the specific function is activated, turned on, or unlocked, as long as the device, system, or component is so suitable for, arranged to, capable of, configured to, enabled to, can be operated, or can be operated. Additionally, although this disclosure may describe or illustrate a particular embodiment as providing particular advantages, that particular embodiment may not provide these advantages, or may provide some or all of these advantages.< / domain> < / domain>

Claims

1. A method, comprising: Through one or more computing systems: receiving speech input from a first client system associated with a first user, the speech input having a plurality of acoustic features; determining, by one or more machine learning models, one or more emotions associated with the speech input based on one or more of the plurality of acoustic features; determining one or more facial features of a first extended reality XR avatar representing the first user based on the one or more emotions; as well as Instructions are sent to a second client system associated with a second user to render the first XR avatar representing the first user, wherein the first XR avatar is rendered using the determined one or more facial features.

2. The method according to claim 1, further comprising: A visual input corresponding to the speech input is received, wherein the visual input has a plurality of visual signals, and wherein determining the one or more emotions is further based on one or more of the plurality of visual signals.

3. The method according to claim 1, further comprising: A motion input corresponding to the speech input is received, wherein the motion input includes a plurality of motion signals, and wherein determining the one or more emotions is further based on one or more of the plurality of motion signals.

4. The method according to claim 1, wherein: The first user conducts a conversation with the second user in an XR environment including the first XR avatar.

5. The method according to claim 1, wherein: The one or more machine learning models include an acoustic model.

6. The method according to claim 5, further comprising: The speech input is converted into a mel-scale spectrogram by the acoustic model, wherein determining the one or more emotions is also based on the mel-scale spectrogram.

7. The method according to claim 1, wherein: Each of the one or more emotions is associated with one or more of a range, an intensity, or a confidence score.

8. The method according to claim 1, wherein: The one or more emotions are respectively associated with one or more confidence scores, and wherein the method further comprises: The one or more emotions are ranked based on confidence scores of the one or more emotions, wherein determining the one or more facial features is based on a highest ranked emotion.

9. The method according to claim 1, wherein: The one or more emotions are respectively associated with one or more confidence scores, and wherein the method further comprises: ranking the one or more emotions based on confidence scores of the one or more emotions; and selecting two or more of the ranked emotions based on the ranking of the one or more emotions; Wherein determining the one or more facial features is based on the selected two or more emotions.

10. The method according to claim 9, further comprising: A mixed emotion is generated based on the selected two or more emotions, wherein determining the one or more facial features is based on the mixed emotion.

11. The method according to claim 1, wherein: Determining the one or more facial features includes: identifying a plurality of facial landmarks of the first XR avatar; and The one or more emotions are mapped to one or more predefined expression shapes of the face of the first XR avatar based on the plurality of facial landmarks.

12. The method according to claim 1, further comprising: One or more lip movements are generated based on the determined one or more emotions using a lip synchronization model, wherein the first XR avatar is also rendered based on the one or more lip movements.

13. The method according to claim 1, further comprising: generating a recommendation for communication content for the first user based on the one or more determined emotions; as well as Instructions are sent to the first client system for presenting a recommendation for the communication content.

14. The method according to claim 1, further comprising: sending the determined one or more emotions to an assistant system executing on the first client system; as well as Receive from the assistant system performance results of one or more tasks determined based on the one or more emotions.

15. The method according to claim 1, further comprising: A transcription is generated for the speech input, wherein determining the one or more emotions is further based on analyzing the transcription.

16. The method according to claim 1, further comprising: A previous emotional state is accessed from a conversation state associated with the first user, wherein determining the one or more emotions is further based on the previous emotional state.

17. The method according to claim 16, wherein: The previous emotional state is determined based on one or more of: previous user input from the first user or previous user input from the second user.

18. The method of claim 1, further comprising: determining whether a wait time for rendering the first XR avatar is greater than a threshold time; as well as One or more audio signals corresponding to the voice input are transmitted to the second client system with a delay time, wherein the delay time is determined based on the waiting time.

19. One or more computer-readable non-transitory storage media containing software that, when executed, is operable to: receiving speech input from a first client system associated with a first user, the speech input having a plurality of acoustic features; determining, by one or more machine learning models, one or more emotions associated with the speech input based on one or more of the plurality of acoustic features; determining one or more facial features of a first extended reality XR avatar representing the first user based on the one or more emotions; as well as Instructions are sent to a second client system associated with a second user to render the first XR avatar representing the first user, wherein the first XR avatar is rendered using the determined one or more facial features.

20. A system, comprising: one or more processors; and a non-transitory memory coupled to the one or more processors and comprising instructions executable by the one or more processors, the one or more processors being operable, when executing the instructions, to: receiving speech input from a first client system associated with a first user, the speech input having a plurality of acoustic features; determining, by one or more machine learning models, one or more emotions associated with the speech input based on one or more of the plurality of acoustic features; determining one or more facial features of a first extended reality XR avatar representing the first user based on the one or more emotions; as well as Instructions are sent to a second client system associated with a second user to render the first XR avatar representing the first user, wherein the first XR avatar is rendered using the determined one or more facial features.

Images