Intelligent automated assistant in media environment

Abstract

To provide systems and processes for operating a digital assistant in a media environment.SOLUTION: In an exemplary embodiment, a user can interact with a digital assistant of a media device while content is displayed by the media device. In one approach, a plurality of exemplary natural language requests can be displayed in response to detecting a user input of a first input type. The plurality of exemplary natural language requests can be contextually-related to the displayed content. In another approach, a user request can be received in response to detecting a user input of a second input type. In particular, the user request can be satisfied while reducing disruption to user consumption of media content.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority from U.S. Provisional Patent Application No. 62 / 215,676, entitled "Intelligent Automated Assistant in a Media Environment," filed September 8, 2015, and U.S. Non-Provisional Patent Application No. 14 / 963,094, entitled "Intelligent Automated Assistant in a Media Environment," filed December 8, 2015, which applications are incorporated by reference herein in their entirety for all purposes.

[0002] This application is related to the following co-pending applications: U.S. Non-provisional Patent Application No. 14 / 963,089, entitled "Intelligent Automated Assistant for Media Search and Playback," filed December 8, 2015 (Attorney Docket No. 106842137900 (P27499US1)), U.S. Non-provisional Patent Application No. 14 / 498,503, entitled "Intelligent Automated Assistant for TV User Interactions," filed September 26, 2014 (Attorney Docket No. 106842065100 (P18133US1)), and U.S. Non-provisional Patent Application No. 14 / 498,391, entitled "Real-time Digital Assistant Knowledge Updates," filed September 26, 2014 (Attorney Docket No. 106842097900 (P22498US1)), which applications are incorporated herein by reference in their entirety for all purposes. [Technical field]

[0003] This application relates generally to intelligent automated assistants, and more particularly to intelligent automated assistants that operate within media environments. [Background technology]

[0004] Intelligent automated assistants (or digital assistants) can provide an intuitive interface between a user and an electronic device. These assistants may allow a user to interact with a device or system using natural language in spoken and / or textual form. For example, a user may access the services of an electronic device by providing spoken user input in natural language form to a virtual assistant associated with the electronic device. The virtual assistant can perform natural language processing on the spoken user input to infer the user's intent and actuate the user's intent into a task. The task can then be performed by executing one or more functions of the electronic device and, in some examples, can return associated output to the user in natural language form.

[0005] Integrating digital assistants into media environments (e.g., televisions, television set-top boxes, cable boxes, gaming devices, streaming media devices, digital video recorders, etc.) can be desirable to assist users with tasks related to media consumption. For example, digital assistants can be utilized to assist users in finding desirable media content to consume. However, user interactions with the digital assistant may include audio and visual output, which can interrupt the consumption of the media content. Therefore, it can be a challenge to integrate digital assistants into media environments in a manner that provides sufficient assistance to the user while minimizing interruptions to the consumption of the media content. Summary of the Invention

[0006] Systems and processes for operating a digital assistant in a media environment are disclosed. In some exemplary processes, user input can be detected while displaying content. The process can determine whether the user input corresponds to a first input format. In accordance with determining that the user input corresponds to the first input format, a plurality of exemplary natural language requests can be displayed. The plurality of exemplary natural language requests can be contextually related to the displayed content.

[0007] In some embodiments, in accordance with a determination that the user input does not correspond to the first input format, the process may determine whether the user input corresponds to a second input format. In accordance with a determination that the user input corresponds to the second input format, the audio data may be sampled. The process may determine whether the audio data comprises a user request. In accordance with a determination that the audio data comprises a user request, a task may be performed that at least partially satisfies the user request. In some examples, the task may include obtaining a result that at least partially satisfies the user request and displaying a second user interface having a portion of the result. The portion of the content may continue to be displayed while the second user interface is displayed, and a display area of ​​the second user interface may be smaller than a display area of ​​the portion of the content.

[0008] In some embodiments, a third user input can be detected while displaying the second user interface. In response to detecting the third user input, the display of the second user interface can be replaced with a display of a third user interface having a portion of the result. The third user interface can occupy at least a majority of the display area of ​​the display unit. Additionally, a second result can be obtained that at least partially satisfies the user request. The second result can be different from the result. The third user interface can include at least a portion of the second result.

[0009] In some embodiments, a fourth user input can be detected while displaying the third user interface. The fourth user input can indicate a direction. In response to detecting the fourth user input, focus of the third user interface can be switched from a first item in the third user interface to a second item in the third user interface. The second item can be positioned in the indicated direction relative to the first item.

[0010] In some embodiments, a fifth user input may be detected while displaying the third user interface. In response to detecting the fifth user input, a search field may be displayed. Additionally, a virtual keyboard interface may be displayed, and input received via the virtual keyboard interface may result in text entry into the search field. Further, in some embodiments, a selectable affordance may be caused to appear on a display of the second electronic device, and selection of the affordance allows text input to be received by the electronic device via a keyboard of the second electronic device.

[0011] In some embodiments, a sixth user input may be detected while displaying the third user interface. In response to detecting the sixth user input, second audio data containing a second user request may be sampled. The process may determine whether the second user request is a request to narrow the results of the user request. In accordance with determining that the second user request is a request to narrow the results of the user request, a subset of the results may be displayed via the third user interface. In accordance with determining that the second user request is not a request to narrow the results of the user request, third results that at least partially satisfy the second user request may be obtained. A portion of the third results may be displayed via the third user interface.

[0012] In some embodiments, the sampled audio data may include user utterances, and a user intent corresponding to the user utterance may be determined. The process may determine whether the user intent includes a request to adjust a state or setting of the application. In accordance with a determination that the user intent includes a request to adjust a state or setting of the application, the state or setting of the application may be adjusted to satisfy the user intent.

[0013] In some embodiments, in response to a determination that the user intent does not include a request to adjust a state or setting of an application on the electronic device, the process may determine whether the user intent is one of a plurality of predetermined request types. In response to a determination that the user intent is one of a plurality of predetermined request types, text-only results that at least partially satisfy the user intent may be displayed.

[0014] In some embodiments, in response to a determination that the user intent is not one of a plurality of predetermined request types, the process may determine whether the displayed content includes media content. In response to a determination that the displayed content includes media content, the process may further determine whether the media content can be paused. In response to a determination that the media content can be paused, the media content is paused, and a result that at least partially satisfies the user intent may be displayed via a third user interface. The third user interface may occupy at least half of the display area of ​​the display unit. In response to a determination that the media content cannot be paused, a result may be displayed via a second user interface while the media content is displayed. The display area occupied by the second user interface may be smaller than the display area occupied by the media content. Furthermore, in some embodiments, in response to a determination that the displayed content does not include media content, a result may be displayed via the third user interface. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram illustrating a system and environment for implementing a digital assistant, according to various embodiments.

[0016] [Figure 2] FIG. 1 is a block diagram illustrating a media system in accordance with various embodiments.

[0017] [Figure 3] FIG. 1 is a block diagram illustrating a user device according to various embodiments.

[0018] [Figure 4A] FIG. 1 is a block diagram illustrating a digital assistant system or a server portion thereof according to various embodiments.

[0019] [Figure 4B] 4B illustrates the functionality of the digital assistant shown in FIG. 4A, according to various embodiments.

[0020] [Figure 4C] 1 illustrates a portion of an ontology according to various embodiments.

[0021] [Figure 5A] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 5B] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 5C] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 5D] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 5E] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 5F] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 5G] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 5H] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 5I] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments.

[0022] In the figure numbers below, Figure 6O has been intentionally omitted to avoid any confusion between the capital letter O and the number 0 (zero). [Figure 6A]5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6B] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6C] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6D] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6E] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6F] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6G] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6H] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6I] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6J] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6K]5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6L] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6M] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6N] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6P] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments. [Figure 6Q] 5A-5I show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 5A-5I, according to various embodiments.

[0023] [Figure 7A] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 7B] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments. [Figure 7C] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments.

[0024] In the figure numbers below, Figure 8O has been intentionally omitted to avoid any confusion between the capital letter O and the number 0 (zero). [Figure 8A]7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8B] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8C] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8D] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8E] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8F] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8G] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8H] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8I] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8J] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8K]7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8L] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8M] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8N] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8P] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8Q] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8R] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8S] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8T] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8U] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8V]7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments. [Figure 8W] 7A-7C show screenshots displayed on a display unit by a media device at various stages of the process shown in FIGS. 7A-7C, according to various embodiments.

[0025] [Figure 9] 1 illustrates a process for operating a digital assistant in a media system according to various embodiments.

[0026] [Figure 10] FIG. 1 illustrates a functional block diagram of an electronic device configured to operate a digital assistant in a media system, according to various embodiments.

[0027] [Figure 11] FIG. 1 illustrates a functional block diagram of an electronic device configured to operate a digital assistant in a media system, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0028] In the following description of the embodiments, reference is made to the accompanying drawings, which show, by way of illustration, specific embodiments that may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the various embodiments.

[0029] This application relates to systems and processes for operating a digital assistant in a media environment. In one exemplary process, user input can be detected while displaying content. The process can determine whether the user input corresponds to a first input format. According to a determination that the user input corresponds to the first input format, a plurality of exemplary natural language requests can be displayed. The plurality of exemplary natural language requests can be contextually related to the displayed content. The contextually relevant exemplary natural language requests can be desirable to conveniently inform a user of the digital assistant's capabilities that are most relevant to the user's current usage situation on a media device. This can encourage the user to utilize the digital assistant's services and can improve the user's interaction experience with the digital assistant.

[0030] In some embodiments, in accordance with a determination that the user input does not correspond to the first input format, the process may determine whether the user input corresponds to a second input format. In accordance with a determination that the user input corresponds to the second input format, the audio data may be sampled. The process may determine whether the audio data comprises a user request. In accordance with a determination that the audio data comprises a user request, a task may be performed that at least partially satisfies the user request.

[0031] In some embodiments, the task performed may depend on the nature of the user request and the content being displayed while the user input of the second input form is detected. If the user request is a request to adjust a state or setting of an application on the electronic device (e.g., turning on subtitles for the displayed media content), the task may include adjusting the state or setting of the application. If the user request is one of a plurality of predetermined request types associated with text-only output (e.g., a request for the current time), the task may include displaying text that satisfies the user request. If the displayed content includes media content and the user request is to obtain and display results, the process may determine whether the media content can be paused. If it is determined that the media content can be paused, the media content may be paused and results that satisfy the user request may be displayed on an enlarged user interface (e.g., the third user interface 626 shown in FIG. 6H). If it is determined that the media content cannot be paused, results that satisfy the user request may be displayed on a reduced user interface (e.g., the second user interface 618 shown in FIG. 6G) while the media content continues to be displayed. The display area of ​​the second user interface can be smaller than the display area of ​​the media content. Furthermore, if the displayed content does not include media content, a result that satisfies the user request can be displayed on the enlarged user interface. By adjusting the output format according to the displayed content and the type of user request, the digital assistant can intelligently balance providing comprehensive assistance while minimizing interruptions to the user's consumption of media content, thereby resulting in an improved user experience. 1. System and Environment

[0032] 1 illustrates an exemplary system 100 for operating a digital assistant according to various embodiments. The terms “digital assistant,” “virtual assistant,” “intelligent automated assistant,” or “automated digital assistant” may refer to any information processing system that interprets natural language input in the form of speech and / or text to infer user intent and perform actions based on the inferred user intent. For example, to perform actions based on the inferred user intent, the system may perform one or more of the following: identify a task flow including steps and parameters designed to realize the inferred user intent, input specific requirements from the inferred user intent into the task flow, execute the task flow by invoking a program, method, service, application programming interface (API), or the like, and generate an output response to the user in an audible (e.g., spoken) and / or visual form.

[0033] Specifically, a digital assistant may have the ability to accept user requests, at least in part, in the form of natural language commands, requests, statements, narratives, and / or queries. Typically, a user request may seek either an informational answer or the performance of a task by the digital assistant. A satisfactory response to a user request may be the provision of the requested informational answer, the performance of the requested task, or a combination of both. For example, a user may ask a digital assistant a question such as, "What time is it in Paris?" The digital assistant may retrieve the requested information and respond, "It's 4:00 PM in Paris." A user may also request the performance of a task, for example, "Find movies starring Reese Witherspoon." In response, the digital assistant may execute the requested search query and display relevant movie titles for the user to select. During the performance of a requested task, the digital assistant may occasionally interact with the user in an ongoing dialog involving multiple information exchanges over an extended period of time. There are many other ways to interact with a digital assistant to request information or the performance of various tasks. In addition to providing text responses and taking programmed actions, the digital assistant can also provide other visual or audio forms of responses, such as words, alerts, music, images, videos, animations, etc. Additionally, as described herein, an example digital assistant can control the playback of media content (e.g., on a television set-top box) and cause the display of media content or other information on a display unit (e.g., a television).

[0034] 1, in some embodiments, a digital assistant can be implemented according to a client-server model. The digital assistant can include a client-side portion 102 (hereinafter, "DA client 102") that executes on a media device 104 and a server-side portion 106 (hereinafter, "DA server 106") that executes on a server system 108. Furthermore, in some embodiments, the client-side portion can also execute on a user device 122. The DA client 102 can communicate with the DA server 106 through one or more networks 110. The DA client 102 can provide client-side functionality such as user-responsive input and output processing and communication with the DA server 106. The DA server 106 can provide server-side functionality for any number of DA clients 102, each resident on a respective device (e.g., the media device 104 and the user device 122).

[0035] Media device 104 can be any suitable electronic device configured to manage and control media content. For example, media device 104 can include a television set-top box, such as a cable box device, a satellite box device, a video player device, a video streaming device, a digital video recorder, a gaming system, a DVD player, a Blu-ray Disc™ player, a combination of such devices, or the like. As shown in FIG. 1 , media device 104 can be part of a media system 128. In addition to media device 104, media system 128 can include a remote control device 124 and a display unit 126. Media device 104 can display media content on display unit 126. Display unit 126 can be any type of display, such as a television display, a monitor, a projector, or the like. In some embodiments, media device 104 can be connected to an audio system (e.g., an audio receiver) and speakers (not shown), which can be integrated with or separate from display unit 126. In other embodiments, display unit 126 and media device 104 may be integrated together in a single device, such as a smart television with advanced processing and network connectivity capabilities. In such embodiments, the functionality of media device 104 may be implemented as an application on the combined device.

[0036] In some embodiments, media device 104 can function as a media control center for multiple types and sources of media content. For example, media device 104 can facilitate user access to live television (e.g., over-the-air, satellite, or cable TV). Thus, media device 104 can include a cable tuner, a satellite tuner, or the like. In some embodiments, media device 104 can also record TV programs for later time-shifted viewing. In other embodiments, media device 104 can provide access to one or more streaming media services, such as cable-delivered on-demand TV programs, videos, and music (e.g., from various free, pay, and subscription-based streaming services) and Internet-delivered TV programs, videos, and music. In still other embodiments, media device 104 can facilitate the playback or display of media content from any other source, such as displaying photos from a mobile user device, playing videos from a coupled storage device, playing music from a coupled music player, or the like. Media device 104 can also include various other combinations of the media control mechanisms described herein, as desired. A detailed description of the media device 104 is provided below with reference to FIG.

[0037] The user device 122 can be any personal electronic device, such as a mobile phone (e.g., a smartphone), a tablet computer, a portable media player, a desktop computer, a laptop computer, a PDA, a wearable electronic device (e.g., digital glasses, a wristband, a watch, a brooch, an armband, etc.), or the like. A detailed description of the user device 122 is provided below with reference to FIG. 3.

[0038] In some examples, a user can interact with media device 104 through user device 122, remote control 124, or interface elements (e.g., buttons, microphone, camera, joystick, etc.) integrated with media device 104. For example, speech input including media-related queries or commands for a digital assistant can be received at user device 122 and / or remote control 124, and the speech input can be used to cause media-related tasks to be performed on media device 104. Similarly, tactile commands to control media on media device 104 can be received at user device 122 and / or remote control 124 (as well as from other devices not shown). Thus, various functions of media device 104 can be controlled in a variety of ways, providing users with multiple options for controlling media content from multiple devices.

[0039] Examples of communication network(s) 110 include a local area network (LAN) and a wide area network (WAN), such as the Internet. Communication network(s) 110 can be implemented using any known network protocol, including various wired or wireless protocols, such as, for example, Ethernet, Universal Serial Bus (USB), FIREWIRE, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0040] The DA server 106 may include a client-facing input / output (I / O) interface 112, one or more processing modules 114, data and models 116, and an I / O interface 118 to external services. The client-facing I / O interface 112 may facilitate client-facing input and output processing for the DA server 106. The one or more processing modules 114 may utilize the data and models 116 to process speech input and determine user intent based on natural language input. Furthermore, the one or more processing modules 114 may perform task execution based on the inferred user intent. In some embodiments, the DA server 106 may communicate with external services 120, such as a telephone service, a calendar service, an information service, a messaging service, a navigation service, a television program service, a streaming media service, a media search service, and the like, through network(s) 110 to complete tasks or obtain information. The I / O interface 118 to external services may facilitate such communication.

[0041] The server system 108 may be implemented on one or more stand-alone data processing devices or on a distributed network of computers. In some embodiments, the server system 108 may also utilize various virtual devices and / or the services of third-party service providers (e.g., third-party cloud service providers) to provide the underlying computing and / or infrastructure resources of the server system 108.

[0042] 1 can include both a client-side portion (e.g., DA client 102) and a server-side portion (e.g., DA server 106), in some embodiments, the functionality of the digital assistant can be implemented as a standalone application installed on a user device or a media device. In addition, the distribution of functionality between the client and server portions of the digital assistant can vary depending on the implementation. For example, in some embodiments, the DA client running on user device 122 or media device 104 can be a thin client that provides only user-facing input and output processing functionality and delegates all other functions of the digital assistant to a back-end server. 2. Media System

[0043] 2 shows a block diagram of a media system 128 according to various embodiments. The media system 128 may include a media device 104 communicatively coupled to a display unit 126, a remote control 124, and a speaker 268. The media device 104 may receive user input via the remote control 124. Media content from the media device 104 may be displayed on the display unit 126.

[0044] 2, media device 104 may include a memory interface 202, one or more processors 204, and a peripherals interface 206. The various components within media device 104 may be coupled to one another by one or more communication buses or signal lines. Media device 104 may further include various subsystems and peripheral devices coupled to peripherals interface 206. The subsystems and peripheral devices may collect information and / or facilitate various functionality of media device 104.

[0045] For example, media device 104 may include a communications subsystem 224. Communications capabilities may be facilitated through one or more wired and / or wireless communications subsystems 224, which may include various communications ports, radio frequency receivers and transmitters, and / or optical (e.g., infrared) receivers and transmitters.

[0046] In some embodiments, media device 104 may further include an I / O subsystem 240 coupled to peripherals interface 206. I / O subsystem 240 may include an audio / video output controller 270. Audio / video output controller 270 may be coupled to display unit 126 and speakers 268 or may provide audio and video output in another manner (e.g., via an audio / video port, wireless transmission, etc.). I / O subsystem 240 may further include a remote controller 242. Remote controller 242 may be communicatively coupled to remote control device 124 (e.g., via a wired connection, Bluetooth, Wi-Fi, etc.).

[0047] Remote control device 124 may include a microphone 272 for capturing audio data (e.g., speech input from a user), button(s) 274 for capturing tactile input, and a transceiver 276 for facilitating communication with media device 104 via remote controller 242. Additionally, remote control device 124 may include a touch-sensitive surface 278, a sensor, or a set of sensors that accept input from a user based on haptic and / or tactile contact. Touch-sensitive surface 278 and remote controller 242 can detect contacts (as well as any movement or disruption of contact) on touch-sensitive surface 278 and translate the detected contacts (e.g., gestures, contact movements, etc.) into interactions with user interface objects (e.g., one or more softkeys, icons, web pages, or images) displayed on display unit 126. In some examples, remote control device 124 may also include other input mechanisms, such as a keyboard, joystick, or the like. In some examples, remote control device 124 may further include an output mechanism, such as a light, a display, a speaker, or the like. Input received at the remote control device 124 (e.g., user speech, button presses, touch movements, etc.) can be communicated to the media device 104 via the remote control device 124. The I / O subsystem 240 can also include other input controller(s) 244. The other input controller(s) 244 can be coupled to other input / control devices 248, such as one or more buttons, rocker switches, thumbwheels, infrared ports, USB ports, and / or pointer devices such as styluses.

[0048] In some embodiments, media device 104 may further include a memory interface 202 coupled to memory 250. Memory 250 may include any electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, a portable computer diskette (magnetic), random access memory (RAM) (magnetic), read-only memory (ROM) (magnetic), erasable programmable read-only memory (EPROM) (magnetic), a portable optical disk such as a CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW, or flash memory such as a compact flash card, a secure digital card, a USB memory device, a memory stick, and the like. In some embodiments, the non-transitory computer-readable storage medium of memory 250 may be used to store instructions (e.g., to perform part or all of the various processes described herein) for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute those instructions in an instruction execution system, apparatus, or device. In other embodiments, the instructions (e.g., to perform part or all of the various processes described herein) may be stored on a non-transitory computer-readable storage medium of server system 108 or distributed between the non-transitory computer-readable storage medium of memory 250 and the non-transitory computer-readable storage medium of server system 108. In the context of this specification, a "non-transitory computer-readable storage medium" may be any medium that can contain or store a program for use by or with an instruction execution system, apparatus, or device.

[0049] In some embodiments, memory 250 can store an operating system 252, a communications module 254, a graphical user interface (GUI) module 256, an on-device media module 258, an off-device media module 260, and an application module 262. Operating system 252 can include instructions for handling basic system services and for performing hardware-dependent tasks. Communications module 254 may facilitate communication with one or more additional devices, one or more computers, and / or one or more servers. Graphical user interface module 256 can facilitate graphical user interface processing. On-device media module 258 can facilitate storage and playback of media content stored locally on media device 104. Off-device media module 260 can facilitate streaming or downloading of media content obtained from an external source (e.g., on a remote server, on user device 122, etc.). Additionally, off-device media module 260 can facilitate receiving broadcast and cable content (e.g., channel tuning). The application modules 262 may facilitate various functionalities of media-related applications, such as web browsing, media processing, gaming, and / or other processes and functions.

[0050] As described herein, memory 250 can also store client-side digital assistant instructions (e.g., in digital assistant client module 264), as well as various user data 266 (e.g., user-specific vocabulary data, preference data, and / or other data such as the user's media search history, media watchlist, recently viewed list, favorite media items, etc.), for example, to provide client-side functionality of the digital assistant. User data 266 can also be used in performing speech recognition to assist the digital assistant or for any other application.

[0051] In various embodiments, digital assistant client module 264 can be capable of accepting audio input (e.g., speech input), text input, touch input, and / or gesture input through various user interfaces (e.g., I / O subsystem 240 or the like) of media device 104. Digital assistant client module 264 can also be capable of providing output in audio (e.g., speech output), visual, and / or tactile form. For example, output can be provided as voice, sound, alerts, text messages, menus, graphics, video, animations, vibrations, and / or combinations of two or more of the above. In operation, digital assistant client module 264 can communicate with a digital assistant server (e.g., DA server 106) using communication subsystem 224.

[0052] In some embodiments, digital assistant client module 264 can gather additional information related to media device 104 and from the media device 104's surrounding environment using various subsystems and peripheral devices to establish a context associated with the user, the current user interaction, and / or the current user input. Such context can also include information from other devices, such as from user device 122. In some embodiments, digital assistant client module 264 can provide the context information, or a subset thereof, along with the user input to the digital assistant server to help infer the user's intent. The digital assistant can also use the context information to determine how to prepare and deliver output to the user. The context information can be further used by media device 104 or server system 108 to assist in accurate speech recognition.

[0053] In some embodiments, contextual information associated with a user input can include sensor information such as lighting, ambient noise, ambient temperature, distance to another object, and the like. Contextual information can further include information associated with the physical state of the media device 104 (e.g., device location, device temperature, power level, etc.) or the software state of the media device 104 (e.g., running processes, installed applications, past and present network activity, background services, error logs, resource usage, etc.). Contextual information can further include information received from the user (e.g., speech input), information requested by the user, and information presented to the user (e.g., information currently or previously displayed by the media device). Contextual information can further include information associated with the state of connected devices or other devices associated with the user (e.g., content displayed on the user device 122, playable content on the user device 122, etc.). Any of these types of contextual information can be provided to the DA server 106 (or used on the media device 104 itself) as contextual information associated with the user input.

[0054] In some embodiments, digital assistant client module 264 can selectively provide information stored on media device 104 (e.g., user data 266) in response to a request from DA server 106. Additionally, or alternatively, the information can be used on media device 104 itself in performing speech recognition and / or digital assistant functions. Digital assistant client module 264 can also elicit additional input from the user via a natural language dialog or other user interface in response to a request by DA server 106. Digital assistant client module 264 can pass the additional input to DA server 106 to assist DA server 106 in intent inference and / or achieving the user's intent expressed in the user request.

[0055] In various embodiments, memory 250 may include additional or fewer instructions. Furthermore, various functions of media device 104 may be implemented in the form of hardware and / or firmware, including in the form of one or more signal processing circuits and / or application specific integrated circuits. 3. User Devices

[0056] 3 illustrates a block diagram of an exemplary user device 122 in accordance with various embodiments. As shown, the user device 122 may include a memory interface 302, one or more processors 304, and a peripherals interface 306. The various components within the user device 122 may be coupled to one another by one or more communication buses or signal lines. The user device 122 may further include various sensors, subsystems, and peripheral devices coupled to the peripherals interface 306. The sensors, subsystems, and peripheral devices may collect information and / or facilitate various functionalities of the user device 122.

[0057] For example, the user device 122 may include a motion sensor 310, a light sensor 312, and a proximity sensor 314 coupled to the peripherals interface 306 to facilitate orientation, light, and proximity sensing functions. One or more other sensors 316, such as a positioning system (e.g., a GPS receiver), a temperature sensor, a biometric sensor, a gyroscope, a compass, an accelerometer, and the like, may also be connected to the peripherals interface 306 to facilitate related functions.

[0058] In some embodiments, a camera subsystem 320 and an optical sensor 322 may be utilized to facilitate camera functions such as taking pictures and recording video clips. Communication functions may be facilitated through one or more wired and / or wireless communication subsystems 324, which may include various communication ports, radio frequency receivers and transmitters, and / or optical (e.g., infrared) receivers and transmitters. An audio subsystem 326 may be coupled to a speaker 328 and a microphone 330 to facilitate voice-enabled functions such as voice recognition, voice duplication, digital recording, and telephone functions.

[0059] In some embodiments, user device 122 may further include an I / O subsystem 340 coupled to peripherals interface 306. I / O subsystem 340 may include a touchscreen controller 342 and / or other input controller(s) 344. Touchscreen controller 342 may be coupled to a touchscreen 346. Touchscreen 346 and touchscreen controller 342 may detect contact and its movement or disruption using any of a number of touch-sensing technologies, such as, for example, capacitive, resistive, infrared, and surface acoustic wave technologies, proximity sensor arrays, and the like. Other input controller(s) 344 may be coupled to other input / control devices 348, such as one or more buttons, rocker switches, thumbwheels, infrared ports, USB ports, and / or pointer devices such as styluses.

[0060] In some embodiments, user device 122 may further include a memory interface 302 coupled to memory 350. Memory 350 may include any electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, a portable computer diskette (magnetic), random access memory (RAM) (magnetic), read-only memory (ROM) (magnetic), erasable programmable read-only memory (EPROM) (magnetic), a portable optical disk such as a CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW, or flash memory such as a compact flash card, a secure digital card, a USB memory device, a memory stick, and the like. In some embodiments, the non-transitory computer-readable storage medium of memory 350 may be used to store instructions (e.g., to perform part or all of the various processes described herein) for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute those instructions in an instruction execution system, apparatus, or device. In other embodiments, the instructions (e.g., to perform part or all of the various processes described herein) may be stored on a non-transitory computer-readable storage medium of server system 108 or distributed between the non-transitory computer-readable storage medium of memory 350 and the non-transitory computer-readable storage medium of server system 108. In the context of this specification, a "non-transitory computer-readable storage medium" may be any medium that can contain or store a program for use by or with an instruction execution system, apparatus, or device.

[0061] In some embodiments, memory 350 can store an operating system 352, a communications module 354, a graphical user interface (GUI) module 356, a sensor processing module 358, a telephony module 360, and an application module 362. Operating system 352 can include instructions for handling basic system services and instructions for performing hardware-dependent tasks. Communications module 354 can facilitate communication with one or more additional devices, one or more computers, and / or one or more servers. Graphical user interface module 356 can facilitate graphical user interface processing. Sensor processing module 358 can facilitate sensor-related processes and functions. Telephony module 360 ​​can facilitate telephony-related processes and functions. Application module 362 can facilitate various functionality of user applications, such as electronic messaging, web browsing, media processing, navigation, imaging, and / or other processes and functions.

[0062] As described herein, memory 350 can also store client-side digital assistant instructions (e.g., in digital assistant client module 364), as well as various user data 366 (e.g., user-specific vocabulary data, preference data, and / or other data such as the user's electronic address book, to-do list, shopping list, television program favorites, etc.), for example, to provide the client-side functionality of the digital assistant. User data 366 can also be used in performing speech recognition to assist the digital assistant or for any other application. Digital assistant client module 364 and user data 366 can be similar to or identical to digital assistant client module 264 and user data 266, respectively, as described above with reference to FIG. 2.

[0063] In various embodiments, memory 350 may include additional or fewer instructions. Furthermore, various functions of user device 122 may be implemented in hardware and / or firmware, including in the form of one or more signal processing circuits and / or application specific integrated circuits.

[0064] In some examples, user device 122 can be configured to control aspects of media device 104. For example, user device 122 can function as a remote control (e.g., remote control device 124). User input received via user device 122 can be transmitted to media device 104 (e.g., using a communications subsystem) to cause corresponding actions to be performed by media device 104. Additionally, user device 122 can be configured to receive instructions from media device 104. For example, media device 104 can hand off tasks to user device 122 to execute and display objects (e.g., selectable affordances) on user device 122.

[0065] It should be understood that system 100 and media system 128 are not limited to the components and configurations shown in Figures 1 and 2, and user device 122, media device 104, and remote control device 124 are likewise not limited to the components and configurations shown in Figures 2 and 3. System 100, media system 128, user device 122, media device 104, and remote control device 124 can all include fewer or other components in multiple configurations according to various embodiments. 4. Digital Assistant System

[0066] FIG. 4A shows a block diagram of a digital assistant system 400 according to various embodiments. In some embodiments, digital assistant system 400 can be implemented on a stand-alone computer system. In some embodiments, digital assistant system 400 can be distributed across multiple computers. In some embodiments, some of the modules and functionality of the digital assistant can be divided into a server portion and a client portion. In this case, the client portion resides on one or more user devices (e.g., device 104 or 122) and communicates with the server portion (e.g., server system 108) over one or more networks, for example, as shown in FIG. 1 . In some embodiments, digital assistant system 400 can be one implementation of server system 108 (and / or DA server 106) shown in FIG. 1 . Note that digital assistant system 400 is merely one example of a digital assistant system, and digital assistant system 400 can have more or fewer components than shown, can combine two or more components, or can have a different configuration or arrangement of components. The various components shown in FIG. 4A may be implemented in the form of hardware, including one or more signal processing circuits and / or application specific integrated circuits, software instructions executed by one or more processors, firmware, or a combination thereof.

[0067] The digital assistant system 400 can include memory 402, one or more processors 404, an I / O interface 406, and a network communication interface 408. These components can communicate with each other through one or more communication buses or signal lines 410.

[0068] In some embodiments, memory 402 may include non-transitory computer-readable media, such as high-speed random access memory and / or non-volatile computer-readable storage media (e.g., one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0069] In some embodiments, I / O interface 406 can couple I / O devices 416 of digital assistant system 400, such as a display, keyboard, touchscreen, and microphone, to user interface module 422. I / O interface 406 can cooperate with user interface module 422 to receive user inputs (e.g., voice input, keyboard input, touch input, etc.) and process them accordingly. In some embodiments, for example, when the digital assistant is implemented on a stand-alone user device, digital assistant system 400 can include any of the components and I / O communication interfaces described with respect to device 104 or 122 in FIG. 2 or FIG. 3, respectively. In some embodiments, digital assistant system 400 can represent the server portion of a digital assistant implementation and can interact with a user through a client-side portion residing on a client device (e.g., user device 104 or 122).

[0070] In some embodiments, network communication interface 408 can include wired communication port(s) 412 and / or wireless transmission and reception circuitry 414. The wired communication port(s) can receive and transmit communication signals via one or more wired interfaces, such as Ethernet, Universal Serial Bus (USB), FIREWIRE®, etc. The wireless circuitry 414 can receive and transmit RF and / or optical signals from / to communication networks and other communication devices. Wireless communication can use any of a number of communication standards, protocols, and technologies, such as GSM®, EDGE, CDMA, TDMA, Bluetooth®, Wi-Fi®, VoIP, Wi-MAX®, or any other suitable communication protocol. Network communication interface 408 can enable communication between digital assistant system 400 and devices using networks, such as the Internet, intranets, and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs).

[0071] In some embodiments, memory 402, or the computer-readable storage medium of memory 402, can store programs, modules, instructions, and data structures, including all or a subset of operating system 418, communications module 420, user interface module 422, one or more applications 424, and digital assistant module 426. In particular, memory 402, or the computer-readable storage medium of memory 402, can store instructions for performing process 800, described below. One or more processors 404 can execute these programs, modules, and instructions and read / write from / to the data structures.

[0072] Operating system 418 (e.g., Darwin®, RTXC®, LINUX®, UNIX®, iOS, OS X®, WINDOWS®, or an embedded operating system such as VxWorks) may include various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware, firmware, and software components.

[0073] The communications module 420 can facilitate communications between the digital assistant system 400 and other devices through the network communications interface 408. For example, the communications module 420 can communicate with a communications subsystem (e.g., 224, 324) of an electronic device (e.g., 104, 122). The communications module 420 can also include various components for processing data received by the wireless circuitry 414 and / or the wired communications port 412.

[0074] The user interface module 422 can receive commands and / or input from a user (e.g., from a keyboard, touchscreen, pointing device, controller, and / or microphone) via the I / O interface 406 and generate user interface objects on the display. The user interface module 422 can also prepare and deliver output (e.g., speech, sounds, animations, text, icons, vibrations, haptic feedback, lights, etc.) to the user via the I / O interface 406 (e.g., through a display, audio channel, speaker, touchpad, etc.).

[0075] Applications 424 can include programs and / or modules configured to be executed by one or more processors 404. For example, if digital assistant system 400 is implemented on a standalone user device, applications 424 can include user applications such as games, calendar applications, navigation applications, or email applications. If digital assistant system 400 is implemented on a server, applications 424 can include, for example, resource management applications, diagnostic applications, or scheduling applications.

[0076] Memory 402 can also store digital assistant module 426 (or the server portion of the digital assistant). In some examples, digital assistant module 426 can include the following submodules, or a subset or superset thereof: I / O processing module 428, speech-to-text (STT) processing module 430, natural language processing module 432, dialog flow processing module 434, task flow processing module 436, service processing module 438, and speech synthesis module 440. Each of these modules can have access to one or more of the following systems or data and models of digital assistant module 426, or a subset or superset thereof: ontology 460, vocabulary index 444, user data 448, task flow model 454, service model 456, and automatic speech recognition (ASR) system 431.

[0077] In some examples, using the processing modules, data, and models implemented in digital assistant module 426, the digital assistant can perform at least some of the following: convert speech input to text, identify the user's intent as expressed in natural language input received from the user, actively elicit and obtain the information necessary to fully infer the user's intent (e.g., by disambiguating words, games, intents, etc.), determine a task flow to achieve the inferred intent, and execute the task flow to achieve the inferred intent.

[0078] In some examples, as shown in FIG. 4B , I / O processing module 428 can interact with a user through I / O device 416 in FIG. 4A or with an electronic device (e.g., device 104 or 122) through network communication interface 408 in FIG. 4A to obtain user input (e.g., speech input) and to provide responses to the user input (e.g., as speech output). I / O processing module 428 can optionally obtain contextual information associated with the user input from the electronic device along with or immediately after receiving the user input. The contextual information can include user-specific data, vocabulary, and / or preferences related to the user input. In some examples, the contextual information also includes information about the software and hardware state of the electronic device at the time the user request is received and / or the user's ambient environment at the time the user request is received. In some examples, I / O processing module 428 can also send follow-up questions to the user regarding the user request and receive answers from the user. When a user request is received by the I / O processing module 428, and the user request may include speech input, the I / O processing module 428 may forward the speech input to the STT processing module 430 (or speech recognizer) for speech-to-text conversion.

[0079] The STT processing module 430 may include one or more ASR systems (e.g., ASR system 431). The one or more ASR systems may process speech input received through the I / O processing module 428 and generate recognition results. Each ASR system may include a front-end speech preprocessor. The front-end speech preprocessor may extract representative features from the speech input. For example, the front-end speech preprocessor may perform a Fourier transform on the speech input to extract spectral features that characterize the speech input as a series of representative multi-dimensional vectors. Furthermore, each ASR system may include one or more speech recognition models (e.g., acoustic models and / or language models) and may implement one or more speech recognition engines. Examples of speech recognition models include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines include a dynamic time warping-based engine and a weighted finite-state transducer (WFST)-based engine. One or more speech recognition models and one or more speech recognition engines can be used to process the extracted representative features of the front-end speech preprocessor to generate intermediate recognition results (e.g., phonemes, phoneme strings, and subwords) and, ultimately, text recognition results (e.g., words, word strings, or strings of tokens). In some embodiments, the speech input can be processed at least partially by a third-party service or on the electronic device (e.g., device 104 or 122) to generate the recognition results. Once the STT processing module 430 generates a recognition result containing a text string (e.g., a word, a string of words, or a string of tokens), the recognition result can be passed to the natural language processing module 432 for intent inference.

[0080] In some embodiments, one or more language models of one or more ASR systems can be configured to be biased toward media-related results. In one embodiment, the one or more language models can be trained using a corpus of media-related text. In another embodiment, the ASR system can be configured to prioritize media-related recognition results. In some embodiments, the one or more ASR systems can include a static language model and a dynamic language model. The static language model can be trained using a general corpus of text, while the dynamic language model can be trained using user-specific text. For example, text corresponding to previous speech input received from a user can be used to generate the dynamic language model. In some embodiments, the one or more ASR systems can be configured to generate recognition results based on the static language model and / or the dynamic language model. Furthermore, in some embodiments, the one or more ASR systems can be configured to prioritize recognition results corresponding to more recently received previous speech input.

[0081] Further details regarding speech-to-text processing are provided in U.S. Utility Patent Application No. 13 / 236,942, filed September 20, 2011, for "Consolidating Speech Recognition Results," the entire disclosure of which is incorporated herein by reference.

[0082] In some implementations, the STT processing module 430 may include and / or have access to a vocabulary of recognizable words via the speech-to-speech transcription module 431. Each vocabulary word may be associated with one or more candidate pronunciations of the word represented in the speech-recognition phonetic alphabet. In particular, the vocabulary of recognizable words may include words associated with multiple candidate pronunciations. For example, the vocabulary may include: The vocabulary words may include the word "tomato" associated with a candidate pronunciation of TIFF2025060631000002.tif6128. Additionally, vocabulary words may be associated with custom candidate pronunciations based on previous speech input from a user. Such custom candidate pronunciations may be stored within the STT processing module 430 and may be associated with a particular user via that user's profile on the device. In some examples, candidate pronunciations for a word may be determined based on the spelling of the word and one or more linguistic and / or phonetic rules. In some examples, candidate pronunciations may be generated manually, for example, based on known standard pronunciations.

[0083] In some embodiments, the candidate pronunciations may be ranked based on the popularity of the candidate pronunciations. For example, TIFF2025060631000003.tif6128 because the former is a more commonly used pronunciation (e.g., among all users, for users in a particular geographic region, or for any other suitable subset of users). In some examples, the candidate pronunciations may be ranked based on whether the candidate pronunciation is a custom candidate pronunciation associated with the user. For example, a custom candidate pronunciation may be ranked higher than a standard candidate pronunciation. This may be useful for recognizing proper nouns that have unique pronunciations that deviate from standard pronunciations. In some examples, the candidate pronunciations may be associated with one or more speech characteristics, such as place of origin, nationality, or ethnicity. For example, the candidate pronunciation TIFF2025060631000004.tif6128 can be associated with the United States, for which the candidate pronunciation TIFF2025060631000005.tif6128 may be associated with the United Kingdom. Further, the ranking of the candidate pronunciations may be based on one or more characteristics of the user (e.g., place of origin, nationality, ethnicity, etc.) stored in the user's profile on the device. For example, it may be determined from the user's profile that the user is associated with the United States. Based on the user's association with the United States, the candidate pronunciations TIFF2025060631000006.tif6128 (associated with the US) TIFF2025060631000007.tif6128 (associated with the United Kingdom). In some examples, one of the ranked candidate pronunciations may be selected as the predicted pronunciation (e.g., the most likely pronunciation).

[0084] When speech input is received, the STT processing module 430 can be used to determine the phonemes that correspond to the speech input (e.g., using an acoustic model) and then attempt to determine words that match the phonemes (e.g., using a language model). For example, the STT processing module 430 can first determine a sequence of phonemes that correspond to a portion of the speech input. If TIFF2025060631000008.tif6128 can be located, then it can determine, based on the lexical index 444, that this string corresponds to the word "tomato."

[0085] In some implementations, the STT processing module 430 may use proximity matching techniques to determine words within an utterance. Thus, for example, the STT processing module 430 may select a phoneme sequence even if that particular phoneme sequence is not one of the candidate phoneme sequences for that word. It can be determined that TIFF2025060631000009.tif6128 corresponds to the word "tomato."

[0086] The digital assistant's natural language processing module 432 ("natural language processor") can take the string of words or tokens ("token string") generated by the STT processing module 430 and attempt to associate the token string with one or more "actionable intents" recognized by the digital assistant. An "actionable intent" can represent a task that can be performed by the digital assistant and can have an associated task flow implemented in task flow model 454. The associated task flow can be a series of programmed actions and steps that the digital assistant takes to perform the task. The scope of the digital assistant's capabilities can depend on the number and type of task flows implemented and stored in task flow model 454, or in other words, the number and type of "actionable intents" that the digital assistant recognizes. However, the effectiveness of the digital assistant can also depend on the assistant's ability to infer the correct "actionable intent(s)" from a user request expressed in natural language.

[0087] In some examples, in addition to the string of words or tokens obtained from STT processing module 430, natural language processing module 432 can also receive context information associated with the user request, for example, from I / O processing module 428. Natural language processing module 432 can optionally use the context information to clarify, complement, and / or further clarify information contained in the string of tokens received from STT processing module 430. Context information can include, for example, user preferences, hardware and / or software state of the user device, sensor information collected before, during, or immediately after the user request, previous interactions (e.g., dialogs) between the digital assistant and the user, and the like. As described herein, context information can be dynamic and can change with time, location, dialog content, and other factors.

[0088] In some embodiments, natural language processing can be based on, for example, ontology 460. Ontology 460 is a hierarchical structure that includes multiple nodes, and each node can represent either an "actionable intention" or an "attribute" that is related to one or more of the "actionable intentions" or other "attributes." As described above, an "actionable intention" can represent a task that a digital assistant has the ability to perform; that is, it can be "actionable" or targeted for execution. An "attribute" can represent a parameter associated with a sub-aspect of an actionable intention or another attribute. A link between an actionable intention node and an attribute node in ontology 460 can define how the parameter represented by the attribute node relates to the task represented by the actionable intention node.

[0089] In some embodiments, ontology 460 can be composed of actionable intent nodes and attribute nodes. Within ontology 460, each actionable intent node can be linked to one or more attribute nodes, either directly or through one or more intermediate attribute nodes. Similarly, each attribute node can be linked to one or more actionable intent nodes, either directly or through one or more intermediate attribute nodes. For example, as shown in FIG. 4C , ontology 460 can include a “media” node (i.e., an actionable intent node). The attribute nodes “actor(s),” “media genre,” and “media title” can each be directly linked to an actionable intent node (i.e., a “media search” node). In addition, the attribute nodes “name,” “age,” “Ullmer scale ranking,” and “nationality” can be subordinate nodes of the attribute node “actor.”

[0090] 4C , ontology 460 may also include a "weather" node (i.e., another actionable intent node). Attribute nodes "date / time" and "location" may each be linked to the "weather search" node. It should be appreciated that in some embodiments, one or more attribute nodes may be associated with two or more actionable intents. In these embodiments, one or more attribute nodes may be linked to respective nodes corresponding to two or more actionable intents in ontology 460.

[0091] An actionable intent node, along with its linked concept nodes, can be described as a "domain." In this description, each domain can be associated with a respective actionable intent and can refer to a group of nodes (and the relationships between the nodes) associated with a particular actionable intent. For example, ontology 460 shown in FIG. 4C can include an example media domain 462 and an example weather domain 464 within ontology 460. Media domain 462 can include the actionable intent node "media search" and the attribute nodes "actor(s)," "media genre," and "media title." Weather domain 464 can include the actionable intent node "weather search" and the attribute nodes "location" and "date / time." In some embodiments, ontology 460 can be composed of many domains. Each domain can share one or more attribute nodes with one or more other domains.

[0092] 4C shows two example domains within ontology 460, other domains may include, for example, "athletes," "stock prices," "directions," "media preferences," "sports teams," and "time," "tell a joke," etc. The "athletes" domain may be associated with an actionable intent node of "search athlete information," and may further include attribute nodes such as "athlete name," "athlete team," and "athlete statistics."

[0093] In some examples, ontology 460 can include all domains (and therefore actionable intents) that the digital assistant has the ability to understand and act upon. In some examples, ontology 460 can be modified, such as by adding or removing domains or entire nodes, or by changing the relationships between nodes in ontology 460.

[0094] In some embodiments, each node in ontology 460 can be associated with a set of words and / or phrases related to the attribute or actionable intent represented by that node. The respective set of words and / or phrases associated with each node can be a so-called "vocabulary" associated with that node. The respective set of words and / or phrases associated with each node can be stored in vocabulary index 444 in association with the attribute or actionable intent represented by the node. For example, returning to FIG. 4C , the vocabulary associated with a node for the attribute of "actor" can include words such as "A-list," "Reese Witherspoon," "Arnold Schwarzenegger," "Brad Pitt," etc. As another example, the vocabulary associated with a node for the actionable intent of "weather search" can include words and phrases such as "weather," "what's it like in," "forecast," etc. Vocabulary index 444 can optionally include words and phrases in different languages.

[0095] Natural language processing module 432 can receive a token sequence (e.g., a text string) from STT processing module 430 and determine which nodes are implied by words in the token sequence. In some embodiments, if a word or phrase in the token sequence is found to be associated with one or more nodes in ontology 460 (via vocabulary index 444), the word or phrase can “trigger” or “activate” those nodes. Based on the amount and / or relative importance of the activated nodes, natural language processing module 432 can select one of the possible intents as the task the user intends the digital assistant to perform. In some embodiments, the domain with the most “triggered” nodes can be selected. In some embodiments, the domain with the highest confidence value (e.g., based on the relative importance of its various triggered nodes) can be selected. In some embodiments, the domain can be selected based on a combination of the number and importance of triggered nodes. In some embodiments, additional factors are also considered when selecting a node, such as whether the digital assistant has previously accurately interpreted a similar request from the user.

[0096] User data 448 may include user-specific information, such as user-specific vocabulary, user preferences, user addresses, the user's default and secondary languages, the user's contact list, and other short-term or long-term information about each user. In some embodiments, natural language processing module 432 may use the user-specific information to supplement information contained in the user input to further clarify the user's intent. For example, for a user request "What's the weather like this week?" natural language processing module 432 may access user data 448 to determine where the user is located, instead of asking the user to explicitly provide such information in their request.

[0097] Additional details of ontology searching based on token strings are described in U.S. Utility Patent Application No. 12 / 341,743, filed December 22, 2008, for "Method and Apparatus for Searching Using an Active Ontology," the entire disclosure of which is incorporated herein by reference.

[0098] In some examples, once the natural language processing module 432 identifies an actionable intent (or domain) based on a user request, the natural language processing module 432 can generate a structured query to represent the identified actionable intent. In some examples, the structured query can include parameters for one or more nodes in the domain for the actionable intent, where at least a portion of the parameters are populated with specific information and requirements specified in the user request. For example, a user may say, "Find other seasons of this TV series." In this case, the natural language processing module 432 can accurately identify the actionable intent as "media search" based on the user input. According to the ontology, a structured query for the "media" domain can include parameters such as {media actor}, {media genre}, {media title}, and the like. In some examples, based on the speech input and text derived from the speech input using the STT processing module 430, the natural language processing module 432 can generate a partial structured query for the restaurant reservation domain. In this case, the partial structured query includes the parameter {media genre="TV series"}. However, in this example, the user's utterance does not contain enough information to complete a structured query associated with the domain. Therefore, other required parameters, such as {media title}, may not be specified in the structured query based on currently available information. In some implementations, the natural language processing module 432 can populate some parameters of the structured query with received context information. For example, the TV series "Mad Men" may be currently playing on the media device. Based on this context information, the natural language processing module 432 can populate the {media title} parameter in the structured query with "Mad Men."

[0099] In some examples, natural language processing module 432 can pass the generated structured query (including any completed parameters) to task flow processing module 436 ("task flow processor"). Task flow processing module 436 can be configured to receive the structured query from natural language processing module 432, complete the structured query as needed, and perform the actions required to "complete" the user's ultimate request. In some examples, the various steps required to complete these tasks can be provided within task flow model 454. In some examples, task flow model 454 can include steps for obtaining additional information from the user and a task flow for performing actions associated with the actionable intent.

[0100] As described above, to complete the structured query, task flow processing module 436 may need to initiate additional dialogue with the user to obtain additional information and / or disambiguate potentially ambiguous utterances. If such dialogue is necessary, task flow processing module 436 may invoke dialog flow processing module 434 to engage in a dialogue with the user. In some examples, dialog flow processing module 434 may determine how (and / or when) to ask the user for additional information and may receive and process user responses. Questions may be provided to the user and answers may be received from the user through I / O processing module 428. In some examples, dialog flow processing module 434 may present dialog output to the user via audio and / or visual output and may receive input from the user via verbal or physical (e.g., click) responses. For example, a user may ask, "How's the weather in Paris?" When task flow processing module 436 invokes dialog flow processing module 434 to determine "location" information for a structured query associated with the domain "weather search," dialog flow processing module 434 can generate a question, such as "Which Paris?", to pass to the user. Additionally, dialog flow processing module 434 can cause affordances associated with "Paris, Texas" and "Paris, France" to be presented for user selection. Once a response is received from the user, dialog flow processing module 434 can then enter the missing information in the structured query or pass the missing information to task flow processing module 436 to complete the missing information from the structured query.

[0101] Once the taskflow processing module 436 completes the structured query for the actionable intent, the taskflow processing module 436 can proceed to execute the final task associated with the actionable intent. Accordingly, the taskflow processing module 436 can execute steps and instructions in the taskflow model 454 according to the specific parameters included in the structured query. For example, a taskflow model for a “media search” actionable intent may include steps and instructions for executing a media search query and retrieving relevant media items. For example, using a structured query such as {media search, media genre=TV series, media title=Mad Men}, the taskflow processing module 436 can execute the steps of (1) executing the media search query using a media database to retrieve relevant media items, (2) ranking the retrieved media items according to relevance and / or popularity, and (3) displaying the media items sorted according to relevance and / or popularity.

[0102] In some embodiments, task flow processing module 436 may employ the assistance of service processing module 438 ("service processing module") to complete a task requested in the user input or to provide an answer for information requested in the user input. For example, service processing module 438 may act on behalf of task flow processing module 436 to perform a media search, retrieve weather information, invoke or interact with applications installed on other user devices, and invoke or interact with third-party services (e.g., social networking websites, media review websites, media subscription services, etc.). In some embodiments, the protocols and APIs required by each service may be specified by a respective service model in service models 456. Service processing module 438 may access the appropriate service model for the service and generate a service request according to the protocols and APIs required by the service associated with the service model.

[0103] For example, a third-party media search service may submit a service model that specifies the parameters required to perform a media search and an API for communicating the values ​​of the required parameters to the media search service. Upon request by the task flow processing module 436, the service processing module 438 may establish a network connection with the media search service and send the required parameters of the media search (e.g., media actors, media genres, media titles) to the online booking interface in a format according to the API of the media search service.

[0104] In some examples, the natural language processing module 432, the dialog flow processing module 434, and the task flow processing module 436 can be used collectively and iteratively to infer and clarify a user's intent, obtain information to further clarify and refine the user's intent, and ultimately generate a response (e.g., output to the user or completion of a task) to achieve the user's intent. The generated response can be a dialog response to the speech input that at least partially achieves the user's intent. Furthermore, in some examples, the generated response can be output as speech output. In these examples, the generated response can be sent to speech synthesis module 440 (e.g., a speech synthesizer), where it can be processed to synthesize a speech-based dialog response. In still other examples, the generated response can be data content related to fulfilling the user request in the speech input.

[0105] The speech synthesis module 440 can be configured to synthesize speech output for presentation to the user. The speech synthesis module 440 synthesizes speech output based on text provided by the digital assistant. For example, the generated dialog response can be in the form of a text string. The speech synthesis module 440 can convert the text string into audible speech output. The speech synthesis module 440 can use any suitable speech synthesis technique to generate the speech output from the text, including, but not limited to, concatenative synthesis, unit selection synthesis, diphone synthesis, domain-specific synthesis, formant synthesis, articulatory synthesis, hidden Markov model (HMM)-based synthesis, and sinusoidal synthesis. In some embodiments, the speech synthesis module 440 can be configured to synthesize individual words based on phoneme strings corresponding to the words. For example, the phoneme strings can be associated with words in the generated dialog response. The phoneme strings can be stored in metadata associated with the words. The speech synthesis module 440 can be configured to directly process the phoneme strings in the metadata to synthesize words in speech form.

[0106] In some examples, instead of (or in addition to) using speech synthesis module 440, speech synthesis can be performed on a remote device (e.g., server system 108), and the synthesized speech can be sent to a user device for output to the user. For example, this can be done in some implementations where output for a digital assistant is generated at a server system. Also, because server systems typically have more processing power or resources than user devices, it can be possible to obtain higher quality speech output than would be possible using client-side synthesis.

[0107] Further details regarding digital assistants can be found in U.S. Utility Patent Application No. 12 / 987,982, entitled "Intelligent Automated Assistant," filed January 10, 2011, and U.S. Utility Patent Application No. 13 / 251,088, entitled "Generating and Processing Task Items That Represent Tasks to Perform," filed September 30, 2011, the entire disclosures of which are incorporated herein by reference. 4. Process for interacting with digital assistants within a media environment

[0108] 5A-5I illustrate a process 500 for operating a digital assistant in a media system, according to various embodiments. Process 500 can be performed using one or more electronic devices implementing a digital assistant. For example, process 500 can be performed using one or more of system 100, media system 128, media device 104, user device 122, or digital assistant system 400, described above. FIGS. 6A-6Q illustrate screenshots displayed on a display unit by a media device at various stages of process 500, according to various embodiments. Process 500 is described below with simultaneous reference to FIGS. 5A-5I and 6A-6Q. It should be understood that some operations in process 500 can be combined, the order of some operations can be changed, and some operations can be omitted.

[0109] At block 502 of process 500, content may be displayed on a display unit (e.g., display unit 126). In this example shown in FIG. 6A , the displayed content may include media content 602 (e.g., a movie, a video, a television program, a video game, etc.) that is playing on a media device (e.g., media device 104). In other examples, the displayed content may include other content associated with the media device, such as content associated with an application running on the media device or a user interface for interacting with the media device's digital assistant. Specifically, the displayed content may include a main menu user interface or a user interface (e.g., second user interface 618 or third user interface 626) having an object or result previously requested by the user.

[0110] At block 504 of process 500, user input may be detected. The user input may be detected while the content of block 502 is being displayed. In some examples, the user input may be detected on a remote control device (e.g., remote control device 124) of the media device. Specifically, the user input may be a user interaction with the remote control device, such as pressing a button (e.g., button 274) or touching a touch-sensitive surface (e.g., touch-sensitive surface 278) of the remote control device. In some examples, the user input may be detected via a second electronic device (e.g., device 122) configured to interact with the media device. In response to detecting the user input, one or more of blocks 506-592 may be executed.

[0111] At block 506 of process 500, a determination may be made as to whether the user input corresponds to a first input format. The first input format may be a default input to the media device. In one example, the first input format may include pressing a particular button on a remote control device and releasing the button within a predetermined period of time after pressing the button (e.g., a short press). The media device may determine whether the user input matches the first input format. One or more of blocks 508-514 may be executed in accordance with a determination that the user input corresponds to the first input format.

[0112] In block 508 of process 500, and referring to FIG. 6B, textual instructions 604 for invoking and interacting with the digital assistant can be displayed. Specifically, instructions 604 can describe the user input required to invoke and interact with the digital assistant. For example, instructions 604 can describe how to perform the second input format described below in block 516.

[0113] At block 510 of process 500 and as shown in FIG. 6B , a passive visual indicator 606 can be displayed on the display unit. The passive visual indicator 606 can indicate that the digital assistant has not yet been invoked. Specifically, a microphone (e.g., microphone 272) of the media device may not be activated in response to detecting user input. Therefore, the passive visual indicator 606 can serve as a visual signal that the digital assistant is not processing voice input. In this example, the visual indicator 606 can be a passive, flat waveform that does not respond to user utterances. Furthermore, the passive visual indicator 606 can include a neutral color (e.g., black, gray, etc.) to indicate its passive status. It should be appreciated that other visual patterns or images can also be contemplated for the passive visual indicator. The passive visual indicator 606 can be displayed simultaneously with the instruction 604. Furthermore, the passive visual indicator 606 can be continuously displayed while one or more of blocks 512-514 are executing.

[0114] At block 512 of process 500 and with reference to FIG. 6C , instructions 608 for performing a typing search can be displayed on the display unit. Specifically, the instructions 608 can describe the user inputs necessary to display a virtual keyboard interface that can be used to perform a typing search. In some embodiments, the instructions 604 for invoking and interacting with the digital assistant and the instructions 608 for performing a typing search can be displayed sequentially at different times. For example, the display of the instructions 608 can replace the display of the instructions 604, or vice versa. In this example, the instructions 604, 608 are in text format. It should be appreciated that in other embodiments, the instructions 604, 608 can be in graphical format (e.g., pictures, symbols, animations, etc.).

[0115] At block 514 of process 500, one or more exemplary natural language requests can be displayed on the display unit. For example, FIGS. 6D-6E show two different exemplary natural language requests 610, 612 displayed on the display unit. In some embodiments, the exemplary natural language requests can be displayed on the display unit via a first user interface. The first user interface can be overlaid on the displayed content. The exemplary natural language requests can provide the user with guidance for interacting with the digital assistant. Additionally, the exemplary natural language requests can inform the user of various functions of the digital assistant. In response to receiving a user utterance corresponding to one of the exemplary natural language requests, the digital assistant can cause the respective action to be performed. For example, in response to a digital assistant on a media device being invoked (e.g., by user input of the second input form at block 504) and a user utterance of "jump forward 30 seconds" being provided (e.g., at block 518), the digital assistant can cause the media content being played on the media device to skip forward 30 seconds.

[0116] The displayed example natural language requests can be contextually related to the displayed content (e.g., media content 602). For example, a set of example natural language requests can be stored on the media device or on a separate server. Each example natural language request in the set of example natural language requests can be associated with one or more context attributes (e.g., currently playing media content, homepage, iTunes® media store, actor, movie, weather, sports, stock prices, etc.). In some embodiments, block 514 can include identifying, from the set of example natural language requests, an example natural language request having a context attribute that corresponds to the content displayed on the display unit. The identified example natural language request can then be displayed on the display unit. Thus, different example natural language requests may be displayed depending on the content displayed on the display unit. Displaying contextually relevant example natural language requests can help conveniently inform the user of the digital assistant features that are most relevant to the user's current usage situation on the media device. This can improve the overall user experience.

[0117] 6D-6E, each of the exemplary natural language requests 610, 612 may be contextually related to the media content 602 on the display unit. Specifically, the exemplary natural language requests 610, 612 may be requests to change or control one or more settings associated with the media content being played on the media device. Such exemplary natural language requests may include requests to turn closed captioning on / off, turn on subtitles in a particular language, rewind / skip ahead, pause playback of the media content, restart playback of the media content, slow down or speed up playback of the media content, increase / decrease the volume (e.g., audio gain) of the media content, and the like. Additionally, other exemplary natural language requests contextually related to media content 602 may include requests to add a media item corresponding to media content 602 to a user's watchlist, to show information related to media content 602 (e.g., actor information, synopsis, release date, etc.), to show other media items or content related to media content 602 (e.g., same series, same season, same actor / director, same genre, etc.), and the like.

[0118] In embodiments in which the displayed content includes content associated with an application of the media device, exemplary contextually relevant natural language requests may include requests to change one or more settings or states of the application. Specifically, exemplary natural language requests may include requests to open or close the application or to operate one or more features of the application.

[0119] In some examples, the displayed content may include a user interface (e.g., second user interface 618 or third user interface 626) for searching, browsing, or selecting an item. Specifically, the displayed user interface may include one or more media items. Further, the focus of the user interface may be on a media item (e.g., media item 623 highlighted by cursor 624 in FIG. 6G ) of the one or more media items. In these examples, exemplary contextually relevant natural language requests may include requests for information related to one or more media items in the displayed user interface or other media items. Specifically, exemplary natural language requests may include requests related to the media item that is the focus of the user interface. In these examples, exemplary natural language requests may include requests such as, "What's this?", "What's the rating for this?", "Who's in this?", "When's the next episode coming out?", "What are more movies like this?", and "What movies do the same actor star in?". In particular examples, information related to a media item or a series of media items, such as the television series Mad Men, may be displayed via the user interface. In this example, exemplary contextually relevant natural language requests may include requests based on one or more attributes of a media item or set of media items (e.g., cast, plot, rating, release date, director, provider, etc.) (e.g., "More shows with January Jones in them.") Additionally, exemplary contextually relevant natural language requests may include requests to play, select, or obtain the focused media item or another media item displayed within the user interface (e.g., "Rent this," "Play this," "Buy this," or "Play How to Train Your Dragon 2")."), or a request to navigate among media items within a user interface (e.g., "Go to comedies," or "Jump to horror movies."). Further, in these examples, exemplary contextually relevant natural language requests may include requests to search for other media items (e.g., "Find me a new comedy," "What are some free classic movies?", or "What are some shows starring Nicole Kidman?").

[0120] In some examples, the displayed content may include media items organized according to a particular category or topic. In these examples, exemplary contextually relevant natural language requests may include requests related to that particular category or topic. For example, in examples where the displayed content includes media items organized according to various actors, exemplary contextually relevant natural language requests may include requests for information or media items related to the actors (e.g., "What movies star Jennifer Lawrence?", "How old is Scarlett Johansson?", or "What's the latest movie with Brad Pitt?"). In other examples where the displayed content includes media items organized according to programming channel or content provider (e.g., a channel page or a TV guide page), exemplary contextually relevant natural language requests may include requests for information or media items related to the programming channel or content provider (e.g., "What show is on in an hour?", "What's on HBO in prime time?", "Tune in to ABC," or "What channel is basketball on?"). In yet another example, where the displayed content includes media items recently selected by the user (e.g., a "recently played" list) or identified as of interest to the user (e.g., a "watch list"), exemplary contextually relevant natural language requests may include requests to watch or continue watching one of the media items (e.g., "Please resume where we left off," "Continue watching Birdman," or "Play this again from the beginning").

[0121] In some examples, the displayed content can include a user interface that includes results or information corresponding to a particular topic. Specifically, the results can be associated with a previous user request (e.g., a request to a digital assistant) and can include information corresponding to a topic such as weather, stock prices, or sports. In these examples, an exemplary contextually relevant natural language request can include a request to narrow the results or a request for additional information related to a particular topic. For example, in examples in which the displayed content includes weather information for a particular location, an exemplary contextually relevant natural language request can include a request to display additional weather information for another location or for a different time period (e.g., "What's it like in New York City?", "What's next week going to bring?", "And what about Hawaii?", etc.). In another example where the displayed content includes information related to a sports team or an athlete, an exemplary contextually relevant natural language request may include a request for additional information related to the sports team or athlete (e.g., "How tall is Shaquille O'Neal?", "When was Tom Brady born?", "When is the 49ers' next game?", "How did Manchester United do in their last game?", "Who is the point guard for the LA Lakers?", etc.). In yet another example where the displayed content includes information related to stock prices, an exemplary contextually relevant natural language request may include a request for additional stock-related information (e.g., "What was the opening price of the S&P 500?", "How is Apple doing?", "What was the closing price of the Dow Jones yesterday?", etc.). Additionally, in some examples, the displayed content may include a user interface containing media search results associated with a previous user request.In these examples, exemplary contextually relevant natural language requests may include requests to narrow the displayed media search results (e.g., "only last year's," "only G-rated," "only free," etc.) or to perform a different media search (e.g., "please find me some good action movies," "can you recommend some Jackie Chan movies," etc.).

[0122] In some examples, the displayed content may include a main menu user interface of the media device. The main menu user interface may be, for example, a home screen or root directory of the media device. In these examples, exemplary contextually relevant natural language requests may include requests expressing various functions of the digital assistant. Specifically, the digital assistant may have a set of core capabilities associated with the media device, and exemplary contextually relevant natural language requests may include requests related to each of the digital assistant's core capabilities (e.g., "What are some good free movies?", "What's the weather like?", "Play the next episode of Breaking Bad?", or "What's Apple's stock price?").

[0123] The exemplary natural language request can be in a natural language format. This can help inform the user that the digital assistant has the ability to understand the natural language request. Furthermore, in some examples, the exemplary natural language request can be contextually ambiguous to inform the user that the digital assistant has the ability to infer the correct user intent associated with the user's request based on the displayed content. Specifically, as shown in the examples described above, the exemplary natural language request can include contextually ambiguous terms such as "this" or "ones," or contextually ambiguous phrases such as "only free stuff" or "how about in New York?" These exemplary natural language requests can inform the user that the digital assistant has the ability to determine the correct context associated with such a request based on the displayed content. This encourages the user to rely on the context of the displayed content when interacting with the digital assistant. This can be desirable to promote a more natural interaction experience with the digital assistant.

[0124] In some embodiments, block 514 may be executed after blocks 508-512. Specifically, the example natural language request may be displayed on the display unit a predetermined time after determining that the user input corresponds to the first input format in block 506. It should be appreciated that in some embodiments, blocks 508-514 may be executed in any order, and in some embodiments, two or more of blocks 508-514 may be executed simultaneously.

[0125] In some embodiments, the example natural language requests are displayed in a predetermined order in rotation. Each example natural language request may be displayed separately at a different time. Specifically, the display of a current example natural language request may be replaced with the display of a subsequent example natural language request. For example, as shown in FIG. 6D , example natural language request 610 may be displayed first. After a predetermined time, the display of example natural language request 610 ("jump forward 30 seconds") may be replaced with the display of example natural language request 612 ("play the next episode"), as shown in FIG. 6E . Thus, in this embodiment, example natural language request 610 and example natural language request 612 are displayed one after the other, rather than simultaneously.

[0126] In some examples, the example natural language requests may be grouped into multiple lists, with each list including one or more example natural language requests. In these examples, block 514 may include displaying the lists of example natural language requests on a display unit. Each list may be displayed at a different time in a predetermined order. Furthermore, the lists may be displayed in rotation.

[0127] While one or more of blocks 508-514 are being executed, the displayed content may continue to be displayed on the display unit. For example, as shown in FIGS. 6B-6E, media content 602 may continue to be played on the media device and displayed on the display unit while blocks 508-512 are being executed. Additionally, while the media content is being played, audio associated with the media content may be output by the media device. In some embodiments, the amplitude of the audio is not reduced in response to detecting user input or in accordance with determining that the user input corresponds to the first input format. This may be desirable to reduce interruptions in the consumption of the media content 602 being played. Thus, the user may continue to follow the media content 602 via the audio output despite elements 604-612 being displayed on the display unit.

[0128] In some examples, as represented by the hollow font of the media content 602 in Figures 6B-6D, in response to detecting user input or determining that the user input corresponds to a first input format, the brightness of the displayed content may be reduced (e.g., by 20-40%). In these examples, the displayed elements 604-612 may be overlaid on the displayed media content 602. The reduced brightness may help make the displayed elements 604-612 more noticeable. At the same time, the media content 602 may still be discernible on the display unit, thereby allowing the user to continue consuming the media content 602 while the elements 604-612 are displayed.

[0129] While performing one of blocks 508-512, a digital assistant can be invoked (e.g., by detecting user input in the second input format in block 504) and a user utterance corresponding to one of the example natural language requests can be received (e.g., in block 518). The digital assistant can then perform a task in response to the received request (e.g., in block 532). Further details regarding invoking and interacting with the digital assistant are provided below with reference to FIGS. 5B-5I. Additionally, while performing one of blocks 508-512, a virtual keyboard interface for performing a typed search can be invoked (e.g., by detecting a fifth user input in block 558). Further details regarding invoking a virtual keyboard interface and performing a typed search are provided below with reference to FIG. 5G.

[0130] Referring back to block 506, one or more of blocks 516-530 of FIG. 5B can be executed in accordance with a determination that the user input does not correspond to the first input format. In block 516, a determination can be made as to whether the user input corresponds to a second input format. The second input format can be a default input to the media device that is different from the first input format. In some examples, the second input format can include pressing a specific button on a remote control device of the media device and holding the button down for longer than a predetermined period (e.g., a long press). The second input format can be associated with invoking a digital assistant. In some examples, the first input format and the second input format can be implemented using the same button on the remote control device (e.g., a button configured to invoke the digital assistant). This can be desirable to intuitively integrate invoking the digital assistant and providing instructions for invoking and interacting with the digital assistant into a single button. Furthermore, inexperienced users may intuitively perform short presses rather than long presses. Therefore, providing instructions in response to detecting a short press can allow instructions to be primarily targeted to less experienced users rather than more experienced users, which can improve the user experience by making instructions readily visible to less experienced users who need guidance most, while allowing more experienced users the option to bypass the instructions.

[0131] Pursuant to a determination at block 516 that the user input corresponds to the second input format, one or more of blocks 518-530 may be executed. In some embodiments, the media content 602 may continue to play on the media device while one or more of blocks 518-530 are executed. Specifically, the media content 602 may continue to play on the media device and be displayed on the display unit while sampling the audio data at block 518 and while performing the task at block 528.

[0132] At block 518 of process 500, audio data may be sampled. Specifically, a first microphone (e.g., microphone 272) of the media device may be activated to begin sampling the audio data. In some embodiments, the sampled audio data may include user utterances from a user. The user utterances may represent user requests directed to the digital assistant. Furthermore, in some embodiments, the user request may be a request to perform a task. Specifically, the user request may be a media search request. For example, with reference to FIG. 6F, the sampled audio data may include the user utterance, "Find me a romantic comedy starring Reese Witherspoon." In other embodiments, the user request may be a request to play a media item or provide specific information (e.g., weather, stock prices, sports, etc.).

[0133] The user utterances in the sampled audio data can be in natural language form. In some embodiments, the user utterances can express user requests that are incompletely specified, where the user utterances do not explicitly qualify all of the information needed to satisfy the user request. For example, the user utterance can be, "Play the next episode." In this embodiment, the user request does not explicitly qualify the media series from which the next episode should be played. Furthermore, in some embodiments, the user utterances can include one or more ambiguous terms.

[0134] The period over which the audio data is sampled can be based on the detection of an endpoint. Specifically, the audio data can be sampled from a start time when user input of the second form of input is first detected to an end time when the endpoint is detected. In some embodiments, the endpoint can be based on user input. Specifically, upon first detecting user input of the second form of input (e.g., pressing a button for longer than a predetermined period), the first microphone can be activated. The first microphone can remain activated to sample the audio data while user input of the second form of input continues to be detected. Upon ceasing detection of user input of the second form of input (e.g., a button is released), the first microphone can be deactivated. Thus, in these embodiments, the endpoint is detected upon detecting the end of the user input. Thus, the audio data is sampled while detecting user input of the second form of input.

[0135] In other embodiments, detecting the endpoint may be based on one or more audio characteristics of the sampled audio data. Specifically, one or more audio characteristics of the sampled audio data may be monitored, and the endpoint may be detected a predetermined time after determining that the one or more audio characteristics do not satisfy one or more predetermined criteria. In yet other embodiments, the endpoint may be detected based on a fixed time period. Specifically, the endpoint may be detected a predetermined time period after initially detecting user input of the second input form.

[0136] In some embodiments, audio associated with the displayed content can be output (e.g., using speaker 268) while block 504 or 516 is being executed. Specifically, the audio can be audio of a media item playing on a media device and displayed on a display unit. The audio can be output from the media device via an audio signal. In these embodiments, the audio associated with the displayed content can be ducked (e.g., the amplitude of the audio is reduced) when the user input is determined to correspond to the second input format and when the audio data is sampled. For example, the audio can be ducked by reducing a gain associated with the audio signal. In other embodiments, the output of audio associated with the media content can be stopped while the audio data is being sampled in block 518. For example, the audio can be stopped by blocking or pausing the audio signal. Ducking or stopping the output of audio can be desirable to reduce background noise in the sampled audio data and increase the relative intensity of the speech signal associated with the user's utterances. Furthermore, ducking or stopping the audio can serve as an audio cue for the user to begin providing speech input to the digital assistant.

[0137] In some embodiments, background audio data can be sampled while sampling audio data to perform noise cancellation. In these embodiments, the remote control device or media device can include a second microphone. The second microphone can be oriented differently from the first microphone (e.g., facing away from the first microphone). The second microphone can be activated to sample background audio data while the audio data is being sampled. In some embodiments, the background audio data can be used to remove background noise in the audio data. In other embodiments, the media device can generate an audio signal to output audio associated with displayed content. The generated audio signal can be used to remove background noise from the audio data. Performing noise cancellation of background noise from audio signals can be particularly suitable for interaction with a digital assistant in a media environment. This may be due to the communal nature of consuming media content, where speech from multiple people may be interwoven in the audio data. Removing background noise in the audio data can result in a higher signal-to-noise ratio in the audio data, which may be desirable when processing the audio data for a user request.

[0138] At block 520 of process 500, and with reference to FIG. 6F, an active visual indicator 614 can be displayed on the display unit. The active visual indicator 614 can indicate to the user that the digital assistant has been invoked and is actively listening. Specifically, the active visual indicator 614 can serve as a visual cue to prompt the user to begin providing speech input to the digital assistant. In some examples, the active visual indicator 614 can include a color and / or visual animation to indicate that the digital assistant has been invoked. For example, as shown in FIG. 6F, the active visual indicator 614 can include an active waveform that responds to one or more characteristics (e.g., amplitude) of the audio data received by the digital assistant. For example, the active visual indicator 614 can display a waveform with a larger amplitude in response to portions of the audio data that are louder and a waveform with a smaller amplitude in response to portions of the audio data that are quieter. Further, in embodiments in which the digital assistant is invoked while displaying passive visual indicator 606 (e.g., FIG. 6E), the display of visual indicator 606 can be replaced with the display of active visual indicator 614. This can provide a natural transition from the instructional user interface shown in FIGS. 6B-6E for illustrating how to invoke and interact with the digital assistant to the active user interface shown in FIG. 6F for actively interacting with the digital assistant.

[0139] At block 522 of process 500, a text representation of the user utterances in the sampled speech data may be determined. For example, the text representation may be determined by performing a speech-to-text (STT) process on the sampled speech data. Specifically, the sampled speech data may be processed using an STT processing module (e.g., STT processing module 430) to convert the user utterances in the sampled speech data into text representations. The text representations may be token strings that represent corresponding text strings.

[0140] In some embodiments, the STT process can be biased toward media-related text results. Biasing can be performed by utilizing a language model trained with a corpus of media-related text. Additionally or alternatively, biasing can be performed by weighting media-related text result candidates more heavily. In this manner, biasing can result in media-related text result candidates being ranked higher than would be the case without biasing. Biasing can be desirable to improve the accuracy of STT processing of media-related user utterances (e.g., movie names, movie actors, etc.). For example, certain media-related words or phrases, such as "Jurassic Park," "Arnold Schwarzenegger," and "Shrek," are rarely found in typical text corpora and therefore may not be successfully recognized during STT processing without biasing toward media-related text results.

[0141] In some embodiments, the text representation can be obtained from a separate device (e.g., DA server 106). Specifically, sampled audio data can be transmitted from the media device to the separate device for performing STT processing. In these embodiments, the media device can indicate to the separate device (e.g., via data transmitted to the separate device along with the sampled audio data) that the sampled audio data is associated with a media application. The indication can bias the STT processing toward media-related text results.

[0142] In some embodiments, the text representation may be based on previous user utterances received by the media device prior to sampling the audio data. In particular, text result candidates in the sampled audio data that correspond to one or more portions of previous user utterances may be weighted more heavily. In some embodiments, the previous user utterances may be used to generate a language model, and the generated language model may be used to determine a text representation of a current user utterance in the sampled audio data. The language model may be dynamically updated as additional user utterances are received and processed.

[0143] Furthermore, in some embodiments, the text representation may be based on the time that the previous user utterance was received prior to sampling the audio data. In particular, text result candidates corresponding to previous user utterances that were received more recently with respect to the sampled audio data may be weighted more heavily than text result candidates corresponding to previous user utterances that were received earlier with respect to the sampled audio data.

[0144] In block 524 of process 500, a text representation can be displayed on a display unit. For example, FIG. 6F shows a text representation 616 corresponding to a user utterance in the sampled audio data. In some embodiments, blocks 522 and 524 can be performed while the audio data is being sampled. Specifically, the text representation 616 of the user utterance can be displayed in a streaming manner such that the text representation 616 is displayed in real time as the audio data is sampled and as STT processing is performed on the sampled audio data. Displaying the text representation 616 can provide the user with confirmation that the digital assistant is correctly processing the user's request.

[0145] At block 526 of process 500, a user intent corresponding to the user utterance may be determined. The user intent may be determined by performing natural language processing on the text representation of block 522. Specifically, a natural language processing module (e.g., natural language processing module 432) may be used to process the text representation to derive the user intent. For example, with reference to FIG. 6F , from the text representation 616 corresponding to “Find romantic comedies starring Reese Witherspoon,” it may be determined that the user intent is to request a search for media items having the romantic comedy genre and Reese Witherspoon as an actor. In some embodiments, block 526 may further include generating, using the natural language processing module, a structured query expressing the determined user intent. In this embodiment, “Find romantic comedies starring Reese Witherspoon” may generate a structured query expressing a media search query for media items having the romantic comedy genre and Reese Witherspoon as an actor.

[0146] In some embodiments, natural language processing for determining user intent can be biased toward media-related user intent. Specifically, a natural language processing module can be trained to identify media-related words and phrases (e.g., media titles, media genres, actors, MPAA movie rating labels, etc.) that trigger media-related nodes in an ontology. For example, the natural language processing module can identify the phrase "Jurassic Park" in a text representation as a movie title, thereby triggering a "media search" node in the ontology associated with the actionable intent of searching for media items. In some embodiments, the bias can be implemented by limiting the nodes in the ontology to a predetermined set of media-related nodes. For example, the set of media-related nodes can be nodes associated with applications of a media device. Furthermore, in some embodiments, the bias can be implemented by weighting media-related candidate user intents more heavily than non-media-related candidate user intents.

[0147] In some examples, the user intent can be obtained from a separate device (e.g., the DA server 106). Specifically, the audio data can be transmitted to the separate device for performing natural language processing. In these examples, the media device can indicate to the separate device (e.g., via data transmitted to the separate device along with the sampled audio data) that the sampled audio data is associated with a media application. The indication can bias the natural language processing toward media-related user intent.

[0148] At block 528 of process 500, a determination may be made as to whether the sampled voice data includes a user request. The determination may be made from the determined user intent of block 526. If the user intent includes a user request to perform a task, the sampled voice data may be determined to include a user request. Conversely, if the user intent does not include a user request to perform a task, the sampled voice data may be determined not to include a user request. Further, in some embodiments, if the user intent cannot be determined from the text representation at block 526 or if the text representation cannot be determined from the sampled voice data at block 522, the sampled voice data may be determined not to include a user request. Following a determination that the voice data does not include a user request, block 530 may be executed.

[0149] At block 530 of process 500, a request for clarification of the user's intent can be displayed on the display unit. In one embodiment, the request for clarification can be a request to the user to repeat the user request. In another embodiment, the request for clarification can be a statement that the digital assistant cannot understand the user's utterance. In yet another embodiment, an error message can be displayed to indicate that the user's intent could not be determined. Furthermore, in some embodiments, no response may be provided following a determination that the voice data does not contain a user request.

[0150] 5C , in accordance with a determination in block 528 that the sampled voice data contains a user request, block 532 can be executed. In block 532 of process 500, a task can be executed to at least partially satisfy the user request. For example, executing the task in block 526 can include executing one or more tasks defined in the generated structured query of block 526. The one or more tasks can be executed using a task flow processing module (e.g., task flow processing module 436) of the digital assistant. In some embodiments, the task can include changing the state or settings of an application on the media device. More specifically, the task can include, for example, selecting or playing a requested media item, opening or closing a requested application, or navigating a displayed user interface in a requested manner. In some embodiments, in block 532, the task can be executed without outputting speech related to the task from the media device. Thus, in these embodiments, a user can provide a request to the digital assistant in the form of speech, but the digital assistant need not provide a response to the user in the form of speech. Rather, the digital assistant may only respond visually by displaying the results on the display unit, which may be desirable to preserve the shared experience of consuming media content.

[0151] In other examples, the task may include retrieving and displaying the requested information. Specifically, performing the task in block 532 may include performing one or more of blocks 534-536. In block 534 of process 500, results that at least partially satisfy the user request may be obtained. The results may be obtained from an external service (e.g., external service 120). In one example, the user request may be a request to perform a media search query, such as "Find me romantic comedies starring Reese Witherspoon." In this example, block 534 may include performing the requested media search (e.g., using a media association database of the external service) and retrieving media items having the romantic comedy genre and Reese Witherspoon as an actor. In other examples, the user request may include requests for other types of information, such as weather, sports, and stock prices, and the respective information may be retrieved in block 534.

[0152] At block 536 of process 500, a second user interface may be displayed on the display unit. The second user interface may include a portion of the results obtained at block 534. For example, as shown in FIG. 6G, a second user interface 618 may be displayed on the display unit. The second user interface 618 may include media items 622 that satisfy the user request "Find a romantic comedy starring Reese Witherspoon." In this example, the media items 622 may include media items such as "Legally Blonde," "Legally Blonde 2," "Hot Pursuit," and "This Means War." The second user interface 618 may further include a text header 620 that describes the results obtained. The text header 620 may rephrase a portion of the user request to give the impression that the user's request has been directly addressed. This provides a more pleasant and interactive experience between the user and the digital assistant. In this example shown in FIG. 6G, the media items 622 are organized in a single column across the second user interface 618. It should be appreciated that in other embodiments, the organization and presentation of media items 622 may differ.

[0153] The second user interface 618 may further include a cursor 624 for navigating and selecting media items 622 within the second user interface 618. The position of the cursor may be indicated by visually highlighting the media item over which the cursor is positioned relative to other media items. For example, in this example, the media item 623 over which the cursor 624 is positioned may be made larger and outlined with a thicker line compared to the other media items displayed within the second user interface 618.

[0154] In some embodiments, at least a portion of the displayed content can continue to be displayed while the second user interface is displayed. For example, as shown in FIG. 6G, the second user interface 618 can be a small pane displayed at the bottom of the display unit, while the media content 602 continues to play on the media device and be displayed on the display unit above the second user interface 618. The second user interface 618 can be overlaid on the playing media content 602. In this embodiment, the display area of ​​the second user interface 618 on the display unit can be smaller than the display area of ​​the media content 602 on the display unit. This can be desirable to reduce the intrusiveness of results displayed by the digital assistant while the user is consuming the media content. It should be appreciated that in other embodiments, the display area of ​​the second user interface relative to the display area of ​​the displayed content can be different. Furthermore, as indicated by the solid font for "Media Playing" in FIG. 6G, displaying the second user interface 618 can return the brightness of the media content 602 to normal (e.g., the brightness in FIG. 6A before detecting user input). This can help indicate to the user that their interaction with the digital assistant is complete, so the user can continue consuming media content 602 while viewing the requested result (e.g., media item 622).

[0155] In embodiments in which media items retrieved from a media search are displayed on the second user interface, the number of media items displayed can be limited. This can be desirable to allow a user to focus on the most relevant results and prevent the user from becoming overwhelmed by the number of results when making a selection. In these embodiments, block 532 can further include determining whether the number of media items in the retrieved results is less than or equal to a predetermined number (e.g., 30, 28, or 25). In accordance with a determination that the number of media items in the retrieved results is less than or equal to the predetermined number, all of the media items in the retrieved results can be included in the second user interface. In accordance with a determination that the number of media items in the retrieved results is greater than the predetermined number, only the predetermined number of media items in the retrieved results can be included in the second user interface.

[0156] Further, in some embodiments, only media items in the retrieved results that are most relevant to the media search request may be displayed in the second user interface. Specifically, each media item in the retrieved results may be associated with a relevance score with respect to the media search request. The displayed media item may have the highest relevance score among the retrieved results. Furthermore, the media items in the second user interface may be arranged according to relevance score. For example, with reference to FIG. 6G , media items with higher relevance scores may be more likely to be located near one side of the second user interface 618 (e.g., the side closer to the cursor 624), while media items with lower relevance scores may be more likely to be located near the opposite side of the second user interface 618 (e.g., the side farther from the cursor 624). Additionally, each media item in the retrieved results may be associated with a popularity rating. The popularity rating may be based on movie critic ratings (e.g., Rotten Tomatoes ratings) or the number of users who selected the media item for playback. In some embodiments, the media items 622 may be arranged within the second user interface 618 based on popularity rating. For example, media items with higher popularity ratings may be more likely to be located on one side of the second user interface 618, while media items with lower popularity ratings may be more likely to be located closer to the opposite side of the second user interface 618.

[0157] As indicated by the different flows (e.g., D, E, F, and G) following block 532 in FIG. 5C , one of blocks 538, 542, 550, or 570 of FIG. 5D, 5E, 5F, or 51, respectively, can be executed after block 532. Blocks 538, 542, 550, or 570 can be executed while displaying the second user interface in block 536. In some embodiments, process 500 can alternatively include a decision step after block 536 to determine the appropriate flow (e.g., D, E, F, or G) to execute. Specifically, a user input can be detected after block 536, and a determination can be made as to whether the detected user input corresponds to the second user input (e.g., block 538), the third user input (e.g., block 542), the fourth user input (e.g., block 550), or the sixth user input (e.g., block 570). For example, one or more of blocks 544-546 may be executed pursuant to a determination that the user input corresponds to the third user input in block 542. Similar decision steps may be included after block 546.

[0158] At block 538 of process 500 and with reference to FIG. 5D , a second user input can be detected. As described above, the second user input can be detected while the second user interface is displayed on the display unit. The second user input can be detected on a remote control device of the media device. For example, the second user input can include a first predetermined movement pattern on the touch-sensitive surface of the remote control device. In one embodiment, the first predetermined movement pattern can include continuous contact movement in a first direction from a first contact point to a second contact point on the touch-sensitive surface. When the remote control device is being held in the intended manner, the first direction can be a downward direction or a direction toward the user. It should be appreciated that other input forms for the second user input can also be contemplated. In response to detecting the second user input, block 540 can be executed.

[0159] At block 540 of process 500, the second user interface may be closed, such that the second user interface is no longer displayed. For example, referring to FIG. 6G, in response to detecting the second user input, the second user interface 618 is no longer displayed. In this example, closing the second user interface 618 may cause the media content 602 to be displayed full screen on the display unit. For example, ceasing to display the second user interface 618 may cause the media content 602 to be displayed as shown in FIG. 6A.

[0160] At block 542 of process 500 and with reference to FIG. 5E , a third user input can be detected. The third user input can be detected while a second user interface is displayed on the display unit. The third user input can be detected on a remote control device of the media device. For example, the third user input can include a second predetermined movement pattern on the touch-sensitive surface of the remote control device. The second predetermined movement pattern can include continuous contact movement in a second direction from a third contact point to a fourth contact point on the touch-sensitive surface. The second direction can be opposite to the first direction. Specifically, when the remote control device is being held in an intended manner, the second direction can be upward or away from the user. In response to detecting the third user input, one or more of blocks 544-546 can be executed. 6G, second user interface 618 may include a graphic indicator 621 (e.g., an arrow) to indicate to the user that providing a third user input can expand second user interface 618. Additionally, graphic indicator 621 may indicate to the user a second direction associated with a second predetermined movement pattern on the touch-sensitive surface for the third user input.

[0161] At block 544 of process 500, second results may be obtained. The obtained second results may be similar to, but not identical to, the results obtained at block 534. In some embodiments, the obtained second results may at least partially satisfy the user request. For example, the obtained second results may share one or more characteristics, parameters, or attributes of the results obtained at block 534. In the embodiment shown in FIGS. 6F-6G, block 544 may include performing one or more additional media search queries related to the media search query performed at block 534. For example, the one or more additional media search queries may include a search for media items having a genre of romantic comedy or a search for media items starring Reese Witherspoon. Thus, the obtained second results may include media items that are romantic comedies (e.g., media item 634) and / or media items starring Reese Witherspoon (e.g., media item 636).

[0162] In some examples, the retrieved second results may be based on a previous user request received prior to detecting the user input in block 504. Specifically, the retrieved second results may include one or more characteristics or parameters of the previous user request. For example, the previous user request may be, "What movies have been released within the last five years?" In this example, the retrieved second results may include media items that are romantic comedy movies starring Reese Witherspoon that have been released within the last five years.

[0163] Further, in some embodiments, block 544 may include retrieving second results that are contextually related to the item on which the second user interface is focused when the third user input is detected. For example, referring to FIG. 6G , when the third user input is detected, cursor 624 may be positioned on media item 623 in second user interface 618. Media item 623 may be, for example, the movie "Legally Blonde." In this embodiment, the retrieved second results may share one or more characteristics, attributes, or parameters associated with the media item "Legally Blonde." Specifically, the retrieved second results may include media items related to attending law school or featuring women in professional occupations, such as "Legally Blonde."

[0164] At block 546 of process 500, a third user interface may be displayed on the display unit. Specifically, the display of the second user interface at block 536 may be replaced with the display of the third user interface at block 546. In some embodiments, in response to detecting the third user input, the second user interface may be expanded to become the third user interface. The third user interface may occupy at least a majority of the display area of ​​the display unit. The third user interface may include a portion of the obtained results of block 534. Further, the third user interface may include a portion of the obtained second results of block 544.

[0165] 6H , the third user interface 626 can occupy substantially the entire display area of ​​the display unit. In this example, the previous display of the media content 602 and the second user interface 618 can be superseded by the display of the third user interface 626. In response to detecting the third user input, playback of the media content can be paused on the media device. This can be desirable to prevent the user from missing any portion of the media content 602 while browsing media items in the third user interface 626.

[0166] The third user interface 626 can include media items 622 that satisfy the user request "Find me a romantic comedy starring Reese Witherspoon." Additionally, the third user interface 626 can include media items 632 that at least partially satisfy the same user request. The media items 632 can include multiple sets of media items, each corresponding to a different characteristic, attribute, or parameter. In this example, the media items 632 can include media item 634, which is a romantic comedy, and media item 636, which stars Reese Witherspoon. Each set of media items can be labeled with a text header (e.g., text headers 628, 630). The text headers can describe one or more attributes or parameters associated with the respective set of media items. Additionally, each text header can be an exemplary user utterance that, when provided by the user to the digital assistant, can cause the digital assistant to retrieve a similar set of media items. For example, referring to text header 628, in response to receiving the user statement "romantic comedy" from the user, the digital assistant can retrieve and display a media item (e.g., media item 634) that is a romantic comedy.

[0167] In the example shown in FIG. 6H, media items 622 are based on an initial user request of "Find me a romantic comedy starring Reese Witherspoon," but it should be appreciated that in other examples, media items 632 can be based on other factors, such as media selection history, media search history, the order in which previous media searches were received, relationships between attributes associated with the media, popularity of the media item, and the like.

[0168] In embodiments where the user request is a media search request, the retrieved second results may be based on the number of media items in the retrieved results of block 534. Specifically, in response to detecting the third user input, a determination may be made as to whether the number of media items in the retrieved results is less than or equal to a predetermined number. In accordance with a determination that the number of media items in the retrieved results is less than or equal to the predetermined number, the retrieved second results may include media items that are different from the media items in the second user interface. The retrieved second results may at least partially satisfy the media search request executed in block 534. At the same time, the retrieved second results may be broader than the retrieved results and may be associated with fewer parameters than all of the parameters defined in the media search request executed in block 534. This may be desirable to provide the user with a broader set of results and more options to select from.

[0169] In some embodiments, a determination may be made as to whether the media search request includes more than one search attribute or parameter in accordance with determining that the number of media items in the retrieved results is less than or equal to a predetermined number in block 534. In accordance with determining that the media search request includes more than one search attribute or parameter, the retrieved second results may include media items associated with the more than one search attribute or parameter. Further, the media items in the retrieved second results may be organized in a third user interface according to the more than one search attribute or parameter.

[0170] 6F-6H, a media search request "Find me romantic comedies starring Reese Witherspoon" may be determined to include more than one search attribute or parameter (e.g., "romantic comedies" and "Reese Witherspoon"). In accordance with determining that the media search request includes more than one search attribute or parameter, retrieved second results may include media items 634 associated with the search parameter "romantic comedies" and media items 636 associated with the search parameter "Reese Witherspoon movies." As shown in FIG. 6H, media items 634 may be organized under a category of "romantic comedies," and media items 636 may be organized under a category of "Reese Witherspoon."

[0171] In some embodiments, in response to determining in block 534 that the number of media items in the retrieved results is greater than a predetermined number, the third user interface may include a first portion and a second portion of the retrieved results. The first portion of the retrieved results may include a predetermined number of media items (e.g., those with the highest relevance scores). The second portion of the retrieved results may be different from the first portion of the retrieved results and may include a greater number of media items than the first portion of the retrieved results. Further, it may be determined whether the media items in the retrieved results include more than one media type (e.g., movies, television programs, music, applications, games, etc.). In response to determining that the media items in the retrieved results include more than one media type, the media items in the second portion of the retrieved results may be organized according to media type.

[0172] In the example shown in FIG. 6I, the results retrieved in block 534 may include a media item that is a romantic comedy starring Reese Witherspoon. In response to determining that the number of media items in the retrieved results is greater than a predetermined number, a first portion of the retrieved results (media items 622) and a second portion of the retrieved results (media items 638) may be displayed in third user interface 626. In response to determining that the retrieved results include more than one media type (e.g., movies and TV shows), media items 638 may be organized according to media type. Specifically, media item 640 may be organized under a category of "movies," and media item 642 may be organized under a category of "TV shows." Furthermore, in some examples, each set of media items (e.g., media items 640, 642) corresponding to each media type (e.g., movies, TV shows) may be sorted according to the most prevalent genre, actor / director, or release date within each set of media items. It should be appreciated that in other embodiments, in response to determining that the media items in the retrieved results are associated with more than one media attribute or parameter, the media items in the second portion of the retrieved results may be organized according to media attribute or parameter (rather than media type).

[0173] In some embodiments, a user input representing a scroll command (e.g., a fourth user input described below in block 550) may be detected. In response to receiving the user input representing the scroll command, the enlarged user interface (or more specifically, items within the enlarged user interface) may be scrolled. While scrolling, a determination may be made as to whether the enlarged user interface has scrolled past a predetermined position within the enlarged user interface. In response to determining that the enlarged user interface has scrolled past a predetermined position within the enlarged user interface, media items within a third portion of the retrieved results may be displayed on the enlarged user interface. The media items within the third portion may be organized according to one or more media content providers (e.g., iTunes, Netflix, HuluPlus, HBO, etc.) associated with the media items within the third portion. It should be appreciated that in other embodiments, in response to determining that the enlarged user interface has scrolled past a predetermined position within the enlarged user interface, other media items may be retrieved. For example, popular media items or media items related to the retrieved results may be retrieved.

[0174] As indicated by the different flows (e.g., B, F, G, and H) proceeding from block 546 in Figure 5E, blocks 550, 558, 566, or 570 of Figure 5F, 5G, 5H, or 51, respectively, may be executed after block 532. Specifically, in some embodiments, blocks 550, 560, 564, or 570 may be executed while displaying the third user interface in block 546.

[0175] At block 550 of process 500, and with reference to FIG. 5F, a fourth user input may be detected. The fourth user input may be detected while the second user interface (e.g., second user interface 618) or the third user interface (e.g., third user interface 626) is displayed on the display unit. In some embodiments, the fourth user input may be detected on a remote control device of the media device. The fourth user input may indicate a direction (e.g., up, down, left, right) on the display unit. For example, the fourth user input may be a touch movement from a first position on the touch-sensitive surface of the remote control device to a second position on the touch-sensitive surface that is to the right of the first position. Thus, the touch movement may correspond to a rightward direction on the display unit. In response to detecting the fourth user input, block 552 may be executed.

[0176] At block 552 of process 500, the focus of the second or third user interface may be switched from the first item to a second item on the second or third user interface. The second item may be positioned in the above-described direction relative to the first item (e.g., the same direction corresponding to the fourth user input). For example, in FIG. 6G, cursor 624 may be positioned on media item 623, causing the focus of second user interface 618 to be on media item 623. In response to detecting a fourth user input corresponding to a rightward direction on the display unit, the focus of second user interface 618 may be switched from media item 623 in FIG. 6G to media item 625 in FIG. 6J, which is located to the right of media item 623. Specifically, the position of cursor 624 may be changed from media item 623 to media item 625. In another example, referring to FIG. 6H, the focus of third user interface 626 may be on media item 623. In response to detecting a fourth user input corresponding to a downward direction on the display unit, the focus of the third user interface 626 may be switched from media item 623 in Figure 6H to media item 627 in Figure 6K, which is positioned downward relative to media item 623. Specifically, the position of cursor 624 may be changed from media item 623 to media item 627.

[0177] At block 554 of process 500, a selection of a media item of the one or more media items may be received via the second user interface or the third user interface. For example, with reference to FIG. 6J , a selection of media item 625 may be received via second user interface 618 by detecting user input corresponding to the user selection while cursor 624 is positioned on media item 625. Similarly, with reference to FIG. 6K , a selection of media item 627 may be received via third user interface 626 by detecting user input corresponding to the user selection while cursor 624 is positioned on media item 627. In response to receiving a selection of a media item of the one or more media items, block 556 may be executed.

[0178] At block 556 of process 500, media content associated with the selected media item may be displayed on the display unit. In some embodiments, the media content may be a movie, video, television program, animation, or the like that is playing on or streaming through the media device. In some embodiments, the media content may be a video game, e-book, application, or program running on the media device. Additionally, in some embodiments, the media content may be information related to the media item. The information may be product information that describes various characteristics of the selected media item (e.g., plot, cast, director, author, release date, rating, duration, etc.).

[0179] At block 558 of process 500, and with reference to FIG. 5G, a fifth user input may be detected. In some examples, the fifth user input may be detected while displaying a third user interface (e.g., third user interface 626). In these examples, the fifth user input may be detected while the focus of the third user interface is on a media item in the top row of the third user interface (e.g., one of media items 622 in third user interface 626 of FIG. 6H). In other examples, the fifth user input may be detected while displaying the first user interface. In these examples, the fifth user input may be detected while performing any one of blocks 508-514. In some examples, the fifth user input may be detected on a remote control of the media device. The fifth user input may be similar to or identical to the third user input. For example, the fifth user input may include a continuous contact motion in the second direction on the touch-sensitive surface (e.g., a swipe-up contact motion). In other examples, the fifth user input can be activation of an affordance. The affordance can be associated with a virtual keyboard interface or a typed search interface. In response to detecting the fifth user input, one or more of blocks 560-564 can be executed.

[0180] At block 560 of process 500, a search field configured to receive typed search input may be displayed. For example, as shown in FIG. 6L, search field 644 may be displayed on the displayed unit. In some embodiments, the search field may be configured to receive a typed search query. The typed search query may be a media-related search query, such as a search for a media item. In some embodiments, the search field may be configured to perform a media-related search based on a text string match between text entered via search field 644 and stored text associated with the media item. Furthermore, in some embodiments, the digital assistant may not be configured to receive input via search field 644. This may encourage the user to interact with the digital assistant via a speech interface rather than a typed interface, promoting a more pleasant interface between the media device and the user. It should be appreciated that in some embodiments, the search field may be inherently displayed within the second user interface (e.g., second user interface 618) or the third user interface (e.g., third user interface 626). In these embodiments, block 566 may not need to be performed.

[0181] At block 562 of process 500, a virtual keyboard interface may be displayed on the display unit. For example, as shown in FIG. 6L, virtual keyboard interface 646 may be displayed. Virtual keyboard interface 646 may be configured such that user input received through virtual keyboard interface 646 results in text entry into a search field. In some embodiments, the virtual keyboard interface may not be used to interact with the digital assistant.

[0182] At block 564 of process 500, focus of the user interface may be switched to a search field. For example, referring to FIG. 6L, at block 568, search field 644 may be highlighted. Additionally, a text entry cursor may be positioned within search field 644. In some implementations, text may be displayed within the search field prompting the user to enter a typed search. As shown in FIG. 6L, text 648 includes the prompt "Type a search."

[0183] At block 566 of process 500, and with reference to FIG. 5H, a seventh user input may be detected. In some embodiments, the seventh user input may be detected while displaying a third user interface (e.g., third user interface 626). In some embodiments, the seventh user input may include pressing a button on a remote control of the electronic device. The button may be, for example, a menu button for navigating to a main menu user interface of the electronic device. It should be appreciated that in other embodiments, the seventh user input may include other types of user input. In response to detecting the seventh user input, block 568 may be executed.

[0184] At block 568 of process 500, displaying the third user interface on the display unit may cease. Specifically, the seventh user input may cause the third user interface to close. In some embodiments, the seventh user input may cause a main menu user interface menu to be displayed instead of the third user interface. Alternatively, in embodiments in which media content (e.g., media content 602) was displayed prior to displaying the third user interface (e.g., third user interface 626) and playback of the media content on the electronic device was paused (e.g., paused in response to detecting the third user input) upon displaying the third user interface, playback of the media content on the electronic device may be resumed in response to detecting the seventh user input. Thus, in response to detecting the seventh user input, the media content may be displayed.

[0185] At block 570 of process 500 and with reference to FIG. 5I, a sixth user input can be detected. As shown in FIG. 6M, the sixth user input can be detected while displaying the third user interface 626. However, in other embodiments, the sixth user input can alternatively be detected while displaying the second user interface (e.g., second user interface 618). When the sixth user input is detected, the second user interface or the third user interface can include a portion of a result that at least partially satisfies the user request. The sixth user input can include an input for invoking a digital assistant of the electronic device. Specifically, the sixth user input can be similar to or identical to the user input of the second input type described above with reference to block 516. For example, the sixth user input can include pressing a particular button on a remote control of the media device and holding the button down for longer than a predetermined period (e.g., a long press). In response to detecting the sixth user input, one or more of blocks 572-592 can be executed.

[0186] At block 572 of process 500, second audio data can be sampled. Block 572 can be similar to or identical to block 518 described above. Specifically, the sampled second audio data can include a second user utterance from a user. The second user utterance can represent a second user request directed to the digital assistant. In some embodiments, the second user request can be a request to perform a second task. For example, with reference to FIG. 6M, the sampled second audio data can include the second user utterance, "only anything with Luke Wilson in it." In this embodiment, the second user utterance can represent a second user request to narrow a previous media search to include only media items in which Luke Wilson appears as an actor. In this embodiment, the second user utterance is in a natural language format. Furthermore, the second user request can be incompletely specified. In this case, the second user utterance does not clearly specify all information necessary to define the user request. For example, the second user utterance does not clearly specify what "the ones" refers to. In other embodiments, the second user request can be a request to play a media item or to provide specific information (eg, weather, stock prices, sports, etc.).

[0187] It should be appreciated that in some embodiments, blocks 520-526 described above can be similarly performed for a sixth user input. Specifically, as shown in FIG. 6M, an active visual indicator 614 can be displayed on the display unit simultaneously with detecting the sixth user input. A second text representation 650 of the second user utterance can be determined (e.g., using the STT processing module 430) and displayed on the display unit. A second user intent corresponding to the second user utterance can be determined (e.g., using the natural language processing module 432) based on the second text representation. In some embodiments, as shown in FIG. 6M, content displayed on the display unit when the sixth user input is detected can be faded or reduced in brightness in response to detecting the sixth user input. This can help make the active visual indicator 614 and the second text representation 650 more noticeable.

[0188] At block 574 of process 500, a determination may be made as to whether the sampled second voice data contains a second user request. Block 574 may be similar to or identical to block 528 described above. Specifically, the determination at block 574 may be made based on a second user intent determined from a second text representation of the second user utterance. Block 576 may be executed in accordance with a determination that the second voice data does not contain a user request. Alternatively, one or more of blocks 578-592 may be executed in accordance with a determination that the second voice data contains a second user request.

[0189] A request for clarification of the user's intent may be displayed on the display unit at block 576 of process 500. Block 576 may be similar to or identical to block 530 described above.

[0190] At block 578 of process 500, a determination may be made as to whether the second user request is a request to narrow the results of the user request. In some embodiments, the determination may be made from a second user intent corresponding to the second user utterance. Specifically, the second user request may be determined to be a request to narrow the results of the user request based on an explicit instruction to narrow the results of the user request identified within the second user utterance. For example, with reference to FIG. 6M, the second text representation 650 may be parsed during natural language processing to determine whether the second user utterance includes predetermined words or phrases corresponding to an explicit intent to narrow the media search results. Examples of words or phrases corresponding to an explicit intent to narrow the media search results may include "just," "only," "filter by," and the like. Therefore, based on the word "just" in the second text representation 650, the second user request may be determined to be a request to narrow media search results associated with the user request, "Find romantic comedies starring Reese Witherspoon." It should be appreciated that other techniques may be implemented for determining whether the second user request is a request to narrow the results of the user request. In accordance with a determination that the second user request is a request to narrow the results of the user request, one or more of blocks 580-582 may be executed.

[0191] At block 580 of process 500, a subset of results that at least partially satisfy the user request may be obtained. In some embodiments, the subset of results may be obtained by filtering existing results according to additional parameters defined in the second user request. For example, the results obtained in block 534 (e.g., including media item 622) may be filtered to identify media items featuring Luke Wilson as an actor. In other embodiments, a new media search query may be performed that combines the requirements of the user request and the second user request. For example, the new media search query may be a search query for media items in the romantic comedy genre and with actors Reese Witherspoon and Luke Wilson. In this embodiment, the new media search query may yield media items such as "Legally Blonde" and "Legally Blonde 2."

[0192] In embodiments in which a sixth user input is detected while displaying the third user interface, additional results related to the user request and / or the second user request may be obtained. The additional results may include media items having one or more attributes or parameters stated in the user request and / or the second user request. Furthermore, the additional results may not include all attributes or parameters stated in the user request and the second user request. For example, with reference to the embodiments shown in FIGS. 6H and 6M, the additional results may include media items having at least one (but not all) of the following attributes or parameters: romantic comedy, Reese Witherspoon, and Luke Wilson. The additional results may be desirable to provide the user with a broader set of results and more options to choose from. Furthermore, the additional results may be related results that are more likely to interest the user.

[0193] At block 582, a subset of the results may be displayed on the display unit. For example, as shown in FIG. 6N, the subset of results may include media items 652, which may include movies such as "Legally Blonde" and "Legally Blonde 2." In this example, the media items 652 are displayed within the top row of the third user interface 626. A text header 656 may describe attributes or parameters associated with the displayed media items 652. Specifically, the text header 656 may include a restatement of the user's intent associated with the second user utterance. In examples where a sixth user input is detected while displaying a second user interface (e.g., the second user interface 618 shown in FIG. 6G), the media items 652 may instead be displayed within the second user interface. In these examples, the media items 652 may be displayed as a single column across the second user interface. It should be appreciated that the manner in which the media items 652 are displayed within the second user interface or the third user interface may vary.

[0194] In an embodiment in which a sixth user input is detected while displaying the third user interface, additional results related to the user request and / or the second user request can be displayed within the third user interface. For example, referring to FIG. 6N, the additional results can include media items 654 having one or more parameters stated in the user request and / or the second user request. Specifically, media items 654 can include media item 658, which is a romantic comedy starring Luke Wilson, and media items 660, which star Luke Wilson and were released within the last 10 years. Each set of media items (e.g., media items 658, 660) can be labeled with a text header (e.g., text headers 662, 664). The text headers can describe one or more parameters associated with the respective set of media items. The text headers can be in natural language format. Furthermore, each text header can be an exemplary user utterance that, when provided by the user to the digital assistant, can cause the digital assistant to retrieve a similar set of media items. For example, referring to text header 662, in response to receiving the user statement "A romantic comedy starring Luke Wilson" from the user, the digital assistant can retrieve and display a media item (e.g., media item 658) that is a romantic comedy starring Luke Wilson.

[0195] Referring again to block 578, it may be determined that the second user request is not a request to narrow the results of the user request. Such a determination may be made based on the absence of any explicit instruction in the second user utterance to narrow the results of the user request. For example, when parsing the second text representation of the second user utterance during natural language processing, a predetermined word or phrase corresponding to an explicit intent to narrow the media search results may not be identified. This may be because the second user request is a request unrelated (e.g., a new request) to a previous user request. For example, the second user request may be "Find me a horror movie," which is a request unrelated to the previous user request of "Find me a romantic comedy starring Reese Witherspoon." Alternatively, the second user request may include ambiguous language that may be interpreted as either a request to narrow the results of the previous user request or a new request unrelated to the previous user request. For example, referring to FIG. 6P, the second user utterance may be "Luke Wilson." This may be interpreted as either a request to narrow the results of a previous user request (e.g., a request to narrow the results to include only media items in which Luke Wilson appears as an actor) or a new request unrelated to the previous user request (e.g., a new media search for media items in which Luke Wilson appears as an actor). In these examples, it may be determined that the second user request is not a request to narrow the results of the user request. In accordance with a determination that the second user request is a request to narrow the results of the user request, one of more of blocks 584-592 may be performed.

[0196] In block 584 of process 500, a second task may be performed to at least partially satisfy the second user request. Block 584 may be similar to block 532 described above, except that the second task of block 584 may be different from the task of block 532. Block 584 may include one or more of blocks 586-588.

[0197] At block 586 of process 500, a third result that at least partially satisfies the second user request may be obtained. Block 586 may be similar to block 534 described above. With reference to the example shown in FIG. 6P, the second user utterance "Luke Wilson" may be interpreted as a request to perform a new media search query to identify media items featuring Luke Wilson as an actor. Thus, in this example, block 586 may include performing the requested media search and retrieving media items featuring Luke Wilson as an actor. It should be appreciated that in other examples, the user request may include a request for other types of information (e.g., weather, sports, stock prices, etc.), and each type of information may be retrieved at block 586.

[0198] At block 588 of process 500, a portion of the third result may be displayed on a display unit. For example, referring to FIG. 6Q, a third result including media items 670 featuring Luke Wilson as an actor (e.g., films such as "Playing It Cool," "The Skeleton Twins," and "You Kill Me") may be displayed within third user interface 626. In this example, media items 670 may be displayed within a top row of third user interface 626. Text headers 678 may describe attributes associated with the displayed media items 670. Specifically, text headers 678 may include a restatement of the determined user intent associated with the second user utterance. In examples where a sixth user input is detected while displaying a second user interface (e.g., second user interface 618 shown in FIG. 6G), media items 670 may be displayed within the second user interface. In these examples, media items 670 may be displayed within a single column across the second user interface. It should be appreciated that in other embodiments, the organization or configuration of media items 670 within the second or third user interface may differ.

[0199] At block 590 of process 500, a fourth result that at least partially satisfies the user request and / or the second user request may be obtained. Specifically, the fourth result may include media items having one or more attributes or parameters defined in the user request and / or the second user request. With reference to the example shown in FIGS. 6P and 6Q, the fourth result may include media items having one or more of the following attributes or parameters: romantic comedy, Reese Witherspoon, and Luke Wilson. For example, the fourth result may include media item 676 having a genre of romantic comedy and starring Luke Wilson. Obtaining the fourth result may be desirable to provide the user with a broader set of results and, therefore, more options from which to choose. Furthermore, the fourth result may be associated with an alternative predicted user intent derived from the second user request and one or more previous user requests to increase the likelihood that the user's actual intent will be satisfied. This may help increase the accuracy and relevance of the results returned to the user, thereby improving the user experience.

[0200] In some examples, at least a portion of the fourth result may include media items having all of the parameters defined in the user request and the second user request. For example, the fourth result may include media item 674 having a genre of romantic comedy and starring Reese Witherspoon and Luke Wilson. Media item 674 may be associated with an alternative intent to refine the results of a previous user request using the second user request. If the user actually intended the second request to be a request to refine the retrieved results, retrieving media item 674 may be desirable to increase the likelihood that the user's actual intent will be satisfied.

[0201] In some examples, a portion of the fourth results may be based on the focus of the user interface at the time the sixth user input is detected. Specifically, the focus of the user interface may be on one or more items in the third user interface when the sixth user input is detected. In this example, the portion of the fourth results may be contextually related to the one or more items on which the user interface is focused. For example, with reference to FIG. 6K , cursor 624 may be positioned on media item 627, and therefore, the focus of third user interface 626 may be on media item 627. In this example, attributes or parameters associated with media item 627 may be utilized to obtain a portion of the fourth results. For example, a category of “Reese Witherspoon movies” associated with media item 627 may be utilized to obtain a portion of the fourth results, and the resulting portion may include media items starring both Reese Witherspoon and Luke Wilson. In another example, media item 627 may be an adventure movie, and therefore, a portion of the fourth results may include media items that are adventure movies starring Luke Wilson.

[0202] At block 592 of process 500, a portion of the fourth results may be displayed. In embodiments in which a sixth user input is detected while displaying the third user interface, a portion of the fourth results may be displayed within the third user interface. For example, as shown in FIG. 6Q, the portion of the fourth results may include media item 672 displayed within a subsequent column of media items 670. Media item 672 may be associated with one or more of the attributes or parameters defined within the second user request and / or the user request (e.g., romantic comedy, Reese Witherspoon, and Luke Wilson). For example, media item 672 may include media item 676, which is a romantic comedy starring Luke Wilson, and media item 674, which is a romantic comedy starring both Reese Witherspoon and Luke Wilson. Each set of media items (e.g., media items 674, 676) may be labeled with a text header (e.g., text headers 680, 682). The text headers can describe one or more attributes or parameters associated with each set of media items. The text headers can be in natural language format. Furthermore, each text header can be an example user utterance that, when provided to the digital assistant by the user, can cause the digital assistant to retrieve a similar set of media items with similar attributes.

[0203] As described above, the second user utterance of "Luke Wilson" can be associated with two possible user intents: a first user intent to conduct a new media search or a second user intent to refine the results of a previous user request. The displayed media item 670 can satisfy the first user intent, and the displayed media item 674 can satisfy the second user intent. In this example, media items 670 and 674 are displayed within the top two rows. In this manner, results for the two most likely user intents associated with the second user request (e.g., a new search or a refinement of a previous search) can be displayed prominently (e.g., within the top two rows) within the third user interface 626. This may be desirable to minimize scrolling or browsing by a user within the third user interface until locating a desired media item to consume. It should be appreciated that there may be various ways in which media items 670 and 674 can be displayed prominently within the third user interface 626 to minimize scrolling and browsing.

[0204] 7A-7C illustrate a process 700 for operating a digital assistant in a media system, according to various embodiments. Process 700 can be performed using one or more electronic devices implementing a digital assistant. For example, process 700 can be performed using one or more of system 100, media system 128, media device 104, user device 122, or digital assistant system 400, described above. FIGS. 8A-8W illustrate screenshots displayed on a display unit by a media device at various stages of process 700, according to various embodiments. Process 700 is described below with simultaneous reference to FIGS. 7A-7C and 8A-8W. It should be understood that some operations in process 700 can be combined, the order of some operations can be changed, and some operations can be omitted.

[0205] At block 702 of process 700, content may be displayed on a display unit (e.g., display unit 126). Block 702 may be similar to or identical to block 502 described above. Referring to FIG. 8A , the displayed content may include media content 802 (e.g., a movie, a video, a television program, a video game, etc.) being played on a media device (e.g., media device 104). In other examples, the displayed content may include other content, such as content associated with an application running on the media device or a user interface for interacting with a digital assistant of the media device. Specifically, the displayed content may include a main menu user interface or a user interface having objects or results previously requested by the user.

[0206] At block 704 of process 700, user input may be detected. Block 704 may be similar to or identical to block 504 described above. The user input may be used to invoke a digital assistant on the media device. In some embodiments, the user input may be detected while the content of block 702 is being displayed. The user input may be detected on a remote control device for the media device (e.g., remote control device 124). For example, the user input may correspond to the second form of input described at block 516 of process 500. Specifically, the user input at block 704 may include pressing a particular button on the remote control device for the media device and holding the button down for longer than a predetermined period (e.g., a long press). In response to detecting the user input, one or more of blocks 706-746 may be executed.

[0207] At block 706 of process 700, audio data may be sampled. Block 706 may be similar to or identical to block 518 described above. The sampled audio data may include a user utterance. The user utterance may represent a user request directed to the digital assistant of the media device. For example, with reference to the example shown in FIG. 8A, the sampled audio data may include the user utterance, "What time is it in Paris?" The user utterance may be in the form of unstructured natural language. In some examples, the request expressed by the user utterance may be underspecified. In this case, information necessary to fulfill the request is missing or not explicitly qualified within the user utterance (e.g., "Play this"). In other examples, the user utterance may not be an explicit request, but rather an indirect question or statement from which a request is inferred (e.g., "What did he say?"). Furthermore, as described in more detail below in block 712, the user utterance may include one or more ambiguous terms.

[0208] At block 708 of process 700, a text representation of the user utterance in the sampled audio data may be determined. Block 708 may be similar to or identical to block 522 described above. Specifically, the text representation may be determined by performing STT processing on the user utterance in the sampled audio data. For example, with reference to FIG. 8A , a text representation 804, "What time is it in Paris?" may be determined from the user utterance in the sampled audio data and displayed on a display unit. As shown, the text representation 804 may be overlaid on the media content 802 while the media content 802 continues to play on the media device.

[0209] In some embodiments, the STT process used to determine the text representation can be biased toward media-related text results. Additionally, or alternatively, the text representation can be based on a previous user utterance received by the media device prior to sampling the audio data. Furthermore, in some embodiments, the text representation can be based on a time at which the previous user utterance was received prior to sampling the audio data. In embodiments in which the text representation is obtained from a separate device (e.g., DA server 106), the media device can indicate to the separate device that the sampled audio data is associated with a media application, and this indication can bias the STT process on the separate device toward media-related text results.

[0210] At block 710 of process 700, a user intent corresponding to the user utterance may be determined. Block 710 may be similar to block 526 described above. Specifically, natural language processing (e.g., by natural language processing module 432) may be used to process the text representation of block 708 to derive the user intent. For example, with reference to FIG. 8A , from text representation 804, “What time is it in Paris?”, it may be determined that the user intent is to request the time in a location named “Paris.” The natural language processing used to determine the user intent may be biased toward media-related user intent. In embodiments where the user intent is obtained from a separate device (e.g., DA server 106), the media device may indicate to the separate device that the sampled voice data is associated with a media application, and this indication may bias the natural language processing on the separate device toward media-related user intent.

[0211] In some examples, user intent can be determined based on prosodic information derived from user utterances in the sampled audio data. Specifically, prosodic information (e.g., tonality, rhythm, volume, stress, intonation, rate, etc.) can be derived from the user utterances to determine the user's attitude, mood, emotion, or feeling. The user intent can then be determined from the user's attitude, mood, emotion, or feeling. For example, the sampled audio data can include the user utterance, "What did he say?" In this example, it can be determined that the user is annoyed or frustrated based on the high volume and stress detected in the user utterance. Based on the user utterances and the determined user emotion, it can be determined that the user intent includes a request to increase the volume of audio associated with media content being played on the media device.

[0212] 7A, block 710 may include one or more of blocks 712-718. Specifically, one or more of blocks 712-718 may be executed if two or more user intents are found to be highly probable and the natural language processing module is unable to narrow down the two or more user intents to a single user intent. For example, this situation may arise when a user utterance contains ambiguous terms that cannot be disambiguated based on available context information.

[0213] At block 712 of process 700, a determination may be made as to whether the user utterance (or a textual representation of the user utterance) includes an ambiguous term. The determination may be made during natural language processing (e.g., using natural language processing module 432) to determine user intent. An ambiguous term may be a word or phrase that has more than one possible interpretation. For example, with reference to FIG. 8A , the term “Paris” in the user utterance “What time is it in Paris?” may be interpreted as the city of Paris, France, or the city of Paris, Texas, USA. Therefore, the term “Paris” in the user utterance may be determined to be an ambiguous term.

[0214] In some examples, context information can be retrieved (e.g., by a digital assistant) to disambiguate potentially ambiguous terms. If disambiguation is successful, it can be determined that the user utterance does not include an ambiguous term. For example, it can be determined that the media content 802 is a movie (e.g., "Ratatouille") that has Paris, France as its setting, and therefore, the user is more likely to be referring to Paris, France than Paris, Texas. In this example, the term "Paris" can be successfully disambiguated to refer to Paris, France, and therefore, it can be determined that the user utterance does not include an ambiguous term.

[0215] In another example, the user utterance can be "play this." In this example, the user utterance does not explicitly limit the specific media item to be played, and therefore, the term "this" can be an ambiguous term that, when interpreted alone, could refer to any media item available to the media device. The term can be disambiguated using contextual information displayed on the display unit by the media device. For example, the digital assistant can determine whether the focus of the displayed user interface is on a media item. Following a determination that the focus of the user interface is on a media item, the digital assistant can disambiguate the term "this" and determine that the term refers to the media item on which the displayed user interface is focused. Based on this determination, in block 712, it can be determined that the user utterance does not include an ambiguous term. Therefore, it can be determined that the user intent is a request to play the media item on which the displayed user interface is focused.

[0216] In examples where terms cannot be disambiguated, a determination may be made at block 712 that the user utterance includes an ambiguous term. In response to determining that the user utterance includes an ambiguous term, one or more of blocks 714-718 may be performed. At block 714 of process 700, two or more candidate user intents may be obtained based on the ambiguous term. The two or more candidate user intents may be most likely candidate user intents determined from the user utterance that cannot be disambiguated. With reference to the example shown in FIG. 8A , the two or more candidate user intents may include a first candidate user intent to request the time in Paris, France and a second candidate user intent to request the time in Paris, Texas.

[0217] At block 716 of process 700, two or more candidate user intents may be displayed on a display unit for user selection. For example, referring to FIG. 8B , a first candidate user intent 810 and a second candidate user intent 808 may be displayed. Additionally, a text prompt 806 may be provided to prompt the user to indicate the actual user intent corresponding to the user utterance by selecting between the first candidate user intent 810 and the second candidate user intent 808. The text prompt 806, the first candidate user intent 810, and the second candidate user intent 808 may be superimposed on the media content 802.

[0218] At block 716 of process 700, a user selection of one of two or more candidate user intentions may be received. In some examples, the user selection may be received via a selection of an affordance corresponding to one of the candidate user intentions. Specifically, as shown in FIG. 8B , each of the two or more candidate user intentions (810, 808) may be displayed on a display unit as a selectable affordance. The media device may receive input from a user (e.g., via a remote control device for the media device) to change focus of the display to one of the affordances. Thereafter, a user selection of a candidate user intention corresponding to that affordance may be received (e.g., via a remote control device for the media device). For example, as shown in FIG. 8B , the media device may receive a user input to move a cursor 812 over an affordance corresponding to a first candidate user intention 810 (e.g., Paris, France). Thereafter, a user selection of the first candidate user intention 810 may be received.

[0219] In other examples, the user selection can be received via a voice interaction with the digital assistant. For example, a second user input can be detected while displaying two or more candidate user intents. The second user input can be similar to or identical to the user input of block 704. Specifically, the second user input can be an input for invoking the digital assistant (e.g., pressing a specific button on a remote control of the media device and holding the button for longer than a predetermined period of time). In response to detecting the second user input, second audio data can be sampled. The second audio data can include a second user utterance representing a user selection of one of two or more interpretations. For example, with reference to FIG. 8C , the second audio data can include the second user utterance "Paris, France." As shown, a text representation 814 of the second user utterance "Paris, France" can be displayed on the display unit. In this example, the second user utterance "Paris, France" can represent a user selection of the first candidate user intent 810 (e.g., Paris, France). Based on the second user utterance, "Paris, France," it may be determined that the first candidate user intent 810 is the actual user intent corresponding to the user utterance, "What time is it in Paris?" Therefore, in block 710, it may be determined that the user intent is to request the time in Paris, France. Upon determining the user intent based on the received user selection, one or more of blocks 720-746 may be performed.

[0220] In some embodiments, blocks 710-718 can be performed without outputting speech from the media device. Specifically, the text prompt 806 and the candidate user intents 808, 810 can be displayed without outputting speech associated with two or more of the candidate user intents 808, 810. Thus, input from the user can be received in the form of speech, but the digital assistant's output can be presented to the user visually (and not in audio form) on the display unit. This can be desirable to preserve the communal experience associated with consuming media content, thereby improving the user experience of the media device.

[0221] Referring back to block 712, in response to determining that the user utterance does not include an ambiguous term, one or more of blocks 720-746 may be executed. At block 720 of process 700, a determination may be made as to whether the user intent corresponds to one of a plurality of core competencies associated with the media device. For example, the media device may be associated with several predetermined core competencies, such as searching for media items, playing media items, and providing information related to media items, weather, stock prices, and sports. If the user intent involves performing a task related to one of the several predetermined core competencies, the user intent may be determined to correspond to one of the several predetermined core competencies. For example, if the user intent is a request for a media item starring Reese Witherspoon, the user intent may be determined to correspond to one of the several predetermined core competencies. In response to determining that the user intent corresponds to one of the several core competencies associated with the electronic device, one or more of blocks 724-746 may be executed.

[0222] Conversely, if the user intent involves performing a task other than one of the predetermined core capabilities, it may be determined that the user intent does not correspond to one of the predetermined core capabilities. For example, if the user intent is a request for map guidance, it may be determined that the user intent does not correspond to one of the predetermined core capabilities. In response to determining that the user intent does not correspond to one of the plurality of core capabilities associated with the electronic device, block 722 may be executed.

[0223] At block 722 of process 700, a second electronic device (e.g., device 122) may be caused to at least partially satisfy the user intent. Specifically, the second electronic device may be caused to perform a task to facilitate satisfying the user intent. In one example, it may be determined that the media device is not configured to fulfill the user intent to request map directions, and therefore the user intent may be transmitted to the second electronic device for fulfilling the user intent. In this example, the second user device may perform a task to display the requested map directions. In other examples, information other than the user intent may be transmitted to the second electronic device to cause the second electronic device to perform a task to facilitate satisfying the user intent. For example, the digital assistant of the media device may determine a task flow or structured query to fulfill the user intent (e.g., using the natural language processing module 432 or the task flow processing module 436), and the task flow or structured query may be transmitted to the second electronic device. The second electronic device may then execute the task flow or structured query to facilitate satisfying the user intent.

[0224] As will become apparent in the description provided below, the level of intrusiveness associated with satisfying a user intent can be based on the nature of the user intent. In some cases, a task associated with satisfying a user intent can be performed without displaying any additional response or output on the display (e.g., block 726). In other cases, only a text response (e.g., without a corresponding visual or audio output) can be provided to satisfy the user intent (e.g., block 732). In still other cases, a user interface with relevant results can be displayed to satisfy the user intent (e.g., blocks 738, 742, or 746). The user interface can occupy a majority or less than a majority of the display unit. Thus, process 700 can intelligently adjust the level of intrusiveness of the output depending on the nature of the user intent. This enables convenient access to the digital assistant's services while reducing unwanted interruptions during media content consumption, thereby improving the overall user experience.

[0225] At block 724 of process 700, a determination may be made as to whether the user intent includes a request to adjust a state or setting of an application on the media device. In response to determining that the user intent includes a request to adjust a state or setting of an application on the media device, block 726 may be executed. At block 726 of process 700, the state or setting of the application may be adjusted to satisfy the user intent.

[0226] In some embodiments, a state or setting may be associated with displayed media content being played on a media device. For example, a request to adjust a state or setting of an application may include a request to control playback of media content by a media device. Specifically, it may include a request to pause, resume, restart, stop, rewind, or fast-forward playback of displayed media content on a media device. It may also include a request to skip forward or backward (e.g., by a specified duration) within the media content to play a desired portion of the media content. Furthermore, a request to adjust a state or setting of an application may include a request to turn on / off subtitles or closed captions (e.g., in a specified language) associated with the displayed media content, increase / decrease the volume of audio associated with the displayed media content, mute / unmute audio associated with the displayed media content, or speed up / slow down the speed at which the displayed media content is played.

[0227] 8E-8F show an example embodiment of a user intent including a request to control playback of media content by a media device. In this embodiment, a digital assistant can be invoked (e.g., at block 704) while playing media content 802. The media content can initially be displayed without displaying subtitles. Sampled audio data (e.g., at block 706) can include a user utterance, "Please turn on English subtitles." As shown in FIG. 8E, a text representation 816 of the user utterance can be displayed on a display unit. Based on the user utterance, at block 710, it can be determined that the user intent includes a request to turn on display of English subtitles for the media content 802. Further, at block 724, it can be determined that the user intent is a request to adjust a state or setting of an application on the electronic device. In response to this determination, English subtitles for the media content 802 can be turned on. As represented by label 817 in FIG. 8F, display of English subtitles associated with the media content 802 can be initiated to satisfy the user intent.

[0228] In another example embodiment shown in FIGS. 8G-8H, the user utterances in the sampled audio data can be natural language expressions indicating that the user could not hear a portion of the audio associated with the media content. Specifically, as shown by text representation 820 in FIG. 8G, the user utterance can be, "What did he say?" In this example, it can be determined (e.g., at block 710) that the user intent includes a request to replay the portion of the media content corresponding to the portion of the audio that the user could not hear. It can also be determined that the user intent includes a request to turn on closed captions to assist with hearing difficulties associated with the media content. Furthermore, based on prosodic information in the user utterances, it can be determined that the user is frustrated or annoyed, and therefore, based on the user's emotion, it can be determined that the user intent includes a request to increase the volume of the audio associated with the media content. In block 724, it can be determined that these user intents are requests to adjust the state or settings of an application of the electronic device. In response to this determination, the media content may be rewound a predetermined period (e.g., 15 seconds) to an earlier portion of the media content (e.g., as represented by label 822 in FIG. 8H), and playback of the media content may be restarted from this earlier portion. Additionally, closed captioning may be turned on (e.g., as represented by label 824 in FIG. 8H) before restarting playback of the media content from the earlier portion. Furthermore, the volume of audio associated with the media content may be increased before restarting playback of the media content from the earlier portion.

[0229] It should be understood that closed captions or subtitles associated with media content can be obtained from a service provider (e.g., a cable provider or a media subscription service). However, in embodiments in which closed captions or subtitles are not available from a service provider, a media device can generate closed captions or subtitles to assist those with hearing difficulties associated with the media content. For example, prior to receiving user utterances in the sampled audio data and while the media content is playing, speech in the audio associated with the media content can be continuously converted to text (e.g., using the STT processing module 730) and stored in association with the media content. In response to a user request to replay an earlier portion of the media content that the user was unable to hear, while the earlier portion of the media content is being replayed, text corresponding to the earlier portion being replayed can be retrieved and displayed.

[0230] In some examples, a state or setting associated with the displayed media content can be adjusted without displaying an additional user interface for performing the adjustment or without providing any text or graphic that conveys confirmation that the state or setting has been adjusted. For example, in the illustrated examples of FIGS. 8E-8H, subtitles (or closed captions) can simply be turned on without explicitly displaying text such as "Subtitles turned on" or displaying a user interface for controlling the display of subtitles. Furthermore, a state or setting can be adjusted without outputting any audio associated with satisfying the user intent. For example, in FIGS. 8E-8H, subtitles (or closed captions) can be turned on without outputting any audio (e.g., speech or non-verbal audio signals) confirming that the subtitles have been turned on. Therefore, the requested action can simply be performed without any additional audible or visual interruption of the media content. In this manner, process 700 can minimize interruptions to the user's consumption of media content while providing convenient access to the services of a digital assistant, thereby improving the user experience.

[0231] In other examples, the request to adjust the state or settings of an application on the media device may include a request to navigate within a user interface of the media device (e.g., the second user interface 818, the third user interface 826, or the main menu user interface). In one example, the request to navigate within a user interface may include a request to switch focus of the user interface from a first object (e.g., a first media item) to a second object (e.g., a second media item) within the user interface. FIGS. 8I-8K show one illustrative example of such a request. As shown in FIG. 8I, the displayed content may include a third user interface 826 having a plurality of media items organized into various categories (e.g., "Romantic Comedies," "Romantic Comedies Starring Reese Witherspoon," and "Movies with Luke Wilson"). As indicated by the position of the cursor 828, the focus of the third user interface 826 may be on a first media item 830 under the "Romantic Comedies" category. The second media item 832 may have the title "Legally Blonde" and may be located under the category of "Romantic Comedies Starring Reese Witherspoon." As shown by text representation 834 in FIG. 8J, the user utterance in the sampled audio data (e.g., in block 706) may be "Go to Legally Blonde." Based on this user utterance, it may be determined (e.g., in block 710) that the user intent is a request to switch the focus of the third user interface 826 from the first media item 830 to the second media item 832 having the title "Legally Blonde."In response to determining (e.g., in block 724) that the user intent is a request to adjust a state or setting of an application on the electronic device, the focus of the third user interface 826 may be switched from the first media item 830 to the second media item 832. For example, as shown in FIG. 8K, the position of the cursor 828 may be changed from the first media item 830 to the second media item 832.

[0232] In another example, a request to navigate within a user interface may include a request to change focus of the user interface to a particular category of results displayed within the user interface. For example, FIG. 8I includes media items associated with the categories of “Romantic Comedies,” “Romantic Comedies Starring Reese Witherspoon,” and “Movies with Luke Wilson.” Rather than “Go to Legally Blonde,” the user utterance in the sampled speech data may instead be “Go to Romantic Comedies Starring Reese Witherspoon.” Based on this user utterance, it may be determined (e.g., in block 710) that “Romantic Comedies Starring Reese Witherspoon” defines a category of media items displayed within the third user interface 826, and therefore, the user intent may be determined to be a request to change focus of the user interface to one or more media items associated with that category. In response to determining (e.g., in block 724) that the user intent is a request to adjust a state or setting of an application on the electronic device, the focus of the third user interface 826 may be shifted to one or more media items associated with the category. For example, as shown in FIG. 8K, the position of the cursor 828 may be shifted to a second media item 832 associated with "Romantic Comedies Starring Reese Witherspoon."

[0233] In yet another example, a request to navigate within a user interface of a media device can include a request to select an object within the user interface. Selecting an object can cause an action associated with the object to be performed. For example, as shown in FIG. 8K, cursor 828 is located on a second media item 832 having the title "Legally Blonde." As shown in FIG. 8L, a digital assistant can be invoked (e.g., in block 704), and a user utterance in the sampled audio data (e.g., in block 706) can be "Play this" (e.g., displayed as text representation 836). Based on this user utterance, it can be determined (e.g., in block 710) that the user intent is a request to play a specific media item. In this example, the user utterance does not explicitly qualify or identify the specific media item to be played. Specifically, the word "this" is ambiguous. However, the digital assistant can obtain contextual information to disambiguate the user intent. For example, it can be determined that the focus of third user interface 826 is on second media item 832 at the time the audio data was sampled. Based on this determination, the second media item 832 may be identified as the media item to be played. In response to determining (e.g., in block 724) that the user intent to play the second media item 832 is a request to adjust a state or setting of an application on the electronic device, action may be performed to facilitate playing the second media item 832. For example, preview information about the second media item 832 may be displayed on a display unit. The preview information may include, for example, a plot summary, a cast list, publicity data, user ratings, and the like.Additionally, or alternatively, the second media item 832 can be played on the media device, and media content associated with the second media item 832 can be displayed on the display unit (e.g., represented by text 838 "Legally Blonde Playing" in FIG. 8M). It should be appreciated that in other embodiments, the media item to be selected can be explicitly identified. For example, rather than "play this," the user utterance can specifically state "play Legally Blonde," and similar actions can be performed to facilitate playing the second media item 832.

[0234] In yet another example, a request to navigate within a user interface of a media device may include a request to view a particular user interface or application of the media device. For example, a user utterance in the sampled audio data may be, "Go to the actor page." In this case, the user intent may include a request to display a user interface associated with browsing for media items related to a particular actor. In another example, a user utterance in the sampled audio data may be, "Take me to the home page." In this case, the user intent may include a request to display a main menu user interface of the media device. In yet another example, a request to navigate within a user interface of a media device may include a request to launch an application on the electronic device. For example, a user utterance in the sampled audio data may be, "Go to the iTunes store." In this case, the user intent may include a request to launch the iTunes store application. It should be appreciated that other requests to adjust the state or settings of an application on the media device may also be contemplated.

[0235] Referring again to block 724, it may be determined that the user intent does not include a request to adjust a state or setting of an application on the electronic device. For example, the user intent may instead be a request to present information related to one or more media items. In response to such a determination, one or more of blocks 728-746 may be executed. At block 728 of process 700, a determination may be made as to whether the user intent is one of a plurality of predetermined request types. In some examples, the plurality of predetermined request types may be requests associated with a text-only response. More specifically, the plurality of predetermined request types may be requests for information that are predetermined to require a text-only response. This is in contrast to requests that are predetermined to require a response that includes a media object (e.g., an image, an animated object, a video, etc.). In some examples, the plurality of predetermined request types may include a request for the current time in a particular location (e.g., "What time is it in Paris?"), a request for a joke (e.g., "Tell me a funny joke"), or a request for information about media content currently playing on the electronic device (e.g., "When was this movie released?"). In response to determining that the user intent is one of a plurality of predetermined request types, one or more of blocks 730-732 may be executed.

[0236] At block 730 of process 700, results that at least partially satisfy the user intent may be obtained. For example, the results may be obtained from an external service (e.g., external service 120) by executing a task flow. At block 732 of process 700, the results obtained at block 730 may be displayed in text format on a display unit. Additionally, the results may be displayed in text format without displaying any corresponding graphics or media-related items corresponding to the results.

[0237] 8M-8P illustrate an example implementation of blocks 728-732. As shown in FIG. 8M, the movie "Legally Blonde" may be initially playing on a media device and displayed on a display unit. While "Legally Blonde" is playing, a digital assistant may be invoked (e.g., in block 704), and a user utterance in the sampled audio data may be, "Who's the lead actress?" For example, as shown in FIG. 8N, a text representation 840 of the user utterance may be displayed on the display unit. Based on the user utterance, it may be determined (e.g., in block 710) that the user intent includes a request to identify the lead actress of a particular media item. Because the user utterance does not specify any particular media item, the user intent may be ambiguous. However, based on the movie "Legally Blonde" being displayed at the time the audio data was sampled, it may be determined that the media item associated with the user intent is "Legally Blonde." In this example, it may be determined (e.g., in block 728) that the user intent is one of a plurality of predetermined request types. Specifically, it may be determined that a text-only response can be provided to satisfy the user intent to identify the leading actress in Legally Blonde. In response to determining that the user intent is one of a plurality of predetermined request types, a search may be performed (e.g., at block 730) in a media association database to obtain "Reese Witherspoon" as the leading actress in the movie "Legally Blonde." As shown in FIG. 8P, text-only result 842 "Reese Witherspoon" may be displayed on the display unit to satisfy the user intent. Text-only result 842 may be overlaid on the displayed media content of "Legally Blonde." Furthermore, the media content of "Legally Blonde" may continue to play while text-only result 842 is displayed.By displaying text-only results 842 (e.g., without displaying graphical results or additional user interfaces to satisfy the user intent), the user intent can be satisfied in an unobtrusive manner and with minimal interruption to the user's consumption of media content, while providing the user with access to the services of a digital assistant, which may be desirable for an improved user experience.

[0238] Referring again to block 728, it may be determined that the user intent is not one of a plurality of predetermined request types. Specifically, the user intent may be a predetermined request type that requires more than just text results to be satisfied. For example, the user intent may be a request to perform a media search query and display media items corresponding to the media search query. In other examples, the user intent may be a request for information other than media items. For example, the user intent may be a request for information associated with a sports team (e.g., "How did the LA Lakers do in the last game?"), an athlete (e.g., "How tall is LeBron James?"), stock prices (e.g., "What was the closing price of the Dow Jones yesterday?"), or weather (e.g., "What is the weather forecast for Paris, France next week?"). In response to determining that the user intent is not one of a plurality of predetermined request types, one or more of blocks 734-746 may be executed.

[0239] At block 734 of process 700, second results that at least partially satisfy the user intent may be obtained. Block 734 may be similar to or identical to block 534 described above. In one embodiment, the user intent may include a request to perform a media search query. In this embodiment, the media search query may be performed at block 734 to obtain second results. Specifically, the second results may include media items that correspond to the media search query.

[0240] In some embodiments, the user intent may not be a media search query. For example, the user intent may be a request to provide a weather forecast for Paris, France (e.g., "What's the weather forecast for Paris, France?"). In this embodiment, the second results retrieved in block 734 may include a seven-day weather forecast for Paris, France. The second results may include non-media data that at least partially satisfy the user intent. Specifically, the seven-day weather forecast for Paris, France may include text data (e.g., date, temperature, and a brief description of the weather conditions) and graphical images (e.g., images of sunny, cloudy, windy, or rainy). Furthermore, in some embodiments, in block 710, the scope of the user intent may be expanded to include a request for media items that at least partially satisfy the user intent. In these embodiments, the second results retrieved in block 734 may further include one or more media items having media content that at least partially satisfy the user intent. For example, in block 734, a media search query for a weather forecast in Paris, France for a relevant time period may be performed, and one or more media items related to the weather forecast in Paris, France may be retrieved. The one or more media items may include, for example, a video clip from a weather channel presenting the weather forecast in Paris, France. In these examples, the non-media data and / or the one or more media items may be displayed within a user interface on the displayed unit (e.g., in blocks 738, 742, or 746, described below).

[0241] At block 736 of process 700, a determination may be made as to whether the displayed content includes media content currently playing on the electronic device. In some embodiments, it may be determined that the displayed content does not include media content currently playing on the electronic device. For example, the displayed content may instead include a user interface, such as a main menu user interface or a third user interface (e.g., third user interface 826). The third user interface may occupy at least a majority of the display area of ​​the display unit. Furthermore, the third user interface may include previous results associated with previous user requests received before detecting the user input at block 704. Following a determination that the displayed content does not include media content, block 738 may be executed.

[0242] At block 738 of process 700, a portion of the second result may be displayed within a third user interface on the display unit. In embodiments in which the displayed content already includes a third user interface at the time the user input at block 704 is received, a portion of the display of the second result in the third user interface may replace a display of a previous result associated with a previous user request. In embodiments in which the displayed content does not include a third user interface at the time the user input at block 704 is received (e.g., the displayed content does not include a main menu user interface), the third user interface may be displayed and the second result may be included within the displayed third user interface.

[0243] In some embodiments, a determination may be made as to whether the second results include a predetermined type of result. The predetermined type of result may be associated with a display area that is less than half of the display area of ​​the display unit. The predetermined type of result may include, for example, results related to stock prices or weather. It should be appreciated that in other embodiments, the predetermined type of result may differ. In response to determining that the second results include a predetermined type of result, a portion of the second results may be displayed within a second user interface on the display unit. The second user interface may occupy less than half of the display area of ​​the display unit. In these embodiments, even if it is determined in block 736 that the displayed content does not include media content, a portion of the second results may be displayed within the second user interface.

[0244] 8Q-8S show example implementations of blocks 734-738. In this example, as shown in FIG. 8Q, the displayed content may initially include a third user interface 826. The third user interface 826 may include previous results from a previous user request. Specifically, the third user interface 826 includes media items 844 from a previously requested media search query. As shown in FIG. 8R, while the third user interface 826 is displayed, a digital assistant may be invoked (e.g., in block 704). The user utterance in the sampled audio data may include, "What movies star Luke Wilson?" A text representation 846 of the user utterance may be displayed on the display unit. In this example, the user intent may be determined (e.g., in block 710) to be a request to perform a media search query for movies starring Luke Wilson. The media search query may be performed (e.g., in block 734), and second results may be obtained. Specifically, the second results may include media items 848 corresponding to movies starring Luke Wilson. Additionally, additional results (e.g., media items 850) related to the user intent or previous user intent may be obtained. These additional results may be obtained in a manner similar to the second result described in block 544.

[0245] 8Q-8S, the displayed content includes only third user interface 826, and therefore, it may be determined (e.g., at block 736) that the displayed content does not include media content currently playing on the electronic device. In response to this determination, a second result may be displayed within third user interface 826. Specifically, as shown in FIG. 8S, the display of media item 844 within third user interface 826 may be replaced with the display of media item 848 within third user interface 826. Additionally, media item 850 may be displayed within third user interface 826.

[0246] As shown in this embodiment, the second result can be presented in the third user interface only after determining that the media content is not being displayed on the display unit. This allows a wider range of results to be displayed in a larger area to increase the likelihood that the user's actual intention will be satisfied. At the same time, by ensuring that the media content is not being displayed on the display unit before presenting the second result in the third user interface, the user's consumption of the media content is not interrupted.

[0247] Referring again to block 736, the displayed content may include media content currently playing on the media device. In these examples, a determination may be made that the displayed content includes media content currently playing on the media device. In accordance with this determination, one or more of blocks 740-746 may be performed.

[0248] At block 740 of process 700, a determination may be made as to whether the playing media content can be paused. Examples of media content that can be paused include on-demand media items, such as on-demand movies and television programs. Examples of media content that cannot be paused include media programs from broadcast or streaming services, as well as live media programs (e.g., sporting events, concerts, etc.). Thus, on-demand media items may not include broadcast or live programs. In accordance with a determination at block 740 that the playing media content cannot be paused, block 742 may be executed. In block 742 of process 700, a second user interface having a portion of the second result may be displayed on the display unit. Block 742 may be similar to block 536 described above. The second user interface may be displayed while the media content is being displayed. The display area occupied by the second user interface on the display unit may be smaller than the display area occupied by the media content on the display unit. In accordance with a determination that the playing media content can be paused, one or more of blocks 744-746 may be executed. At block 744 of process 700, the media content being played may be paused on the media device. At block 746 of process 700, a third user interface having a portion of the second result may be displayed. The third user interface may be displayed while the media content is paused.

[0249] 8T-8W illustrate example implementations of blocks 740-746. As shown in FIG. 8T, media content 802 playing on a media device can be displayed on a display unit. While displaying media content 802, a digital assistant can be activated (e.g., at block 704). A user utterance in the sampled audio data can be, "Tell me some movies starring Luke Wilson." A text representation 846 of the user utterance can be displayed on the display unit. As described above, a user intent can be determined (e.g., at block 710) to be a request to retrieve media items for movies starring Luke Wilson. A corresponding media search query can be performed (e.g., at block 734), and second results can be obtained. The second results can include media items 848 for movies starring Luke Wilson. In implementations where it is determined (e.g., at block 744) that media content 802 cannot be paused, media items 848 can be displayed in second user interface 818 while media content 802 continues to be displayed on the display unit (e.g., FIG. 8U). Displaying the media item 848 in the second user interface 818 may be desirable to allow the media content 802 to continue to be available for user consumption while the media item 848 is displayed to satisfy a user intent. This prevents the user from missing any portion of the media content 802 that cannot be paused or replayed. Alternatively, in embodiments where it is determined (e.g., at block 744) that the media content 802 can be paused, playback of the media content 802 on the media device can be paused, and the media item 848 can be displayed in the third user interface 826 on the display unit (e.g., FIG. 8S).Displaying the third user interface 826 may be desirable to allow a wider range of media items associated with various alternative user intents (e.g., media item 850) to be displayed along with the requested media item (e.g., media item 848), thereby increasing the likelihood that the user's actual intent will be satisfied. At the same time, the media content 802 is paused so that the user does not miss any portion of the media content 802. By modifying the user interface used to display the media item 848 based on whether the media content 802 can be paused, the user intent associated with the user utterance can be comprehensively achieved while reducing interruptions to the user's consumption of the media content 802. This can enhance the overall user experience.

[0250] In some examples, as shown in FIG. 8V, the displayed content may include a second user interface 818 in addition to the media content 802 currently playing on the media device. In these examples, the second user interface 818 may include media items 852 related to a previous user request (e.g., a request for a romantic comedy starring Reese Witherspoon). While displaying the media content 802 and the second user interface 818, a digital assistant may be invoked (e.g., at block 704). As shown in FIG. 8W, the sampled audio data may include a user utterance, "What movies are there starring Luke Wilson?" A text representation 846 of the user utterance may be displayed on the display unit. Based on the user utterance, it may be determined (e.g., at block 710) that the user intent is a request to retrieve media items for movies starring Luke Wilson. A corresponding media search query may be performed (e.g., at block 734) to retrieve second results (e.g., media items 848). In these examples, the display of media item 852 may replace the display of media item 848 in second user interface 818 (eg, FIG. 8U).

[0251] 9 shows a process 900 for interacting with a digital assistant in a media system according to various embodiments. Process 900 can be performed using one or more electronic devices implementing a digital assistant. For example, process 900 can be performed using one or more of system 100, media system 128, media device 104, user device 122, or digital assistant system 400 described above. It should be understood that some operations in process 900 can be combined, the order of some operations can be changed, and some operations can be omitted.

[0252] At block 902 of process 900, content may be displayed on a display unit. Block 902 may be similar to or identical to block 502 described above. In some embodiments, the displayed content may include media content (e.g., movies, videos, television programs, video games, etc.). Additionally, or alternatively, the displayed content may include a user interface. For example, the displayed content may include a first user interface having one or more exemplary natural language requests (e.g., as shown in FIGS. 6D-6E). In other embodiments, the displayed content may include a third user interface (e.g., third user interface 626) having results from a previous user request (e.g., previously requested media items). The third user interface may occupy at least a majority of the display area of ​​the display unit.

[0253] At block 904 of process 900, a user input may be detected while displaying the content of block 902. The user input may be similar to or identical to the fifth user input described at block 558. Specifically, the user input may be detected on a remote control device for the media device. For example, the user input may include a predetermined movement pattern on a touch-sensitive surface of the remote control device. In some embodiments, the user input may be detected via a second electronic device (e.g., device 122) that is different from the media device. The second electronic device may be configured to wirelessly control the media device. In response to detecting the user input, one or more of blocks 906-914 may be executed.

[0254] At block 906 of process 900, a virtual keyboard interface (e.g., virtual keyboard interface 646) may be displayed on the display unit. Block 906 may be similar to or identical to block 562 described above. The virtual keyboard interface may be overlaid on at least a portion of the first user interface or the third user interface. Additionally, a search field (e.g., search field 644) may be displayed on the display unit. The virtual keyboard interface may be configured such that user input received via the virtual keyboard interface results in text entry into the search field.

[0255] At block 908 of process 900, a selectable affordance may be displayed on the second electronic device (e.g., on the touchscreen 346 of device 122). The second electronic device may be a different device than the media device's remote control. Selection of the affordance may allow text input to be received by the media device via a keyboard of the second electronic device. For example, selection of the affordance may cause a virtual keyboard interface (e.g., similar to virtual keyboard interface 646) to be displayed on the second electronic device. Input into the virtual keyboard interface of the second electronic device may cause corresponding text to be entered into a search field (e.g., search field 644).

[0256] At block 910 of process 900, text input may be received via a keyboard (e.g., a virtual keyboard interface) of the second electronic device. Specifically, a user may enter text via the keyboard of the second electronic device, and the text input may be transmitted to and received by the media device. The text input may represent a user request. For example, the text input may be "Jurassic Park," which may represent a request to perform a search for media items associated with the search string "Jurassic Park."

[0257] At block 912 of process 900, results that at least partially satisfy the user request may be obtained. For example, a media search may be performed using the text input, and corresponding media items may be retrieved. In a particular embodiment, where the text input is "Jurassic Park," media items having the title "Jurassic Park" or having a common actor or director with the movie "Jurassic Park" may be retrieved. In another embodiment, where the text input is "Reese Witherspoon," media items in which Reese Witherspoon is an actress may be retrieved.

[0258] At block 914 of process 900, a user interface may be displayed on the display unit. The user interface may include at least a portion of the results. For example, the user interface may include media items obtained as a result of the media search performed at block 912.

[0259] It should be appreciated that, although particular blocks of processes 500, 700, and 900 are described above as being performed by a device or system (e.g., media device 104, user device 122, or digital assistant system 400), in some embodiments, more than one device can be used to perform the block. For example, in a block in which a determination is made, a first device (e.g., media device 104) can obtain the determination from a second device (e.g., server system 108). Similarly, in a block in which content, an object, text, or a user interface is displayed, a first device (e.g., media device 104) can cause the content, object, text, or user interface to be displayed on a second device (e.g., display unit 126). 5. Electronic Devices

[0260] According to some embodiments, FIG. 10 shows a functional block diagram of an electronic device 1000 configured in accordance with the principles of various described embodiments, for example, to provide voice control of media playback and real-time updates of virtual assistant knowledge. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to implement the principles of various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 10 may be combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or division, or further definition, of the functional blocks described herein.

[0261] 10 , electronic device 1000 may include an input unit 1003 (e.g., remote control device 124, or the like) configured to receive user input, such as tactile input, gesture input, etc., an audio input unit 1004 (e.g., microphone 272, or the like) configured to receive audio data, a speaker unit 106 (e.g., speaker 268, or the like) configured to output audio, and a communication unit 1007 (e.g., communication subsystem 224, or the like) configured to send and receive information from external devices over a network. In some examples, electronic device 1000 may optionally include a display unit 1002 (e.g., display unit 126, or the like) configured to display media, interfaces, and other content. The electronic device 1000 may further include an input unit 1003, an audio input unit 1004, a speaker unit 1006, a communication unit 1007, and optionally a processing unit 1008 coupled to the display unit 1002. In some embodiments, the processing unit 1008 may include a display enabling unit 1010, a detection unit 1012, a determination unit 1014, a sampling unit 1016, an output unit 1018, an execution unit 1020, an acquisition unit 1022, and a switching unit 1024.

[0262] According to some embodiments, the processing unit 1008 is configured to display content (e.g., using the display enabling unit 1010) on a display unit (e.g., the display unit 1002 or a separate display unit). The processing unit 1008 is further configured to detect a user input (e.g., using the detection unit 1012). The processing unit 1008 is further configured to determine whether the user input corresponds to a first input format (e.g., using the determination unit 1014). The processing unit 1008 is further configured to display a plurality of exemplary natural language requests on the display unit (e.g., using the display enabling unit 1010) in accordance with a determination that the user input corresponds to the first input format. The plurality of exemplary natural language requests are contextually related to the displayed content, and receiving a user utterance corresponding to one of the plurality of exemplary natural language requests causes the digital assistant to perform a respective action.

[0263] In some examples, the user input is detected on a remote control of the electronic device. In some examples, the first form of input includes pressing a button on the remote control and releasing the button within a predetermined period of time. In some examples, the plurality of exemplary natural language requests is displayed on a display unit via a first user interface, the first user interface being overlaid on the displayed content. In some examples, the displayed content includes media content, and the media content continues to play while displaying the plurality of exemplary natural language requests.

[0264] In some embodiments, the processing unit 1008, in accordance with a determination that the user input corresponds to the first input format, is further configured to display on the display unit (e.g., using the display enablement unit 1010) a visual indicator indicating that the digital assistant is not processing the voice input.

[0265] In some embodiments, upon determining that the user input corresponds to the first input format, a plurality of example natural language requests are displayed on the display unit after a predetermined time, hi some embodiments, each of the plurality of example natural language requests is displayed separately at a different time in a predetermined order.

[0266] In some embodiments, the processing unit 1008 is further configured to display (e.g., using the display enabling unit 1010) multiple lists of example natural language requests, each list being displayed in turn at a different time.

[0267] In some embodiments, processing unit 1008 is further configured to determine (e.g., with the determining unit 1014) whether the user input corresponds to a second input format in accordance with a determination that the user input does not correspond to the first input format. Processing unit 1008 is further configured to sample the audio data (e.g., with the sampling unit 1016 and the audio input unit 1004) in accordance with a determination that the user input corresponds to the second input format. Processing unit 1008 is further configured to determine (e.g., with the determining unit 1014) whether the audio data comprises a user request. Processing unit 1008 is further configured to perform (e.g., with the executing unit 1020) a task that at least partially satisfies the user request in accordance with a determination that the audio data comprises a user request.

[0268] In some embodiments, the second form of input includes pressing a button on a remote control of the electronic device and holding the button down for more than a predetermined period of time.

[0269] In some embodiments, the processing unit 1008 is further configured to display, on the display unit (e.g., using the display enabling unit 1010), a request for clarification of the user's intent, in accordance with a determination that the voice data does not contain a user request.

[0270] In some embodiments, the displayed content includes media content, and the media content continues to play on the electronic device while sampling the audio data and while performing the task.

[0271] In some embodiments, processing unit 1008 is further configured to output (e.g., with output unit 1018) sound associated with the media content (e.g., with speaker unit 1006). Processing unit 1008 is further configured to reduce the amplitude of the sound (e.g., with output unit 1018) in accordance with determining that the user input corresponds to the second input format.

[0272] In some examples, the task is performed without outputting speech associated with the task from the electronic device. In some examples, the audio data is sampled while detecting the user input. In some examples, the audio data is sampled for a predetermined period of time after detecting the user input.

[0273] In some examples, the audio data is sampled via a first microphone (e.g., audio input unit 1004) on a remote control of the electronic device. The processing unit 1008 is further configured to sample background audio data (e.g., using the sampling unit 1016 and the audio input unit 1004) via a second microphone (e.g., a second audio input unit of the electronic device 1000) on the remote control while sampling the audio data. The processing unit 1008 is further configured to remove background noise in the audio data using the background audio data (e.g., using the output unit 1018).

[0274] In some embodiments, audio associated with the displayed content is output from the electronic device via an audio signal, and the processing unit 1008 is further configured to use the audio signal to remove background noise in the audio data (e.g., using the output unit 1018).

[0275] In some embodiments, the processing unit 1008 is further configured to, in response to detecting the user input, display a visual cue on the display unit (e.g., using the display enabling unit 1010) prompting the user to provide a speech request.

[0276] In some embodiments, the processing unit 1008 is further configured to obtain (e.g., with the obtaining unit 1022) results that at least partially satisfy the user request. The processing unit 1008 is further configured to display (e.g., with the display enabling unit 1010) a second user interface on the display unit. The second user interface includes a portion of the results, and at least a portion of the content remains displayed while the second user interface is displayed, and a display area of ​​the second user interface on the display unit is smaller than a display area of ​​the at least a portion of the content on the display unit. In some embodiments, the second user interface is overlaid on the displayed content.

[0277] In some embodiments, the result portion includes one or more media items. The processing unit 1008 is further configured to receive (e.g., with the detection unit 1012) a selection of a media item from the one or more media items via the second user interface. The processing unit 1008 is further configured to display (e.g., with the display enabling unit 1010) media content associated with the selected media item on the display unit.

[0278] In some examples, the processing unit 1008 is further configured to detect a second user input (e.g., with the detection unit 1012) while displaying the second user interface. The processing unit 1008 is further configured to cease displaying the second user interface (e.g., with the display enabling unit 1010) in response to detecting the second user input.

[0279] In some embodiments, the second user input is detected on a remote control of the electronic device, the second user input comprising a first predetermined movement pattern on the touch-sensitive surface of the remote control.

[0280] In some embodiments, the processing unit 1008 is further configured to detect a third user input (e.g., with the detection unit 1012) while displaying the second user interface. In response to detecting the third user input, the processing unit 1008 is further configured to replace the display of the second user interface with a display of the third user interface on the display unit (e.g., with the display enabling unit 1010). The third user interface includes at least a portion of the results, and the third user interface occupies at least a majority of the display area of ​​the display unit.

[0281] In some embodiments, the third user input is detected on a remote control of the electronic device, the third user input comprising a second predetermined movement pattern on the touch-sensitive surface of the remote control.

[0282] In some embodiments, the processing unit 1008 is further configured to, in response to detecting the third user input, obtain (e.g., with the obtaining unit 1022) a second result that is different from the result, the second result at least partially satisfying the user request, and the third user interface including at least a portion of the second result.

[0283] In some embodiments, the second result is based on a user request received before detecting the user input, and in some embodiments, focus of the second user interface is on an item in the results portion while the third user input is detected, and the second result is contextually related to the item.

[0284] In some embodiments, the displayed content includes media content. The processing unit 1008 is further configured to pause (e.g., with the execution unit 1020) playback of the media content on the electronic device in response to detecting the third user input.

[0285] In some embodiments, at least a portion of the results include one or more media items. The processing unit 1008 is further configured to receive (e.g., with the detection unit 1012) a selection of a media item from the one or more media items via the third user interface. The processing unit 1008 is further configured to display (e.g., with the display enabling unit 1010) media content associated with the media item on the display unit.

[0286] In some embodiments, the processing unit 1008 is further configured to detect (e.g., with the detection unit 1012) a fourth user input associated with a direction on the display unit while displaying the third user interface. In response to detecting the fourth user input, the processing unit 1008 is further configured to switch the focus of the third user interface from the first item to a second item on the third user interface (e.g., with the switching unit 1024), the second item being positioned in the direction relative to the first item.

[0287] In some examples, the processing unit 1008 is further configured to detect a fifth user input (e.g., with the detection unit 1012) while displaying the third user interface. The processing unit 1008 is further configured to display a search field (e.g., with the display enabling unit 1010) in response to detecting the fifth user input. The processing unit 1008 is further configured to display a virtual keyboard interface on the display unit (e.g., with the display enabling unit 1010), wherein input received via the virtual keyboard interface causes text entry into the search field.

[0288] In some embodiments, the processing unit 1008 is further configured to detect a sixth user input (e.g., with the detection unit 1012) while displaying the third user interface. The processing unit 1008 is further configured to sample second audio data (e.g., with the sampling unit 1016 and the audio input unit 1004) in response to detecting the sixth user input. The second audio data comprises a second user request. The processing unit 1008 is further configured to determine (e.g., with the determination unit 1014) whether the second user request is a request to narrow the results of the user request. The processing unit 1008 is further configured to display a subset of the results via the third user interface (e.g., with the display enabling unit 1010) in accordance with a determination that the second user request is a request to narrow the results of the user request.

[0289] In some embodiments, the subset of the results is displayed in a top row of the third user interface. The processing unit 1008 is further configured to, in accordance with determining that the second user request is not a request to narrow the user request results, retrieve (e.g., with the retrieving unit 1018) third results that at least partially satisfy the second user request. The processing unit 1008 is further configured to display (e.g., with the display enabling unit 101) a portion of the third results via the third user interface. In some embodiments, the portion of the third results is displayed in a top row of the third user interface.

[0290] In some embodiments, the processing unit 1008 is further configured to obtain (e.g., with the obtaining unit 1022) a fourth result that at least partially satisfies the user request or the second user request. The processing unit 1008 is further configured to display (e.g., with the display enabling unit 1010) a portion of the fourth result via the third user interface.

[0291] In some embodiments, the fourth results portion is displayed in a row after the top row of the third user interface.

[0292] In some embodiments, the focus of the third user interface is on one or more items in the third user interface while the sixth user input is detected, and the fourth result is contextually related to the one or more items.

[0293] In some examples, the processing unit 1008 is further configured to detect a seventh user input (e.g., with the detection unit 1012) while displaying the third user interface. In response to detecting the seventh user input, the processing unit 1008 is further configured to cease displaying the third user interface (e.g., with the display enabling unit 1010).

[0294] In some embodiments, the displayed content is media content, and in response to detecting the third user input, playback of the media content on the electronic device is paused. The processing unit 1008 is further configured to resume (e.g., with the execution unit 1020) playback of the media content on the electronic device in response to detecting a seventh user input. In some embodiments, the seventh user input includes pressing a menu button on a remote control of the electronic device.

[0295] According to some embodiments, the processing unit 1008 is further configured to display the content on the display unit (e.g., using the display enabling unit 1010). The processing unit 1008 is further configured to detect a user input (e.g., using the detection unit 1012) while displaying the content. The processing unit 1008 is further configured to display a user interface on the display unit (e.g., using the display enabling unit 1010) in response to detecting the user input. The user interface includes a plurality of exemplary natural language requests contextually related to the displayed content, and receiving a user utterance corresponding to one of the plurality of exemplary natural language requests causes the digital assistant to perform a respective action.

[0296] In some embodiments, the displayed content includes media content. In some embodiments, the plurality of exemplary natural language requests includes natural language requests to change one or more settings associated with the media content. In some embodiments, the media content continues to play while the user interface is displayed.

[0297] In some embodiments, the processing unit 1008 is further configured to output (e.g., using the output unit 1018) audio associated with the media content. The amplitude of the audio is not reduced in response to detecting a user input. In some embodiments, the displayed content includes a main menu user interface.

[0298] In some examples, the plurality of exemplary natural language requests includes exemplary natural language requests associated with each of a plurality of core capabilities of the digital assistant. In some examples, the displayed content includes a second user interface with results associated with a previous user request. In some examples, the plurality of exemplary natural language requests includes a natural language request to refine results. In some examples, the user interface includes textual instructions for invoking and interacting with the digital assistant. In some examples, the user interface includes a visual indicator indicating that the digital assistant is not receiving voice input. In some examples, the user interface is overlaid on the displayed content.

[0299] In some embodiments, the processing unit 1008 is further configured to, in response to detecting the user input, reduce the brightness of the displayed content (e.g., using the display enabling unit 1010) to enhance the user interface.

[0300] In some examples, the user input is detected on a remote control device for the electronic device. In some examples, the user input includes pressing a button on the remote control device and releasing the button within a predetermined period after pressing the button. In some examples, the button is configured to invoke a digital assistant. In some examples, the user interface includes textual instructions for displaying a virtual keyboard interface.

[0301] In some examples, the processing unit 1008 is further configured to detect a second user input (e.g., with the detection unit 1012) after displaying the user interface. The processing unit 1008 is further configured to display a virtual keyboard interface on the display unit (e.g., with the display unit 1012) in response to detecting the second user input.

[0302] In some examples, the processing unit 1008 is further configured to change focus of the user interface to a search field on the user interface (e.g., with the display enabling unit 1010). In some examples, the search field is configured to receive a text search query via a virtual keyboard interface. In some examples, the virtual keyboard interface cannot be used to interact with the digital assistant. In some examples, the second user input includes a predetermined movement pattern on a touch-sensitive surface of a remote control device of the electronic device.

[0303] In some examples, the plurality of example natural language requests are displayed at a predetermined time after detecting the user input. In some examples, the processing unit 1008 is further configured to display each of the plurality of example natural language requests one by one in a predetermined order (e.g., using the display enabling unit 1010). In some examples, the processing unit 1008 is further configured to replace the display of a previously displayed example natural language request of the plurality of example natural language requests with a subsequent example natural language request of the plurality of example natural language requests (e.g., using the display enabling unit 1010).

[0304] In some examples, the content includes a second user interface having one or more items, where focus of the second user interface is on an item of the one or more items when user input is detected, and where the plurality of exemplary natural language requests are contextually related to the item of the one or more items.

[0305] According to some embodiments, the processing unit 1008 is further configured to display the content on the display unit (e.g., with the display enabling unit 1010). The processing unit 1008 is further configured to detect a user input (e.g., with the detection unit 1012). The processing unit 1008 is further configured to display one or more example suggestions of natural language utterances (e.g., with the display enabling unit 1010) in response to detecting the user input. The one or more example suggestions are contextually related to the displayed content and, when uttered by the user, cause the digital assistant to perform a corresponding action.

[0306] In some embodiments, processing unit 1008 is further configured to detect a second user input (e.g., with detection unit 1012). Processing unit 1008 is further configured to sample audio data (e.g., with sampling unit 1016) in response to detecting the second user input. Processing unit 1008 is further configured to determine (e.g., with determination unit 1014) whether the sampled audio data includes one of the one or more suggested example natural language utterances. Processing unit 1008 is further configured to perform a corresponding action on the utterance (e.g., with execution unit 1020) in accordance with a determination that the sampled audio data includes one of the one or more suggested example natural language utterances.

[0307] According to some embodiments, the processing unit 1008 is further configured to display the content on the display unit (e.g., using the display enabling unit 1010). The processing unit 1008 is further configured to detect user input (e.g., using the detection unit 1012) while displaying the content. The processing unit 1008 is further configured to sample audio data (e.g., using the sampling unit 1016) in response to detecting the user input. The audio data includes user utterances expressing a media search request. The processing unit 1008 is further configured to retrieve a plurality of media items satisfying the media search request (e.g., using the retrieval unit 1022). The processing unit 1008 is further configured to display at least a portion of the plurality of media items via a user interface on the display unit (e.g., using the display enabling unit 1010).

[0308] In some embodiments, the content continues to be displayed on the display unit while at least a portion of the plurality of media items is displayed, and the display area occupied by the user interface is smaller than the display area occupied by the content.

[0309] In some embodiments, the processing unit 1008 is further configured to determine (e.g., with the determining unit 1014) whether the number of media items in the plurality of media items is less than or equal to a predetermined number. Following a determination that the number of media items in the plurality of media items is less than or equal to the predetermined number, at least a portion of the plurality of media items includes the plurality of media items.

[0310] In some embodiments, following a determination that the number of media items in the plurality of media items is greater than the predetermined number, the number of media items in at least a portion of the plurality of media items is equal to the predetermined number.

[0311] In some embodiments, each of the plurality of media items is associated with a relevance score with respect to the media search request, and the relevance score of at least a portion of the plurality of media items is highest among the plurality of media items.

[0312] In some embodiments, each of at least a portion of the plurality of media items is associated with a popularity rating, and at least a portion of the plurality of media items are arranged within the user interface based on the popularity rating.

[0313] In some embodiments, processing unit 1008 is further configured to detect a second user input (e.g., with detection unit 1012) while displaying at least a portion of the plurality of media items. In response to detecting the second user input, processing unit 1008 is further configured to expand the user interface (e.g., with display enabling unit 1010) to occupy at least half of the display area of ​​the display unit.

[0314] In some embodiments, processing unit 1008 is further configured to determine (e.g., with determining unit 1014) whether the number of media items in the plurality of media items is less than or equal to a predetermined number in response to detecting the second user input. Processing unit 1008 is further configured to obtain a second plurality of media items that at least partially satisfy the media search request in accordance with determining that the number of media items in the plurality of media items is less than or equal to the predetermined number, wherein the second plurality of media items is different from at least a portion of the media items. Processing unit 1008 is further configured to display the second plurality of media items on the display unit (e.g., with display enabling unit 101) via an enlarged user interface.

[0315] In some embodiments, the processing unit 1008 is further configured to determine (e.g., with the determining unit 1014) whether the media search request includes more than one search parameter. In accordance with a determination that the media search request includes more than one search parameter, the second plurality of media items is organized in the enlarged user interface according to the one or more search parameters of the media search request.

[0316] In some embodiments, the processing unit 1008 is further configured to display (e.g., with the display enabling unit 1010) at least a second portion of the plurality of media items via an enlarged user interface in accordance with a determination that the number of media items in the plurality of media items is greater than a predetermined number. The at least second portion of the plurality of media items is different from at least a portion of the plurality of media items.

[0317] In some embodiments, at least a second portion of the plurality of media items includes two or more media types, and the at least a second portion of the plurality of media items is organized within the expanded user interface according to each media type of the two or more media types.

[0318] In some embodiments, processing unit 1008 is further configured to detect a third user input (e.g., with detection unit 1012). In response to detecting the third user input, processing unit 1008 is further configured to cause the magnified user interface to scroll (e.g., with display enabling unit 1010). Processing unit 1008 is further configured to determine (e.g., with determination unit 1014) whether the magnified user interface has scrolled past a predetermined position on the magnified user interface. In response to determining that the magnified user interface has scrolled past a predetermined position on the magnified user interface, processing unit 1008 is further configured to display at least a third portion of the plurality of media items on the magnified user interface (e.g., with display enabling unit 1010). The at least third portion of the plurality of media items is organized on the magnified user interface according to one or more media content providers associated with the third plurality of media items.

[0319] The operations described above with reference to Figures 5A-5I are optionally performed by components shown in Figures 1-3 and 4A-4B. For example, the display operations 502, 508-514, 520, 524, 530, 536, 546, 556, 560, 562, 576, 582, 588, 592, the detection operations 504, 538, 542, 550, 558, 566, 570, the determination operations 506, 516, 522, 526, 528, 574, 578, the sampling operations 518, 572, the execution operations 532, 584, the acquisition operations 534, 544, 580, 586, 590, the abort operations 540, 568, the receiving unit 554, and the switching operations 552, 564 may be performed by one or more of the operating system 252, the GUI module 256, the application module 262, the digital assistant module 426, and the processor(s) 204, 404. It will be clear to one skilled in the art how other processes may be implemented based on the components shown in Figures 1-3 and 4A-4B.

[0320] According to some embodiments, FIG. 11 shows a functional block diagram of an electronic device 1100 configured in accordance with the principles of various described embodiments, for example, to provide voice control of media playback and real-time updates of virtual assistant knowledge. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to implement the principles of various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 11 may be combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or division, or further definition, of the functional blocks described herein.

[0321] 11 , electronic device 1100 may include an input unit 1103 (e.g., remote control device 124, or the like) configured to receive user input, such as tactile input, gesture input, etc., an audio input unit 1104 (e.g., microphone 272, or the like) configured to receive audio data, a speaker unit 116 (e.g., speaker 268, or the like) configured to output audio, and a communication unit 1107 (e.g., communication subsystem 224, or the like) configured to send and receive information from external devices over a network. In some examples, electronic device 1100 may optionally include a display unit 1102 (e.g., display unit 126, or the like) configured to display media, interfaces, and other content. The electronic device 1100 may further include an input unit 1103, an audio input unit 1104, a speaker unit 1106, a communication unit 1107, and optionally a processing unit 1108 coupled to the display unit 1102. In some embodiments, the processing unit 1108 may include a display enabling unit 1110, a detection unit 1112, a determination unit 1114, a sampling unit 1116, an output unit 1118, an execution unit 1120, an acquisition unit 1122, an identification unit 1124, and a transmission unit 1126.

[0322] According to some embodiments, the processing unit 1108 is configured to display content (e.g., with the display enabling unit 1110) on a display unit (e.g., the display unit 1102 or a separate display unit). The processing unit 1108 is further configured to detect user input (e.g., with the detection unit 1112) while displaying the content. In response to detecting the user input, the processing unit 1108 is further configured to sample audio data (e.g., with the sampling unit 1016 and the audio input unit 1104). The audio data includes user utterances. The processing unit 1108 is further configured to obtain (e.g., with the obtaining unit 1122) a determination of a user intention corresponding to the user utterances. The processing unit 1108 is further configured to obtain (e.g., with the obtaining unit 1122) a determination of whether the user intention includes a request to adjust a state or setting of an application on the electronic device. The processing unit 1108 is further configured, in response to obtaining a determination that the user intent includes a request to adjust a state or setting of an application on the electronic device, to adjust (e.g., using the task execution unit 1120) a state or setting of the application to satisfy the user intent.

[0323] In some embodiments, the request to adjust a state or setting of an application on the electronic device includes a request to play a particular media item. Adjusting the state or setting of the application to satisfy the user intent includes playing the particular media item.

[0324] In some embodiments, the displayed content includes a user interface having media items, and user utterances do not explicitly limit the particular media item to be played. Processing unit 1108 is further configured to determine whether the focus of the user interface is on the media item (e.g., with determining unit 1114). Processing unit 1108 is further configured to identify the media item as the particular media item to be played (e.g., with identifying unit 1124) in accordance with determining that the focus of the user interface is on the media item.

[0325] In some embodiments, the request to adjust a state or setting of an application on the electronic device includes a request to launch the application on the electronic device. In some embodiments, the displayed content includes media content being played on the electronic device, and the state or setting is related to the media content being played on the electronic device. In some embodiments, the request to adjust a state or setting of an application on the electronic device includes a request to fast forward or rewind media content being played on the electronic device. In some embodiments, the request to adjust a state or setting of an application on the electronic device includes a request to skip forward or backward within the media content to play a particular portion of the media content. In some embodiments, the request to adjust a state or setting of an application on the electronic device includes a request to pause playback of media content on the electronic device. In some embodiments, the request to adjust a state or setting of an application on the electronic device includes a request to turn on or off subtitles for the media content.

[0326] In some embodiments, the displayed content includes a user interface having a first media item and a second media item.

[0327] In some embodiments, the request to adjust a state or setting of an application on the electronic device includes a request to switch focus of a user interface from a first media item to a second media item. Adjusting the state or setting of the application to satisfy the user intent includes switching focus of the user interface from the first media item to the second media item.

[0328] In some embodiments, the displayed content includes media content being played on the media device. The user utterance is a natural language expression indicating that the user missed a portion of audio associated with the media content. The request to adjust a state or setting of an application on the electronic device includes a request to replay a portion of the media content that corresponds to the portion of audio that the user missed. The processing unit 1108 is further configured to rewind the media content by a predetermined amount (e.g., using the task execution unit 1120) to an earlier portion of the media content and restart playing the media content from the earlier portion (e.g., using the task execution unit 1120).

[0329] In some embodiments, the processing unit 1108 is further configured to turn on closed captioning (eg, using the task execution unit 1120) before restarting playback of the media content from an earlier portion.

[0330] In some embodiments, the request to adjust a state or setting of an application on the electronic device further includes a request to increase the volume of audio associated with the media content, and adjusting the state or setting of the application further includes increasing the volume of audio associated with the media content before restarting playback of the media content from an earlier portion.

[0331] In some embodiments, speech in audio associated with the media content is converted to text, and adjusting a state or setting of the application further includes displaying a portion of the text while restarting playback of the media content from a previous portion.

[0332] In some embodiments, the processing unit 1108 is further configured to obtain (e.g., with the obtaining unit 1122) a determination of a user emotion associated with the user utterance. The user intent is determined based on the determined user emotion.

[0333] In some examples, the processing unit 1108 is further configured, in response to obtaining a determination that the user intention does not include a request to adjust a state or setting of an application on the electronic device, to obtain (e.g., with the obtaining unit 1122) a determination of whether the user intention is one of a plurality of predefined request types. In response to obtaining a determination that the user intention is one of the plurality of predefined request types, the processing unit 1108 is further configured to obtain (e.g., with the obtaining unit 1122) a result that at least partially satisfies the user intention and display the result in text format on the display unit (e.g., with the display enabling unit 1110).

[0334] In some embodiments, the plurality of predetermined request types includes a request for the current time at a particular location. In some embodiments, the plurality of predetermined request types includes a request for a joke. In some embodiments, the plurality of predetermined request types includes a request for information about media content being played on the electronic device. In some embodiments, the textual results are overlaid on the displayed content. In some embodiments, the displayed content includes media content being played on the electronic device, and the media content continues to play while the textual results are displayed.

[0335] In some embodiments, the processing unit 1108 is further configured, in response to obtaining a determination that the user intention is not one of a plurality of predetermined request types, to obtain (e.g., with the obtaining unit 1122) a result that at least partially satisfies the second user intention and determine (e.g., with the determining unit 1114) whether the displayed content includes media content being played on the electronic device. The processing unit 1108 is further configured to determine (e.g., with the determining unit 1114) whether the media content can be paused in accordance with the determination that the displayed content includes media content. The processing unit 1108 is further configured to display (e.g., with the display enabling unit 1110) on the display unit a second user interface having a portion of the second result in accordance with the determination that the media content cannot be paused. A display area occupied by the second user interface on the display unit is smaller than a display area occupied by the media content on the display unit.

[0336] In some embodiments, the user intent includes a request for a weather forecast for a particular location. The user intent includes a request for information associated with a sports team or an athlete. In some embodiments, the user intent is not a media search query, and the second result includes one or more media items having media content that at least partially satisfies the user intent. In some embodiments, the second result further includes non-media data that at least partially satisfies the user intent. In some embodiments, the user intent is a media search query, and the second result includes a plurality of media items corresponding to the media search query.

[0337] In some embodiments, the processing unit 1108 is further configured to display (e.g., using the display enabling unit 1110) a third user interface on the display unit having a portion of the second result in accordance with a determination that the displayed content does not include media content currently playing on the electronic device, wherein the third user interface occupies a majority of the display area of ​​the display unit.

[0338] In some embodiments, the displayed content includes a main menu user interface.

[0339] In some embodiments, the displayed content includes a third user interface having previous results associated with previous user requests received prior to detecting the user input, and pursuant to a determination that the displayed content does not include media content currently playing on the electronic device, the display of the previous results in the third user interface is replaced with the display of the second results.

[0340] In some examples, processing unit 1108 is further configured to determine (e.g., with determining unit 1114) whether the displayed content includes a second user interface with previous results from a previous user request in accordance with a determination that the displayed content includes media content currently playing on the electronic device. In accordance with a determination that the displayed content includes a second user interface with previous results from a previous user request, the previous results are replaced with the second results.

[0341] In some embodiments, the processing unit 1108 is further configured, in accordance with determining that the media content can be paused, to pause playback of the media content on the electronic device (e.g., using the task execution unit 1120) and display on the display unit (e.g., using the display enabling unit 1110) a third user interface having a portion of the second result, wherein the third user interface occupies a majority of the display area of ​​the display unit.

[0342] In some embodiments, the processing unit 1108 is further configured to transmit the audio data to a server (e.g., with the transmission unit 1126 and using the communication unit 1107) for performing natural language processing, and to indicate to the server (e.g., with the transmission unit 1126) that the audio data is associated with a media application. The indication biases the natural language processing toward media-related user intent.

[0343] In some embodiments, the processing unit 1108 is further configured to transmit the audio data to a server (eg, the transmitting unit 1126) for speech-to-text processing.

[0344] In some embodiments, the processing unit 1108 is further configured to indicate to the server (e.g., using the transmission unit 1126) that the audio data is associated with a media application, the indication biasing the speech-to-text process toward media-related text results.

[0345] In some embodiments, the processing unit 1108 is further configured to obtain (e.g., using the obtaining unit 1122) a text representation of the user utterance, the text representation being based on a previous user utterance received before sampling the audio data.

[0346] In some embodiments, the text representation is based on the time the previous user utterance was received before sampling the audio data.

[0347] In some examples, the processing unit 1108 is further configured to obtain (e.g., with the obtaining unit 1122) a determination that the user intent does not correspond to one of a plurality of core capabilities associated with the electronic device. The processing unit 1108 is further configured to cause the second electronic device to perform (e.g., with the task executing unit 1120) a task to facilitate satisfying the user intent.

[0348] In some examples, the processing unit 1108 is further configured to obtain (e.g., with the obtaining unit 1122) a determination of whether the user utterance includes the ambiguous term. In response to obtaining a determination that the user utterance includes the ambiguous term, the processing unit 1108 is further configured to obtain (e.g., with the obtaining unit 1122) two or more candidate user intentions based on the ambiguous term and display the two or more candidate user intentions on the display unit (e.g., with the display enabling unit 1110).

[0349] In some examples, the processing unit 1108, while displaying the two or more candidate user intentions, is further configured to receive (e.g., with the detection unit 1112) a user selection of one of the two or more candidate user intentions. The user intention is determined based on the user selection.

[0350] In some embodiments, the processing unit 1108 is further configured to detect (e.g., with the detection unit) a second user input. In response to detecting the second user input, the processing unit 1108 is further configured to sample (e.g., with the sampling unit 1116) second audio data. The second audio data includes a second user utterance expressing a user selection.

[0351] In some examples, the two or more interpretations are displayed without outputting utterances associated with the two or more candidate user intents.

[0352] According to some embodiments, the processing unit 1108 is further configured to display content (e.g., with the display enabling unit 1110) on a display unit (e.g., the display unit 1102 or a separate display unit). The processing unit 1108 is further configured to detect user input (e.g., with the detection unit 1112) while displaying the content. The processing unit 1108 is further configured to display a virtual keyboard interface on the display unit (e.g., with the display enabling unit 1110) in response to detecting the user input. The processing unit 1108 is further configured to cause a selectable affordance to appear on a display of the second electronic device (e.g., with the task execution unit 1120). Selection of the affordance enables text input to be received by the electronic device (e.g., using the communication unit 1107) via a keyboard of the second electronic device.

[0353] In some embodiments, the processing unit 1108 is further configured to receive a text input via a keyboard of the second electronic device (e.g., using the detection unit 1112), the text input representing a user request. The processing unit 1108 is further configured to obtain a result (e.g., using the obtaining unit 1122) that at least partially satisfies the user request, and to display a user interface on the display unit (e.g., using the display enabling unit 1110), the user interface including at least a portion of the result.

[0354] In some examples, the displayed content includes a second user interface having a plurality of exemplary natural language requests. In some examples, the displayed content includes media content. In some examples, the displayed content includes a third user interface having results from previous user requests, the third user interface occupying at least a majority of a display area of ​​the display unit. In some examples, a virtual keyboard interface is overlaid on at least a portion of the third user interface. In some examples, the user input is detected via a remote control device of the electronic device, the remote control device and the second electronic device being different devices. In some examples, the user input includes a predetermined movement pattern on a touch-sensitive surface of the remote control device. In some examples, the user input is detected via the second electronic device.

[0355] The operations described above with reference to Figures 7A-7C and 9 are optionally performed by the components shown in Figures 1-3 and 4A. The operations described above with reference to Figures 7A-7C and 9 are optionally performed by the components shown in Figures 1-3 and 4A-4B. For example, the display operations 702, 716, 732, 736, 738, 742, 746, 902, 906, 914, the detection operations 704, 718, 904, 910, the determination operations 708, 710, 712, 714, 720, 724, 728, 736, 740, the sampling operation 706, the execution operations 722, 726, 744, 908, the acquisition operations 730, 734, 912, and the switching operations 552, 564 may be performed by one or more of the operating system 252, 352, the GUI module 256, 356, the application module 262, 362, the digital assistant module 426, and the processor(s) 204, 304, 404. It will be clear to one skilled in the art how other processes may be implemented based on the components shown in Figures 1-3 and 4A-4B.

[0356] According to some embodiments, a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) is provided that stores one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for performing any of the methods described herein.

[0357] According to some embodiments, there is provided an electronic device (eg, a portable electronic device) that includes means for performing any of the methods described herein.

[0358] According to some embodiments, there is provided an electronic device (eg, a portable electronic device) that includes a processing unit configured to perform any of the methods described herein.

[0359] According to some embodiments, there is provided an electronic device (e.g., a portable electronic device) that includes one or more processors and memory that stores one or more programs executed by the one or more processors, the one or more programs including instructions to perform any of the methods described herein.

[0360] Exemplary methods, non-transitory computer-readable storage media, systems, and electronic devices are described in the following sections. 1. A method for operating a digital assistant in a media system, the method comprising: In an electronic device having one or more processors and a memory, Displaying content on a display unit; Detecting a user input; Determining whether the user input corresponds to a first input format; In response to determining that the user input corresponds to the first input format, Displaying a plurality of example natural language requests on a display unit, the example natural language requests being contextually related to the displayed content, and receiving a user utterance corresponding to one of the example natural language requests causing the digital assistant to perform a respective action; A method comprising: 2. The method of item 1, wherein the user input is detected on a remote control device of the electronic device. 3. The method of item 2, wherein the first form of input includes pressing a button on a remote control device and releasing the button within a predetermined period of time. 4. A method according to any one of items 1 to 3, wherein a plurality of exemplary natural language requests are displayed on a display unit via a first user interface, and the first user interface is superimposed on the displayed content. 5. The method of any one of items 1 to 4, wherein the displayed content includes media content, and the media content continues to play while displaying the plurality of exemplary natural language requests. 6. The method of any one of items 1 to 5, further comprising, in accordance with a determination that the user input corresponds to the first input format, displaying a visual indicator on the display unit indicating that the digital assistant is not processing the voice input. 7. The method according to any one of items 1 to 6, wherein, upon determining that the user input corresponds to the first input format, a plurality of exemplary natural language requests are displayed on the display unit after a predetermined time. 8. The method of any one of items 1 to 7, wherein each of the plurality of exemplary natural language requests is displayed separately at a different time in a predetermined order. 9. Displaying a plurality of example natural language requests 9. The method of any one of items 1 to 8, comprising displaying a plurality of lists of example natural language requests, each list being displayed in rotation at a different time. 10. In response to a determination that the user input does not correspond to the first input format, Determining whether the user input corresponds to a second input format; In response to determining that the user input corresponds to a second input format, Sampling audio data; determining whether the voice data includes a user request; performing a task that at least partially satisfies the user request in accordance with a determination that the voice data embodies the user request; 10. The method of any one of items 1 to 9, further comprising: 11. The method of claim 10, wherein the second form of input includes pressing a button on a remote control of the electronic device and holding the button down for longer than a predetermined period of time. 12. 12. The method of claim 10 or 11, further comprising displaying, on a display unit, a request for clarification of user intent in accordance with a determination that the voice data does not contain a user request. 13. The method of any one of items 10 to 12, wherein the displayed content includes media content, and the media content continues to play on the electronic device while sampling the audio data and while performing the task. 14. outputting audio associated with the media content; reducing an amplitude of the audio in accordance with determining that the user input corresponds to a second input format; Item 14. The method of item 13, further comprising: 15. The method of any one of items 10 to 14, wherein the task is performed without outputting speech related to the task from the electronic device. 16. The method of any one of items 10 to 15, wherein the audio data is sampled while detecting user input. 17. The method of any one of items 10 to 15, wherein the audio data is sampled for a predetermined period of time after detecting a user input. 18. Audio data is sampled via a first microphone on a remote control device of an electronic device, and the method comprises: sampling background audio data via a second microphone on the remote control device while sampling the audio data; removing background noise in the audio data using the background audio data; 18. The method of any one of items 10 to 17, further comprising: 19. Audio associated with the displayed content is output from the electronic device via an audio signal, and the method comprises: removing background noise in the audio data using the audio signal; 19. The method of any one of items 10 to 18, further comprising: 20. 20. The method of any one of items 10 to 19, further comprising, in response to detecting a user input, displaying a visual cue on the display unit prompting the user to provide a speech request. 21. The task to be performed is Obtaining a result that at least partially satisfies the user request; displaying a second user interface on a display unit, the second user interface including a portion of the results, wherein at least a portion of the content continues to be displayed while the second user interface is displayed, and a display area of ​​the second user interface on the display unit is smaller than a display area of ​​at least a portion of the content on the display unit; 21. The method according to any one of items 10 to 20, comprising: 22. The method of item 21, wherein a second user interface is overlaid on the displayed content. 23. The resulting portion includes one or more media items, and the method comprises: receiving a selection of a media item from the one or more media items via a second user interface; Displaying, on a display unit, media content associated with the selected media item; 23. The method according to item 21 or 22, further comprising: twenty four. Detecting a second user input while displaying the second user interface; ceasing to display the second user interface in response to detecting the second user input; 23. The method according to item 21 or 22, further comprising: 25. The method of claim 24, wherein a second user input is detected on a remote control device of the electronic device, the second user input comprising a first predetermined movement pattern on the touch-sensitive surface of the remote control device. 26. detecting a third user input while displaying the second user interface; in response to detecting a third user input, replacing the display of the second user interface with a display of a third user interface on the display unit, the third user interface including at least a portion of the results, the third user interface occupying at least a majority of a display area of ​​the display unit; 23. The method according to item 21 or 22, further comprising: 27. The method of claim 26, wherein a third user input is detected on a remote control device of the electronic device, the third user input comprising a second predetermined movement pattern on the touch-sensitive surface of the remote control device. 28. In response to detecting a third user input, 28. The method of claim 26 or 27, further comprising obtaining a second result different from the result, the second result at least partially satisfying the user request, and the third user interface including at least a portion of the second result. 29. The method of item 28, wherein the second result is based on a user request received before detecting the user input. 30. The method of item 28 or 29, wherein the focus of the second user interface is on an item in the results portion while a third user input is detected, and the second result is contextually related to the item. 31. The method of any one of items 26 to 30, wherein the displayed content includes media content, and in response to detecting a third user input, playback of the media content on the electronic device is paused. 32. At least a portion of the results include one or more media items, and the method comprises: receiving a selection of a media item from the one or more media items via a third user interface; Displaying media content associated with the media item on a display unit; 32. The method of any one of items 26 to 31, further comprising: 33. detecting a fourth user input associated with a direction on the display unit while displaying the third user interface; In response to detecting a fourth user input, Switching focus of the third user interface from the first item to a second item on the third user interface, the second item being positioned in the direction relative to the first item; and 33. The method of any one of items 26 to 32, further comprising: 34. detecting a fifth user input while displaying the third user interface; In response to detecting a fifth user input, Displaying a search field; displaying a virtual keyboard interface on the display unit, wherein input received via the virtual keyboard interface causes text entry into the search field; 34. The method of any one of items 26 to 33, further comprising: 35. detecting a sixth user input while displaying the third user interface; In response to detecting a sixth user input, sampling second audio data, the second audio data comprising a second user request; determining whether the second user request is a request to narrow the results of the user request; In response to determining that the second user request is a request to narrow the results of the user requests, Displaying the subset of results via a third user interface; and 35. The method of any one of items 26 to 34, further comprising: 36. The method of item 35, wherein a subset of the results is displayed in the top row of a third user interface. 37. In response to a determination that the second user request is not a request to narrow the results of the user requests, obtaining a third result that at least partially satisfies the second user request; and Displaying a portion of the third results via a third user interface; and 37. The method according to item 35 or 36, further comprising: 38. The method of item 37, wherein the third result portion is displayed at the top of the third user interface. 39. obtaining a fourth result that at least partially satisfies the user request or the second user request; displaying a portion of the fourth result via a third user interface; and 39. The method of any one of items 35 to 38, further comprising: 40. The method of item 39, wherein the fourth result portion is displayed in a row after the top row of the third user interface. 41. The method of item 39 or 40, wherein the focus of the third user interface is on one or more items in the third user interface while the sixth user input is detected, and the fourth result is contextually related to the one or more items. 42. detecting a seventh user input while displaying the third user interface; ceasing to display the third user interface in response to detecting the seventh user input; and 42. The method of any one of items 26 to 41, further comprising: 43. The method of item 42, wherein the displayed content is media content, and playback of the media content on the electronic device is paused in response to detecting a third user input, and playback of the media content on the electronic device is resumed in response to detecting a seventh user input. 44. The method of item 42 or 43, wherein the seventh user input includes pressing a menu button on a remote control device of the electronic device. 45. A method for operating a digital assistant in a media system, the method comprising: In an electronic device having one or more processors and a memory, Displaying content on a display unit; Detecting user input while displaying content; In response to detecting user input, Displaying a user interface on a display unit, the user interface including a plurality of example natural language requests contextually related to the displayed content, wherein receiving a user utterance corresponding to one of the plurality of example natural language requests causes the digital assistant to perform a respective action; A method comprising: 46. ​​The method of item 45, wherein the displayed content includes media content. 47. The method of claim 46, wherein the plurality of exemplary natural language requests includes a natural language request to change one or more settings associated with the media content. 48. The method of claim 46 or 47, wherein the media content continues to play while the user interface is displayed. 49. 49. The method of any one of items 46 to 48, further comprising outputting audio associated with the media content, wherein the amplitude of the audio is not reduced in response to detecting user input. 50. The method of item 45, wherein the displayed content includes a main menu user interface. 51. The method of item 50, wherein the plurality of exemplary natural language requests includes exemplary natural language requests associated with each of a plurality of core capabilities of the digital assistant. 52. The method of claim 45, wherein the displayed content includes a second user interface having results associated with a previous user request. 53. The method of item 52, wherein the plurality of exemplary natural language requests includes a natural language request to narrow results. 54. The method of any one of items 45 to 53, wherein the user interface includes textual instructions for invoking and interacting with the digital assistant. 55. The method of any one of items 45 to 54, wherein the user interface includes a visual indicator that indicates that the digital assistant is not receiving voice input. 56. The method of any one of items 45 to 55, wherein the user interface is overlaid on the displayed content. 57. 57. The method of any one of items 45 to 56, further comprising, in response to detecting a user input, reducing the brightness of the displayed content to enhance the user interface. 58. The method of any one of items 45 to 57, wherein the user input is detected on a remote control device of the electronic device. 59. The method of item 58, wherein the user input includes pressing a button on a remote control device and releasing the button within a predetermined period of time after pressing the button. 60. The method of item 59, wherein the button is configured to invoke a digital assistant. 61. The method of any one of items 45 to 60, wherein the user interface includes textual instructions for displaying a virtual keyboard interface. 62. detecting a second user input after displaying the user interface; In response to detecting the second user input, displaying a virtual keyboard interface on the display unit; 62. The method of any one of items 45 to 61, further comprising: 63. Item 63. The method of item 62, further comprising changing focus of the user interface to a search field on the user interface. 64. The method of item 63, wherein the search field is configured to receive a text search query via a virtual keyboard interface. 65. The method of any one of items 45 to 64, wherein a virtual keyboard interface cannot be used to interact with the digital assistant. 66. The method of any one of items 45 to 65, wherein the second user input comprises a predetermined movement pattern on a touch-sensitive surface of a remote control device of the electronic device. 67. The method of any one of items 45 to 66, wherein the plurality of exemplary natural language requests is displayed at a predetermined time after detecting a user input. 68. Displaying a plurality of example natural language requests Item 68. The method of any one of items 45 to 67, further comprising displaying each of the plurality of example natural language requests one by one in a predetermined order. 69. Displaying sequentially Item 69. The method of item 68, further comprising replacing a display of a previously displayed example natural language request of the plurality of example natural language requests with a subsequent example natural language request of the plurality of example natural language requests. 70. The method of any one of items 45 to 69, wherein the content includes a second user interface having one or more items, the focus of the second user interface is on an item of the one or more items when user input is detected, and the plurality of exemplary natural language requests are contextually related to the item of the one or more items. 71. A method for operating a digital assistant in a media system, the method comprising: In an electronic device having one or more processors and a memory, Displaying content on a display unit; Detecting a user input; In response to detecting user input, Displaying one or more example suggestions of natural language utterances that are contextually relevant to the displayed content and that, when uttered by the user, cause the digital assistant to perform a corresponding action; A method comprising: 72. Detecting a second user input; In response to detecting a second user input, Sampling audio data; determi...

Claims

1. 1. A method for operating a digital assistant in a media system, the method comprising: In an electronic device having one or more processors and a memory, Displaying content on a display unit; Detecting a user input; determining whether the user input corresponds to a first input format; in response to determining that the user input corresponds to a first input format, Displaying a plurality of exemplary natural language requests on the display unit, the plurality of exemplary natural language requests being contextually related to the displayed content, and receiving a user utterance corresponding to one of the plurality of exemplary natural language requests causes the digital assistant to perform a respective action; and The method includes:

2. The method of claim 1 , wherein the user input is detected on a remote control device of the electronic device.

3. The method of claim 2 , wherein the first form of input comprises pressing a button on the remote control device and releasing the button within a predetermined period of time.

4. The method of claim 1 , wherein the plurality of example natural language requests are displayed on the display unit via a first user interface, the first user interface being overlaid on the displayed content.

5. The method of claim 1 , wherein the displayed content comprises media content, and the media content continues to play while displaying the plurality of example natural language requests.

6. 10. The method of claim 1, further comprising: displaying a visual indicator on the display unit indicating that the digital assistant is not processing voice input in accordance with a determination that the user input corresponds to a first input format.

7. The method of claim 1 , wherein upon determining that the user input corresponds to a first input format, the plurality of example natural language requests are displayed on the display unit after a predetermined time.

8. The method of claim 1 , wherein each of the plurality of example natural language requests is displayed separately at a different time in a predetermined order.

9. Displaying the plurality of example natural language requests includes: The method of claim 1 , comprising displaying a plurality of lists of example natural language requests, each list being displayed in rotation at a different time.

10. in response to determining that the user input does not correspond to a first input format, determining whether the user input corresponds to a second input format; in response to determining that the user input corresponds to a second input format, Sampling the audio data; determining whether the voice data includes a user request; performing a task that at least partially satisfies a user request in accordance with a determination that the voice data contains a user request; and The method of claim 1 further comprising:

11. 11. The method of claim 10, wherein the second form of input comprises pressing a button on a remote control for the electronic device and holding the button down for longer than a predetermined period of time.

12. The method of claim 10 , further comprising: displaying, on the display unit, a request for clarification of user intent in accordance with a determination that the voice data does not contain a user request.

13. 11. The method of claim 10, wherein the displayed content comprises media content, and the media content continues to play on the electronic device while sampling the audio data and while performing the task.

14. outputting audio associated with the media content; reducing an amplitude of the audio in accordance with a determination that the user input corresponds to a second input format; The method of claim 13 further comprising:

15. The method of claim 10 , wherein the task is performed without outputting speech related to the task from the electronic device.

16. The method of claim 10 , wherein the audio data is sampled while detecting the user input.

17. The method of claim 10 , wherein the audio data is sampled for a predetermined period of time after detecting the user input.

18. The audio data is sampled via a first microphone on a remote control device of the electronic device, the method comprising: while sampling the audio data, sampling background audio data via a second microphone on the remote control device; removing background noise in the audio data using the background audio data; The method of claim 10 further comprising:

19. sound associated with the displayed content is output from the electronic device via an audio signal, the method comprising: The method of claim 10 , further comprising using the audio signal to remove background noise in the audio data.

20. The method of claim 10 , further comprising, in response to detecting the user input, displaying a visual cue on the display unit prompting a user to provide a speech request.

21. The task to be performed is obtaining a result that at least partially satisfies the user request; and displaying a second user interface on the display unit, the second user interface including a portion of the results, wherein at least a portion of the content continues to be displayed while the second user interface is displayed, and a display area of ​​the second user interface on the display unit is smaller than a display area of ​​the at least a portion of the content on the display unit; The method of claim 10, comprising:

22. The method of claim 21 , wherein the second user interface is overlaid on the displayed content.

23. wherein the portion of the results includes one or more media items, the method further comprising: receiving a selection of a media item of the one or more media items via the second user interface; displaying, on the display unit, media content associated with the selected media item; 22. The method of claim 21 further comprising:

24. detecting a second user input while displaying the second user interface; ceasing to display the second user interface in response to detecting the second user input; and 22. The method of claim 21 further comprising:

25. 25. The method of claim 24, wherein the second user input is detected on a remote control of the electronic device, the second user input comprising a first predetermined movement pattern on a touch-sensitive surface of the remote control.

26. detecting a third user input while displaying the second user interface; in response to detecting the third user input, replacing the display of the second user interface with a display of a third user interface on the display unit, the third user interface including at least the portion of the result, the third user interface occupying at least a majority of a display area of ​​the display unit; 22. The method of claim 21 further comprising:

27. 27. The method of claim 26, wherein the third user input is detected on a remote control of the electronic device, the third user input comprising a second predetermined movement pattern on a touch-sensitive surface of the remote control.

28. in response to detecting the third user input, 27. The method of claim 26, further comprising obtaining a second result different from the result, the second result at least partially satisfying the user request, and the third user interface including at least a portion of the second result.

29. 30. The method of claim 28, wherein the second result is based on a user request received prior to detecting the user input.

30. 30. The method of claim 28, wherein focus of the second user interface is on an item in the portion of results while the third user input is detected, and the second result is contextually related to the item.

31. 27. The method of claim 26, wherein the displayed content includes media content, and in response to detecting the third user input, playing of the media content on the electronic device is paused.

32. wherein at least the portion of the results includes one or more media items, the method further comprising: receiving a selection of a media item of the one or more media items via the third user interface; displaying, on the display unit, media content associated with the media item; 27. The method of claim 26, further comprising:

33. detecting a fourth user input associated with a direction on the display unit while displaying the third user interface; In response to detecting the fourth user input, switching focus of the third user interface from a first item to a second item on the third user interface, the second item being positioned in the direction relative to the first item; and 27. The method of claim 26, further comprising:

34. detecting a fifth user input while displaying the third user interface; In response to detecting the fifth user input, Displaying a search field; and displaying a virtual keyboard interface on the display unit, where input received via the virtual keyboard interface causes text entry into the search field; 27. The method of claim 26, further comprising:

35. detecting a sixth user input while displaying the third user interface; In response to detecting the sixth user input, sampling second audio data, the second audio data comprising a second user request; and determining whether the second user request is a request to narrow the results of the user requests; in response to determining that the second user request is a request to narrow the results of the user requests, displaying the subset of results via the third user interface; and 27. The method of claim 26, further comprising:

36. 36. The method of claim 35, wherein the subset of the results is displayed on a top tier of the third user interface.

37. in response to a determination that the second user request is not a request to narrow the results of the user requests, obtaining a third result that at least partially satisfies the second user request; and displaying a portion of the third results via the third user interface; and 36. The method of claim 35, further comprising:

38. 38. The method of claim 37, wherein the portion of the third result is displayed on a top tier of the third user interface.

39. obtaining a fourth result that at least partially satisfies the user request or the second user request; and displaying a portion of the fourth result via the third user interface; and 36. The method of claim 35, further comprising:

40. 40. The method of claim 39, wherein the portion of the fourth result is displayed in a row after the top row of the third user interface.

41. 40. The method of claim 39, wherein focus of the third user interface is on one or more items in the third user interface while the sixth user input is detected, and the fourth result is contextually related to the one or more items.

42. detecting a seventh user input while displaying the third user interface; ceasing to display the third user interface in response to detecting the seventh user input; and 27. The method of claim 26, further comprising:

43. 43. The method of claim 42, wherein the displayed content is media content, and in response to detecting the third user input, playback of the media content on the electronic device is paused, and in response to detecting the seventh user input, playback of the media content on the electronic device is resumed.

44. 43. The method of claim 42, wherein the seventh user input comprises pressing a menu button on a remote control for the electronic device.

45. 1. A method for operating a digital assistant in a media system, the method comprising: In an electronic device having one or more processors and a memory, Displaying content on a display unit; detecting a user input while displaying the content; In response to detecting the user input, Displaying a user interface on the display unit, the user interface including a plurality of example natural language requests contextually related to the displayed content, and receiving a user utterance corresponding to one of the plurality of example natural language requests causes the digital assistant to perform a respective action; and The method includes:

46. 46. ​​The method of claim 45, wherein the displayed content comprises media content.

47. 47. The method of claim 46, wherein the plurality of exemplary natural language requests comprises a natural language request to change one or more settings associated with the media content.

48. 47. The method of claim 46, wherein the media content continues to play while the user interface is displayed.

49. 47. The method of claim 46, further comprising: outputting an audio associated with the media content, the audio not being reduced in amplitude in response to detecting the user input.

50. 46. ​​The method of claim 45, wherein the displayed content includes a main menu user interface.

51. 51. The method of claim 50, wherein the plurality of example natural language requests includes example natural language requests associated with each of a plurality of core capabilities of the digital assistant.

52. 46. ​​The method of claim 45, wherein the displayed content includes a second user interface having results associated with a previous user request.

53. 53. The method of claim 52, wherein the plurality of exemplary natural language requests includes a natural language request to refine the results.

54. 46. ​​The method of claim 45, wherein the user interface includes textual instructions for invoking and interacting with the digital assistant.

55. 46. ​​The method of claim 45, wherein the user interface includes a visual indicator that indicates that the digital assistant is not receiving voice input.

56. 46. ​​The method of claim 45, wherein the user interface is overlaid on the displayed content.

57. 46. ​​The method of claim 45, further comprising, in response to detecting the user input, reducing a brightness of the displayed content to accentuate the user interface.

58. 46. ​​The method of claim 45, wherein the user input is detected on a remote control device of the electronic device.

59. 60. The method of claim 58, wherein the user input comprises pressing a button on the remote control device and releasing the button within a predetermined period of time after pressing the button.

60. 60. The method of claim 59, wherein the button is configured to invoke the digital assistant.

61. 46. ​​The method of claim 45, wherein the user interface includes textual instructions for displaying a virtual keyboard interface.

62. detecting a second user input after displaying the user interface; displaying a virtual keyboard interface on the display unit in response to detecting the second user input; 46. ​​The method of claim 45, further comprising:

63. 63. The method of claim 62, further comprising changing focus of the user interface to a search field on the user interface.

64. 64. The method of claim 63, wherein the search field is configured to receive a text search query via the virtual keyboard interface.

65. 46. ​​The method of claim 45, wherein the virtual keyboard interface cannot be used to interact with the digital assistant.

66. 46. ​​The method of claim 45, wherein the second user input comprises a predetermined movement pattern on a touch-sensitive surface of a remote control device of the electronic device.

67. 46. ​​The method of claim 45, wherein the plurality of example natural language requests is displayed at a predetermined time after detecting the user input.

68. Displaying the plurality of example natural language requests includes:

46. ​​The method of claim 45, further comprising displaying each of the plurality of example natural language requests one at a time in a predetermined order.

69. The sequential displaying 70. The method of claim 68, further comprising replacing a display of a previously displayed example natural language request of the plurality of example natural language requests with a subsequent example natural language request of the plurality of example natural language requests.

70. 46. ​​The method of claim 45, wherein the content includes a second user interface having one or more items, a focus of the second user interface is on an item of the one or more items when the user input is detected, and the plurality of example natural language requests are contextually related to the item of the one or more items.

71. 1. A method for operating a digital assistant in a media system, the method comprising: In an electronic device having one or more processors and a memory, Displaying content on a display unit; Detecting a user input; In response to detecting the user input, Displaying one or more example suggestions of natural language utterances that are contextually relevant to the displayed content and that, when uttered by the user, cause the digital assistant to perform a corresponding action; The method includes:

72. Detecting a second user input; In response to detecting the second user input, Sampling the audio data; determining whether the sampled speech data includes one of the one or more suggested examples of natural language utterances; in response to a determination that the sampled speech data contains one of the one or more suggested examples of a natural language utterance, performing the corresponding action on the utterance; 72. The method of claim 71, further comprising:

73. 1. A method for operating a digital assistant in a media system, the method comprising: In an electronic device having one or more processors and a memory, Displaying content on a display unit; detecting a user input while displaying the content; In response to detecting the user input, sampling voice data, the voice data including user utterances; obtaining a determination of a user intent corresponding to the user utterance; and obtaining a determination whether the user intent includes a request to adjust a state or setting of an application on the electronic device; In response to obtaining a determination that the user intent includes a request to adjust a state or setting of an application on the electronic device, adjusting the state or the setting of the application to satisfy the user intent; The method includes:

74. 74. The method of claim 73, wherein the request to adjust a state or setting of an application on the electronic device comprises a request to play a particular media item, and adjusting the state or setting of the application to satisfy the user intent comprises playing the particular media item.

75. The displayed content includes a user interface having media items, and the user utterances do not explicitly limit the particular media items to be played, the method comprising: determining whether focus of the user interface is on the media item; identifying the media item as the particular media item to be played in accordance with determining that focus of the user interface is on the media item; 75. The method of claim 74, further comprising:

76. 74. The method of claim 73, wherein the request to adjust a state or setting of an application on the electronic device comprises a request to launch the application on the electronic device.

77. 74. The method of claim 73, wherein the displayed content includes media content currently playing on the electronic device, and the state or setting is related to the media content currently playing on the electronic device.

78. 80. The method of claim 77, wherein the request to adjust a state or setting of an application on the electronic device comprises a request to fast forward or rewind the media content being played on the electronic device.

79. 78. The method of claim 77, wherein the request to adjust a state or setting of an application on the electronic device comprises a request to skip forward or backward within the media content to play a particular portion of the media content.

80. 80. The method of claim 77, wherein the request to adjust a state or setting of an application on the electronic device comprises a request to pause the playing of the media content on the electronic device.

81. 80. The method of claim 77, wherein the request to adjust a state or setting of an application on the electronic device comprises a request to turn on or off subtitles for the media content.

82. the displayed content includes a user interface having a first media item and a second media item; the request to adjust a state or setting of an application on the electronic device comprises a request to switch focus of the user interface from the first media item to the second media item; adjusting the state or the settings of the application to satisfy the user intent includes switching focus of the user interface from the first media item to the second media item.

74. The method of claim 73.

83. the displayed content includes media content currently playing on the media device; the user utterance being a natural language expression indicating that the user did not hear a portion of the audio associated with the media content; wherein the request to adjust a state or setting of an application on the electronic device includes a request to replay a portion of the media content that corresponds to the portion of the audio that the user could not hear; adjusting the state or the settings of the application; rewinding the media content a predetermined amount to an earlier portion of the media content; restarting playback of the media content from the previous portion; and 74. The method of claim 73, comprising:

84. adjusting the state or the settings of the application; 84. The method of claim 83, further comprising turning on closed captioning before restarting playback of the media content from the previous portion.

85. the request to adjust a state or setting of an application on the electronic device further comprises a request to increase a volume of the audio associated with the media content; and adjusting the state or settings of the application further comprises increasing the volume of the audio associated with the media content before restarting playback of the media content from the previous portion.

84. The method of claim 83.

86. converting speech in the audio associated with the media content into text; adjusting the state or the settings of the application further comprises displaying the portion of the text while restarting playback of the media content from the previous portion.

84. The method of claim 83.

87. obtaining a determination of a user intent corresponding to the user utterance; 74. The method of claim 73, further comprising obtaining a determination of a user emotion associated with the user utterance, the user intent being determined based on the determined user emotion.

88. In response to obtaining a determination that the user intent does not include a request to adjust a state or setting of an application on the electronic device, obtaining a determination whether the user intent is one of a plurality of predetermined request types; In response to obtaining a determination that the user intent is one of a plurality of predetermined request types, obtaining a result that at least partially satisfies the user intent; and displaying the results in text format on the display unit; 74. The method of claim 73, further comprising:

89. 90. The method of claim 88, wherein the plurality of predefined request types includes a request for a current time at a particular location.

90. 90. The method of claim 88, wherein the plurality of predetermined request types includes a request to submit a joke.

91. 90. The method of claim 88, wherein the plurality of predetermined request types includes a request for information about media content being played on the electronic device.

92. 90. The method of claim 88, wherein the results in text form are overlaid on the displayed content.

93. 90. The method of claim 88, wherein the displayed content comprises media content currently playing on the electronic device, the media content continuing to play while the results in text form are displayed.

94. In response to obtaining a determination that the user intent is not one of a plurality of predetermined request types, obtaining a second result that at least partially satisfies the user intent; and determining whether the displayed content includes media content currently playing on the electronic device; In response to a determination that the displayed content includes media content, determining whether the media content can be paused; displaying, in accordance with a determination that the media content cannot be paused, a second user interface on the display unit having a portion of the second result, wherein a display area occupied by the second user interface on the display unit is smaller than a display area occupied by the media content on the display unit; 89. The method of claim 88, further comprising:

95. 95. The method of claim 94, wherein the user intent includes a request for a weather forecast for a particular location.

96. 95. The method of claim 94, wherein the user intent includes a request for information associated with a sports team or athlete.

97. 95. The method of claim 94, wherein the user intent is not a media search query and the second results include one or more media items having media content that at least partially satisfies the user intent.

98. 100. The method of claim 97, wherein the second result further comprises non-media data that at least partially satisfies the user intent.

99. 95. The method of claim 94, wherein the user intent is a media search query and the second results include a plurality of media items that correspond to the media search query.

100. 95. The method of claim 94, further comprising: displaying on the display unit a third user interface having a portion of the second result in accordance with a determination that the displayed content does not include media content currently being played on the electronic device, the third user interface occupying a majority of the display area of ​​the display unit.

101. 101. The method of claim 100, wherein the display content includes a main menu user interface.

102. the displayed content includes the third user interface having previous results associated with previous user requests received prior to detecting the user input; a display of the previous result in the third user interface is replaced with the display of the second result in accordance with a determination that the displayed content does not include media content currently playing on the electronic device. The method of claim 100.

103. In response to the determination that the displayed content includes media content currently playing on the electronic device, 95. The method of claim 94, further comprising: determining whether the displayed content includes the second user interface having previous results from a previous user request; and, in accordance with a determination that the displayed content includes the second user interface having previous results from a previous user request, the previous results are replaced with the second results.

104. In response to a determination that the media content may be paused, pausing the playing of the media content on the electronic device; displaying on the display unit the third user interface having a portion of the second results, the third user interface occupying a majority of the display area of ​​the display unit; and 95. The method of claim 94, further comprising:

105. transmitting the speech data to a server for performing natural language processing; indicating to the server that the voice data is associated with a media application, the indicating biasing the natural language processing toward media-related user intent; 74. The method of claim 73, further comprising:

106. 74. The method of claim 73, further comprising transmitting the audio data to a server for performing speech-to-text processing.

107. 107. The method of claim 106, further comprising: instructing the server that the audio data is associated with a media application, said instructing biasing the speech-to-text process toward media-related text results.

108. 107. The method of claim 106, further comprising obtaining a textual representation of the user utterance, the textual representation being based on a previous user utterance received prior to sampling the audio data.

109. 109. The method of claim 108, wherein the textual representation is based on a time that the previous user utterance was received prior to sampling the audio data.

110. obtaining a determination that the user intent does not correspond to one of a plurality of core capabilities associated with the electronic device; causing a second electronic device to perform a task to facilitate satisfying the user intent; and 74. The method of claim 73, further comprising:

111. obtaining the determination of user intent, obtaining a determination of whether the user utterances include ambiguous terms; In response to obtaining a determination that the user utterance includes an ambiguous term, obtaining two or more candidate user intents based on the ambiguous terms; and displaying the two or more candidate user intentions on the display unit; and 74. The method of claim 73, further comprising:

112. 112. The method of claim 111, further comprising receiving a user selection of one of the two or more candidate user intents while displaying the two or more candidate user intents, wherein the user intent is determined based on the user selection.

113. receiving the user selection, Detecting a second user input; In response to detecting the second user input, sampling second audio data, the second audio data including a second user utterance expressing the user selection; and 113. The method of claim 112, further comprising:

114. 112. The method of claim 111, wherein the two or more interpretations are displayed without outputting utterances associated with the two or more candidate user intents.

115. 1. A method for operating a digital assistant in a media system, the method comprising: In an electronic device having one or more processors and a memory, Displaying content on a display unit; detecting a user input while displaying the content; In response to detecting the user input, sampling voice data, the voice data including user utterances expressing a media search request; obtaining a plurality of media items satisfying the media search request; displaying at least a portion of the plurality of media items on the display unit via a user interface; The method includes:

116. 116. The method of claim 115, wherein the content continues to be displayed on the display unit while the at least a portion of the plurality of media items is being displayed, and a display area occupied by the user interface is smaller than a display area occupied by the content.

117. 116. The method of claim 115, further comprising determining whether a number of media items in the plurality of media items is less than or equal to a predetermined number, and wherein in accordance with a determination that the number of media items in the plurality of media items is less than or equal to a predetermined number, the at least a portion of the plurality of media items includes the plurality of media items.

118. 118. The method of claim 117, wherein, in accordance with a determination that a number of media items in the plurality of media items is greater than a predetermined number, a number of media items in the at least a portion of the plurality of media items is equal to the predetermined number.

119. 116. The method of claim 115, wherein each of the plurality of media items is associated with a relevance score for the media search request, and the relevance score of the at least a portion of the plurality of media items is highest among the plurality of media items.

120. 116. The method of claim 115, wherein each of the at least a portion of the plurality of media items is associated with a popularity rating, and the at least a portion of the plurality of media items are arranged within the user interface based on the popularity rating.

121. detecting a second user input while displaying the at least a portion of the plurality of media items; In response to detecting the second user input, expanding the user interface to occupy at least a majority of a display area of ​​the display unit; 116. The method of claim 115, further comprising:

122. In response to detecting the second user input, determining whether a number of media items in the plurality of media items is less than or equal to a predetermined number; in response to determining that a number of media items in the plurality of media items is less than or equal to a predetermined number; obtaining a second plurality of media items that at least partially satisfy the media search request, the second plurality of media items being different from the at least a portion of the media items; and displaying the second plurality of media items on the display unit via the enlarged user interface; 122. The method of claim 121, further comprising:

123. 123. The method of claim 122, further comprising determining whether the media search request includes more than one search parameter, and in accordance with the determination that the media search request includes more than one search parameter, the second plurality of media items are organized within the enlarged user interface according to the one or more search parameters of the media search request.

124. in response to determining that a number of media items in the plurality of media items is greater than the predetermined number, 123. The method of claim 122, further comprising displaying at least a second portion of the plurality of media items via the enlarged user interface, the at least second portion of the plurality of media items being different from the at least a portion of the plurality of media items.

125. 125. The method of claim 124, wherein the at least a second portion of the plurality of media items includes two or more media types, and the at least a second portion of the plurality of media items is organized within the enlarged user interface according to each media type of the two or more media types.

126. detecting a third user input; in response to detecting the third user input, scrolling the enlarged user interface; and determining whether the magnified user interface has scrolled past a predetermined position on the magnified user interface; displaying at least a third portion of the plurality of media items on the magnified user interface in response to determining that the magnified user interface has scrolled past a predetermined position on the magnified user interface, the at least third portion of the plurality of media items being organized on the magnified user interface according to one or more media content providers associated with the third plurality of media items; 123. The method of claim 122, further comprising:

127. 1. A method for operating a digital assistant in a media system, the method comprising: In an electronic device having one or more processors and a memory, Displaying content on a display unit; detecting a user input while displaying the content; In response to detecting the user input, displaying a virtual keyboard interface on the display unit; causing a selectable affordance to appear on a display of a second electronic device, selection of the affordance enabling text input to be received by the second electronic device via a keyboard of the second electronic device; The method includes:

128. receiving a text input via the keyboard of the second electronic device, the text input expressing a user request; obtaining a result that at least partially satisfies the user request; and displaying a user interface on the display unit, the user interface including at least a portion of the results; and 128. The method of claim 127, further comprising:

129. 128. The method of claim 127, wherein the displayed content includes a second user interface having a plurality of example natural language requests.

130. 130. The method of claim 129, wherein the displayed content comprises media content.

131. 128. The method of claim 127, wherein the displayed content includes a third user interface having results from a previous user request, the third user interface occupying at least a majority of a display area of ​​the display unit.

132. 132. The method of claim 131, wherein the virtual keyboard interface is overlaid on at least a portion of the third user interface.

133. 128. The method of claim 127, wherein the user input is detected via a remote control device for the electronic device, the remote control device and the second electronic device being different devices.

134. 134. The method of claim 133, wherein the user input comprises a predetermined movement pattern on a touch-sensitive surface of the remote control device.

135. 128. The method of claim 127, wherein the user input is detected via the second electronic device.

136. 136. A non-transitory computer-readable storage medium comprising computer-executable instructions for performing the method of any one of claims 1 to 135.

137. 137. A non-transitory computer readable storage medium according to claim 136; a processor configured to execute the computer-executable instructions; and A system comprising:

138. 136. Apparatus comprising means for carrying out the method of any one of claims 1 to 135.

139. 1. An electronic device comprising: an input unit configured to receive a user input; a processing unit coupled to the input unit; The processing unit comprises: Displaying the content on a display unit; Detecting a user input via the input unit; determining whether the user input corresponds to a first input format; in response to determining that the user input corresponds to a first input format, Displaying a plurality of exemplary natural language requests on the display unit. wherein the plurality of exemplary natural language requests are contextually related to the displayed content, and receiving a user utterance corresponding to one of the plurality of exemplary natural language requests causes the digital assistant to perform a respective action.

140. A voice input unit coupled to the processing unit, the processing unit comprising: in response to determining that the user input does not correspond to a first input format, determining whether the user input corresponds to a second input format; in response to determining that the user input corresponds to a second input format, Sampling audio data using the audio input unit; determining whether the voice data comprises a user request; In response to determining that the voice data contains a user request, perform a task that at least partially satisfies the user request.

140. The electronic device of claim 139, further configured as follows:

141. The processing unit comprises: Obtaining results that at least partially satisfy the user request; Displaying a second user interface on the display unit 140. The electronic device of claim 139, wherein the second user interface includes a portion of the results, and wherein at least a portion of the content continues to be displayed while the second user interface is displayed, and wherein a display area of ​​the second user interface on the display unit is smaller than a display area of ​​the at least a portion of the content on the display unit.

142. The processing unit comprises: detecting a second user input via the input unit while displaying the second user interface; ceasing to display the second user interface in response to detecting the second user input.

142. The electronic device of claim 141, further configured as follows:

143. The processing unit comprises: detecting a third user input via the input unit while displaying the second user interface; In response to detecting the third user input, replacing the display of the second user interface with a display of a third user interface on the display unit.

142. The electronic device of claim 141, further configured such that the third user interface includes at least the portion of the results, and the third user interface occupies at least half of a display area of ​​the display unit.

144. The processing unit comprises: detecting, via the input unit, a fourth user input associated with a direction on the display unit while displaying the third user interface; In response to detecting the fourth user input, Switching focus of the third user interface from a first item to a second item on the third user interface.

144. The electronic device of claim 143, further configured such that the second item is positioned in the direction relative to the first item.

145. The processing unit comprises: detecting a fifth user input via the input unit while displaying the third user interface; In response to detecting the fifth user input, Display the search field, Displaying a virtual keyboard interface on the display unit 144. The electronic device of claim 143, configured to:

146. The processing unit comprises: detecting a sixth user input via the input unit while displaying the third user interface; In response to detecting the sixth user input, sampling second voice data, the second voice data comprising a second user request; determining whether the second user request is a request to narrow the results of the user requests; in response to determining that the second user request is a request to narrow the results of the user requests, Displaying the subset of results via the third user interface.

144. The electronic device of claim 143, further configured as follows:

147. 1. An electronic device comprising: an input unit configured to receive a user input; an audio input unit configured to receive audio data; a processing unit coupled to the input unit and the audio input unit, the processing unit comprising: Displaying the content on a display unit; detecting a user input via the input unit while displaying the content; responsive to detecting the user input, sampling speech data using the speech input unit, the sampled speech data including user utterances; Obtaining a determination of a user intent corresponding to the user utterance; Obtaining a determination whether the user intent includes a request to adjust a state or setting of an application on the electronic device; In response to obtaining a determination that the user intent includes a request to adjust a state or setting of an application on the electronic device, adjusting the state or the setting of the application to satisfy the user intent. The electronic device is configured to:

148. The processing unit comprises: In response to obtaining a determination that the user intent does not include a request to adjust a state or setting of an application on the electronic device, obtaining a determination whether the user intent is one of a plurality of predetermined request types; In response to obtaining a determination that the user intent is one of a plurality of predetermined request types, obtaining a result that at least partially satisfies the user intent; Displaying the results in text format on the display unit.

148. The electronic device of claim 147, further configured as follows:

149. The processing unit comprises: In response to obtaining a determination that the user intent is not one of a plurality of predetermined request types, obtaining a second result that at least partially satisfies the user intent; determining whether the displayed content includes media content currently playing on the electronic device; In response to a determination that the displayed content includes media content, determining whether the media content can be paused; displaying, on the display unit, a second user interface having a portion of the second result in accordance with determining that the media content cannot be paused.

149. The electronic device of claim 148, further configured such that a display area occupied by the second user interface on the display unit is smaller than a display area occupied by the media content on the display unit.

150. The processing unit comprises:

150. The electronic device of claim 149, further configured to display on the display unit a third user interface having a portion of the second result in accordance with a determination that the displayed content does not include media content currently being played on the electronic device, the third user interface occupying a majority of the display area of ​​the display unit.

151. The processing unit comprises: In response to a determination that the media content may be paused, pausing the playing of the media content on the electronic device; Displaying the third user interface on the display unit, the third user interface having a portion of the second result. and wherein the third user interface occupies a majority of the display area of ​​the display unit.

150. The electronic device of claim 149.

152. 1. An electronic device comprising: an input unit configured to receive a user input; a processing unit coupled to the input unit; The processing unit comprises: Displaying the content on a display unit; detecting a user input via the input unit while displaying the content; In response to detecting the user input, displaying a virtual keyboard interface on the display unit; Making a selectable affordance appear on a display of a second electronic device wherein selection of the affordance enables text input to be received by the electronic device via a keyboard of the second electronic device.

153. The processing unit comprises: receiving text input via the keyboard of the second electronic device, the text input representing a user request; Obtaining results that at least partially satisfy the user request; Displaying a user interface on the display unit 153. The electronic device of claim 152, further configured such that the user interface includes at least a portion of the results.

Images