Intelligent automated assistants in media environments

Abstract

The present invention discloses a system and process for operating a digital assistant in a media environment. In an exemplary embodiment, a user can interact with a digital assistant of a media device while content is displayed by the media device. In one method, multiple exemplary natural language requests can be displayed in response to detecting a user input of a first input type. The multiple exemplary natural language requests can be contextually related to the displayed content. In another method, a user request can be received in response to detecting a user input of a second input type. A task that at least partially satisfies the user request can be performed. The task performed can depend on the nature of the user request and the content being displayed by the media device. Specifically, the user request can be satisfied while reducing interference in the user's consumption of media content.

Description

[0001] Cross-references to related patent applications

[0002] This application is a divisional application of an invention patent application with an international application date of August 16, 2016, which entered the Chinese national phase on April 20, 2017, Chinese national application number 201680003291.7, and invention name “Intelligent Automated Assistant in Media Environment”.

[0003] This patent application claims priority to U.S. Provisional Serial No. 62 / 215,676, filed on September 8, 2015, and entitled “Intelligent Automated Assistant in a Media Environment,” and Provisional Serial No. 14 / 963,094, filed on December 8, 2015, and entitled “Intelligent Automated Assistant in a Media Environment,” which are hereby incorporated by reference in their entirety for all purposes.

[0004] This patent application is related to the following co-pending patent applications: U.S. non-provisional patent application serial number 14 / 963,089, filed on December 8, 2015, “Intelligent Automated Assistant for Media Search and Playback” (Agent Docket Number 106842137900 (P27499US1)); U.S. non-provisional patent application serial number 14 / 498,503, filed on September 26, 2014, “Intelligent Automated Assistant for TV User Interactions” (Agent Docket Number 106842065100 (P18133US1)); and U.S. non-provisional patent application serial number 14 / 498,391, filed on September 26, 2014, “Real-time Digital Assistant Knowledge Updates” (Agent Docket Number 106842097900 (P22498US1)), all three of which are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0005] The present invention relates generally to intelligent automated assistants, and more particularly to intelligent automated assistants operating in a media environment. Background Art

[0006] Intelligent automated assistants (or digital assistants) can provide an intuitive interface between a user and an electronic device. These assistants can allow a user to interact with a device or system using natural language in voice form and / or text form. For example, a user can access the services of an electronic device by providing verbal user input in natural language form to a virtual assistant associated with the electronic device. The virtual assistant can perform natural language processing on the verbal user input to infer the user's intent and implement the user's intent into the task. The task can then be performed by executing one or more functions of the electronic device, and in some examples, the relevant output can be returned to the user in natural language form.

[0007] It is desirable to integrate digital assistants into media environments (e.g., televisions, television set-top boxes, cable boxes, gaming devices, streaming devices, digital video recorders, etc.) to assist users in performing tasks associated with media consumption. For example, a digital assistant may be used to assist in finding desired media content for consumption. However, user interaction with a digital assistant may include audio and video outputs that may interfere with the consumption of media content. Therefore, it is challenging to integrate a digital assistant into a media environment in a manner that provides sufficient assistance to the user while minimizing interference with the consumption of media content. Summary of the invention

[0008] The present invention discloses a system and process for operating a digital assistant in a media environment. In some exemplary processes, user input may be detected while displaying content. The process may determine whether the user input corresponds to a first input type. Based on determining that the user input corresponds to the first input type, multiple exemplary natural language requests may be displayed. The multiple exemplary natural language requests may be contextually related to the displayed content.

[0009] In some embodiments, based on determining that the user input does not correspond to the first input type, the process may determine whether the user input corresponds to the second input type. Based on determining that the user input corresponds to the second input type, the audio data may be sampled. The process may determine whether the audio data contains a user request. Based on determining that the audio data contains a user request, a task that at least partially satisfies the user request may be performed. In some examples, the task may include obtaining a result that at least partially satisfies the user request, and displaying a second user interface with a portion of the result. A portion of the content may continue to be displayed while the second user interface is displayed, and the display area of ​​the second user interface may be smaller than the display area of ​​the portion of the content.

[0010] In some embodiments, a third user input may be detected while displaying the second user interface. In response to detecting the third user input, the display of the second user interface may be replaced with the display of a third user interface having the portion of the result. The third user interface may occupy at least a majority of the display area of ​​the display unit. In addition, a second result that at least partially satisfies the user request may be obtained. The second result may be different from the result. The third user interface may include at least a portion of the second result.

[0011] In some embodiments, a fourth user input may be detected while the third user interface is displayed. The fourth user input may indicate a direction. In response to detecting the fourth user input, the focus of the third user interface may be switched from a first item in the third user interface to a second item in the third user interface. The second item may be positioned in the indicated direction relative to the first item.

[0012] In some embodiments, a fifth user input may be detected while displaying the third user interface. In response to detecting the fifth user input, a search field may be displayed. In addition, a virtual keyboard interface may be displayed, wherein input received via the virtual keyboard interface results in text input in the search field. In addition, in some embodiments, a selectable affordance may be caused to appear on the display of the second electronic device, wherein selection of the affordance causes the text input to be received by the electronic device via the keyboard of the second electronic device.

[0013] In some embodiments, a sixth user input may be detected while displaying the third user interface. In response to detecting the sixth user input, second audio data containing a second user request may be sampled. The process may determine whether the second user request is a request for refining the results of the user request. Based on determining that the second user request is a request for refining the results of the user request, a subgroup of the results may be displayed via the third user interface. Based on determining that the second user request is not a request for refining the results of the user request, a third result that at least partially satisfies the second user request may be obtained. A portion of the third result may be displayed via the third user interface.

[0014] In some embodiments, the sampled audio data may include a user utterance, and a user intent corresponding to the user utterance may be determined. The process may determine whether the user intent includes a request to adjust a state or setting of an application. Based on determining that the user intent includes a request to adjust a state or setting of an application, the state or setting of the application may be adjusted to satisfy the user intent.

[0015] In some embodiments, based on determining that the user intent does not include a request to adjust a state or setting of an application on the electronic device, the process can determine whether the user intent is a predetermined request type from a plurality of predetermined request types. Based on determining that the user intent is a predetermined request type from a plurality of predetermined request types, a plain text result that at least partially satisfies the user intent can be displayed.

[0016] In some embodiments, based on determining that the user intent is not a predetermined request type among a plurality of predetermined request types, the process may determine whether the displayed content includes media content. Based on determining that the displayed content includes media content, the process may further determine whether the media content can be paused. Based on determining that the media content can be paused, the media content is paused, and a result that at least partially satisfies the user intent may be displayed via a third user interface. The third user interface may occupy at least a majority of the display area of ​​the display unit. Based on determining that the media content cannot be paused, the result may be displayed via a second user interface when the media content is displayed. The display area occupied by the second user interface may be smaller than the display area occupied by the media content. In addition, in some embodiments, based on determining that the displayed content does not include media content, the result may be displayed via a third user interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A block diagram of a system and environment for implementing a digital assistant according to various examples is shown.

[0018] Figure 2 A block diagram of a media system according to various examples is shown.

[0019] Figure 3 A block diagram of a user device according to various examples is shown.

[0020] Figure 4A A block diagram of a digital assistant system or a server portion thereof according to various examples is shown.

[0021] Figure 4B Various examples are shown Figure 4A Functions of the digital assistant shown.

[0022] Figure 4C A portion of an ontology according to various examples is shown.

[0023] FIG. 5A to FIG. 5I A process for operating a digital assistant for a media system according to various examples is shown.

[0024] FIG. 6A to FIG. 6Q According to various examples, FIG. 5A to FIG. 5I6O is intentionally omitted to avoid any confusion between the capital letter O and the number 0 (zero).

[0025] 7A to 7C A process for operating a digital assistant for a media system according to various examples is shown.

[0026] FIG. 8A to FIG. 8W According to various examples, 7A to 7C 8O is intentionally omitted to avoid any confusion between the capital letter O and the number 0 (zero).

[0027] Fig. 9 A process for operating a digital assistant for a media system according to various examples is shown.

[0028] Fig.10 A functional block diagram of an electronic device configured as a digital assistant to operate a media system according to various examples is shown.

[0029] Fig.11 A functional block diagram of an electronic device configured as a digital assistant to operate a media system according to various examples is shown.

[0030] Fig.12 is a block diagram illustrating a system and environment for implementing a digital assistant according to various examples.

[0031] Fig.13 is a block diagram illustrating a media system according to various examples.

[0032] Fig.14 is a block diagram illustrating a user device according to various examples.

[0033] Fig.15A is a block diagram illustrating a digital assistant system or a server portion thereof according to various examples.

[0034] Fig. 15B Various examples are shown Fig.15A The capabilities of the digital assistant shown in .

[0035] Fig. 15C A portion of an ontology according to various examples is shown.

[0036] FIG. 16A to FIG. 16E A process for operating a digital assistant for a media system according to various examples is shown.

[0037] FIG. 17A to FIG. 17K According to various examples, FIG. 16A to FIG. 16EThe various stages of the process shown in the screenshots are displayed by the media device on the display unit.

[0038] Fig.18 A functional block diagram of an electronic device configured as a digital assistant to operate a media system according to various examples is shown.

[0039] Fig.19 An exemplary system for controlling television user interactions using a virtual assistant is shown.

[0040] Fig. 20 A block diagram of an exemplary user device according to various examples is shown.

[0041] Fig.21 A block diagram of an exemplary media control device in a system for controlling television user interactions is shown.

[0042] FIG. 22A to FIG. 22E An exemplary voice input interface over video content is shown.

[0043] Fig.23 An exemplary media content interface over video content is shown.

[0044] FIG. 24A to FIG. 24B An exemplary media details interface on video content is shown.

[0045] FIG. 25A to FIG. 25B An exemplary media transition interface is shown.

[0046] FIG. 26A to FIG. 26B An exemplary voice input interface on menu content is shown.

[0047] Fig. 27 An exemplary virtual assistant results interface on menu content is shown.

[0048] Fig.28 An exemplary process of using a virtual assistant to control television interactions and using different interfaces to display associated information is shown.

[0049] Fig.29 Exemplary television media content on a mobile user device is shown.

[0050] Fig.30 Exemplary television control using a virtual assistant is shown.

[0051] Fig.31 Exemplary picture and video content on a mobile user device is shown.

[0052] Fig.32 Exemplary media display control using a virtual assistant is shown.

[0053] Fig.33An exemplary virtual assistant interaction is shown with results on a mobile user device and a media display device.

[0054] Fig.34 An exemplary virtual assistant interaction is shown with media results on a media display device and a mobile user device.

[0055] Fig.35 Exemplary proximity-based media device control is shown.

[0056] Fig.36 An exemplary process for controlling television interactions using a virtual assistant and multiple user devices is shown.

[0057] Fig.37 An exemplary voice input interface with a virtual assistant query regarding background video content is shown.

[0058] Fig.38 An exemplary informative virtual assistant response on video content is shown.

[0059] Fig.39 An exemplary voice input interface with a virtual assistant query for media content associated with background video content is shown.

[0060] Fig.40 An exemplary virtual assistant response interface with selectable media content is shown.

[0061] FIG. 41A to FIG. 41B An exemplary page of a program menu is shown.

[0062] Fig.42 An exemplary media menu divided into categories is shown.

[0063] Fig.43 An exemplary process for controlling television interaction using media content viewing history and media content shown on a display is shown.

[0064] Fig.44 An exemplary interface with virtual assistant query suggestions based on background video content is shown.

[0065] Fig.45 An exemplary interface for confirming selection of a suggested query is shown.

[0066] FIG. 46A to FIG. 46B An exemplary virtual assistant answer interface based on a selected query is shown.

[0067] Fig.47 Shown are media content notifications and an exemplary interface with virtual assistant query suggestions based on the notifications.

[0068] Fig.48A mobile user device is shown with exemplary picture and video content that can be played on a media control device.

[0069] Fig.49 An exemplary mobile user device interface with virtual assistant query suggestions based on playable user device content and based on video content shown on a separate display is shown.

[0070] Fig.50 An exemplary interface with virtual assistant query suggestions based on playable content from a separate user device is shown.

[0071] Fig.51 An exemplary process for suggesting a virtual assistant interaction for controlling media content is shown.

[0072] Fig.52 A functional block diagram of an electronic device configured to control television interactions using a virtual assistant and display associated information using different interfaces according to various examples is shown.

[0073] Fig.53 A functional block diagram of an electronic device configured to control television interactions using a virtual assistant and multiple user devices according to various examples is shown.

[0074] Fig.54 A functional block diagram of an electronic device configured to control television interaction using media content shown on a display and a media content viewing history is shown according to various examples.

[0075] Fig.55 A functional block diagram of an electronic device configured to suggest virtual assistant interactions for controlling media content according to various examples is shown.

[0076] Fig.56 An exemplary system for providing voice control of media playback and real-time updating of virtual assistant knowledge is shown.

[0077] Fig.57 A block diagram of an exemplary user device according to various examples is shown.

[0078] Fig.58 A block diagram of an exemplary media control device in a system for providing voice control of media playback is shown.

[0079] Fig.59 An exemplary process for voice control of media playback according to various examples is shown.

[0080] Fig.60 An exemplary data feed is shown that associates events in a media stream with specific times in the media stream.

[0081] Fig.61 An exemplary virtual assistant query response is shown that prompts video playback based on events in a media stream.

[0082] Fig.62 Exemplary events occurring before and after a playback position that may be used to interpret a user query are shown.

[0083] Fig.63 An exemplary awards ceremony data feed is shown that associates events in a media stream with specific times in the media stream.

[0084] Fig.64 An exemplary television program data feed is shown that associates events in a media stream with specific times in the media stream.

[0085] Fig.65 Exemplary closed caption text associated with a particular time in a video is shown, which may be used to respond to a user query.

[0086] Fig.66A A television display is shown with exemplary video content that may be used to interpret a user query.

[0087] Fig.66B A mobile user device is shown with exemplary images and textual content that may be used to interpret a user query.

[0088] Fig.67 An exemplary process for integrating information into a digital assistant's knowledge and responding to user requests is shown.

[0089] Fig.68 A functional block diagram of an electronic device configured to provide voice control of media playback and real-time updating of virtual assistant knowledge according to various examples is shown.

[0090] Fig.69 A functional block diagram of an electronic device configured to integrate information into digital assistant knowledge and respond to user requests according to various examples is shown. DETAILED DESCRIPTION

[0091] In the following description of the examples, reference will be made to the accompanying drawings, in which specific examples that may be implemented are shown by way of illustration. It should be understood that other examples may be used and structural changes may be made without departing from the scope of the various examples.

[0092] The present invention relates to a system and process for operating a digital assistant in a media environment. In an exemplary process, user input can be detected when displaying content. The process can determine whether the user input corresponds to a first input type. Based on determining that the user input corresponds to the first input type, multiple exemplary natural language requests can be displayed. The multiple exemplary natural language requests can be contextually related to the displayed content. It can be expected that the context-related exemplary natural language requests can conveniently inform the user of the capabilities of the digital assistant that are most relevant to the user's current usage conditions on the media device. This can encourage users to use digital assistant services and can also improve the user's interactive experience with the digital assistant.

[0093] In some embodiments, based on determining that the user input does not correspond to the first input type, the process may determine whether the user input corresponds to a second input type. Based on determining that the user input corresponds to the second input type, the audio data may be sampled. The process may determine whether the audio data includes a user request. Based on determining that the audio data includes a user request, a task that at least partially satisfies the user request may be performed.

[0094] In some embodiments, the task performed may depend on the nature of the user request and the content displayed when the user input of the second input type is detected. If the user request is a request to adjust the state or settings of an application on the electronic device (e.g., to turn on subtitles for displayed media content), the task may include adjusting the state or settings of the application. If the user request is a predetermined request type from a plurality of predetermined request types associated with plain text output (e.g., a request for the current time), the task may include displaying text that satisfies the user request. If the displayed content includes media content and the user request requires obtaining and displaying results, the process may determine whether the media content can be paused. If it is determined that the media content can be paused, the media content is paused and may be displayed in an expanded user interface (e.g., Figure 6H The result satisfying the user request may be displayed on a third user interface 626 shown in FIG. 1 . If it is determined that the media content cannot be paused, the result satisfying the user request may be displayed on a narrowed user interface (e.g., Figure 6G The second user interface 618 shown in . The display area of ​​the second user interface can be smaller than the display area of ​​the media content. In addition, if the displayed content does not include media content, the results that meet the user request can be displayed on the expanded user interface. By adjusting the output format based on the displayed content and the type of user request, the digital assistant can intelligently strike a balance by providing comprehensive assistance while minimizing interference with the user's consumption of media content. This can improve the user experience.

[0095] 1. System and Environment

[0096] Figure 1 An exemplary system 100 for operating a digital assistant according to various examples is shown. The terms "digital assistant," "virtual assistant," "intelligent automated assistant," or "automatic digital assistant" may refer to any information processing system for interpreting natural language input in spoken and / or textual form to infer user intent and perform actions based on the inferred user intent. For example, to take action based on the inferred user intent, the system may perform one or more of the following: identify a task flow using steps and parameters designed to implement the inferred user intent; input specific requirements from the inferred user intent into the task flow; execute the task flow by calling a program, method, service, application programming interface (API), etc.; and generate an output response to the user in auditory (e.g., voice) and / or visual form.

[0097] Specifically, the digital assistant may be able to accept user requests that are at least partially in the form of natural language commands, requests, statements, narrations, and / or inquiries. Typically, a user request may seek an informative answer from the digital assistant or seek a task from the digital assistant. A satisfactory response to a user request may be to provide the requested informative answer, perform the requested task, or a combination of the two. For example, a user may ask a digital assistant a question, such as "What time is it in Paris?" The digital assistant may retrieve the requested information and answer "It is 4:00 p.m. in Paris." The user may also request to perform a task, such as "Find me a movie starring Reese Witherspoon." In response, the digital assistant may perform the requested search query and display the relevant movie titles for the user to select from. During the execution of the requested task, the digital assistant may sometimes interact with the user over a long period of time in a continuous conversation involving multiple information exchanges. There are many other methods of interacting with a digital assistant to request information or perform various tasks. In addition to providing text responses and taking programmed actions, the digital assistant may also provide responses in other visual or audio forms, such as responses in the form of speech, alarms, music, images, videos, animations, etc. Furthermore, as discussed herein, an exemplary digital assistant can control playback of media content (e.g., on a television set-top box) and display media content or other information on a display unit (e.g., a television).

[0098] like Figure 1As shown in , in some examples, the digital assistant can be implemented according to a client-server model. The digital assistant may include a client-side portion 102 (hereinafter referred to as "DA client 102") executed on a media device 104, and a server-side portion 106 (hereinafter referred to as "DA server 106") executed on a server system 108. In addition, in some examples, the client-side portion may also be executed on a user device 122. The DA client 102 can communicate with the DA server 106 via one or more networks 110. The DA client 102 can provide client-side functionality, such as user-oriented input and output processing, and communication with the DA server 106. The DA server 106 can provide server-side functionality for any number of DA clients 102, each of which resides on a corresponding device (e.g., a media device 104 and a user device 122).

[0099] The media device 104 may be any suitable electronic device configured to manage and control media content. For example, the media device 104 may include a television set-top box, such as a cable box device, a satellite box device, a video player device, a video streaming device, a digital video recorder, a gaming system, a DVD player, a Blu-ray Disc TM players, combinations of such devices, etc. Figure 1 As shown, the media device 104 can be part of a media system 128. In addition to the media device 104, the media system 128 may include a remote control 124 and a display unit 126. The media device 104 can display media content on the display unit 126. The display unit 126 can be any type of display, such as a television display, a monitor, a projector, etc. In some examples, the media device 104 can be connected to an audio system (e.g., an audio receiver) and speakers (not shown) that can be integrated or separated from the display unit 126. In other examples, the display unit 126 and the media device 104 can be incorporated into a single device, such as a smart TV with advanced processing capabilities and network connectivity. In such examples, the functions of the media device 104 can be executed as an application on the combined device.

[0100] In some examples, media device 104 may be used as a media control center for multiple types and sources of media content. For example, media device 104 may facilitate user access to live television (e.g., over-the-air, satellite, or cable television). Therefore, media device 104 may include a cable tuner or a satellite tuner, etc. In some examples, media device 104 may also record television programs for later time-shifted viewing. In other examples, media device 104 may provide access to one or more streaming services, such as access to on-demand television programs, videos, and music delivered by cable, as well as television programs, videos, and music delivered by the Internet (e.g., from various free, paid, and subscription-based streaming services). In other examples, media device 104 may facilitate playback or display of media content from any other source, such as displaying photos from a mobile user device, playing videos from a coupled storage device, playing music from a coupled music player, etc. Media device 104 may also include various other combinations of the media control features discussed herein as needed. Reference below Figure 2 The media device 104 is described in detail.

[0101] The user device 122 may be any personal electronic device, such as a mobile phone (e.g., a smart phone), a tablet computer, a portable media player, a desktop computer, a laptop computer, a PDA, a wearable electronic device (e.g., digital glasses, a wristband, a watch, a brooch, an armband, etc.), etc. Figure 3 The user equipment 122 is described in detail.

[0102] In some examples, a user may interact with media device 104 through user device 122, remote control 124, or interface elements (e.g., buttons, microphones, cameras, joysticks, etc.) integrated with media device 104. For example, voice input including media-related queries or commands for a digital assistant may be received at user device 122 and / or remote control 124 and may be used to cause media-related tasks to be performed on media device 104. Likewise, tactile commands for controlling media on media device 104 may be received at user device 122 and / or remote control 124 (as well as other devices not shown). Thus, various functions of media device 104 may be controlled in a variety of ways, giving the user a variety of options for controlling media content from multiple devices.

[0103] Examples of one or more communication networks 110 may include local area networks (LANs) and wide area networks (WANs), such as the Internet. One or more communication networks 110 may be implemented using any known network protocol, including various wired or wireless protocols, such as, for example, Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0104] The DA server 106 may include a client-oriented input / output I / O interface 112, one or more processing modules 114, data and models 116, and an I / O interface 118 to external services. The client-oriented I / O interface 112 may facilitate client-oriented input and output processing of the DA server 106. One or more processing modules 114 may utilize the data and models 116 to process voice input and determine user intent based on natural language input. In addition, one or more processing modules 114 may perform tasks based on inferred user intent. In some examples, the DA server 106 may communicate with external services 120 (such as telephone services, calendar services, information services, messaging services, navigation services, television program playback services, streaming services, media search services, etc.) via one or more networks 110 to complete tasks or obtain information. The I / O interface 118 to external services may facilitate such communications.

[0105] The server system 108 may be implemented on one or more stand-alone data processing devices or distributed networks of computers. In some examples, the server system 108 may also employ various virtual devices and / or services of third-party service providers (e.g., third-party cloud service providers) to provide potential computing resources and / or infrastructure resources of the server system 108.

[0106] Although Figure 1 The digital assistant shown in the may include both a client-side portion (e.g., DA client 102) and a server-side portion (e.g., DA server 106), but in some examples, the functionality of the digital assistant may be implemented as a stand-alone application installed on a user device or a media device. In addition, the division of functionality between the client portion and the server portion of the digital assistant may vary in different specific implementations. For example, in some examples, a DA client executed on a user device 122 or a media device 104 may be a thin client that provides only user-facing input and output processing functions and delegates all other functions of the digital assistant to a backend server.

[0107] 2. Media System

[0108] Figure 2 1 shows a block diagram of a media system 128 according to various examples. The media system 128 may include a media device 104 communicatively coupled to a display unit 126, a remote control 124, and a speaker 268. The media device 104 may receive user input via the remote control 124. Media content from the media device 104 may be displayed on the display unit 126.

[0109] In this example, if Figure 2 As shown, the media device 104 may include a memory interface 202, one or more processors 204, and a peripheral device interface 206. The various components in the media device 104 may be coupled together via one or more communication buses or signal lines. The media device 104 may also include various subsystems and peripheral devices coupled to the peripheral device interface 206. The subsystems and peripheral devices may collect information and / or facilitate various functions of the media device 104.

[0110] For example, the media device 104 may include a communication subsystem 224. Communication functions may be facilitated by one or more wired and / or wireless communication subsystems 224, which may include various communication ports, radio frequency receivers and transmitters, and / or optical (e.g., infrared) receivers and transmitters.

[0111] In some examples, the media device 104 may also include an I / O subsystem 240 coupled to the peripheral device interface 206. The I / O subsystem 240 may include an audio / video output controller 270. The audio / video output controller 270 may be coupled to the display unit 126 and the speaker 268, or may be capable of providing audio and video output in other ways (e.g., via an audio / video port, wireless transmission, etc.). The I / O subsystem 240 may also include a remote controller 242. The remote controller 242 may be communicatively coupled to the remote control 124 (e.g., via a wired connection, Bluetooth, Wi-Fi, etc.).

[0112] The remote control 124 may include a microphone 272 for capturing audio data (e.g., voice input from a user), a button 274 for capturing tactile input, and a transceiver 276 for facilitating communication with the media device 104 via the remote controller 242. In addition, the remote control 124 may include a touch-sensitive surface 278, a sensor, or a sensor group that accepts input from a user based on tactile and / or tactile contact. The touch-sensitive surface 278 and the remote controller 242 may detect contact (and any movement or interruption of contact) on the touch-sensitive surface 278 and convert the detected contact (e.g., gestures, contact actions, etc.) into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the display unit 126. In some examples, the remote control 124 may also include other input mechanisms, such as a keyboard, a joystick, etc. In some examples, the remote control 124 may also include output mechanisms, such as a light, a display, a speaker, etc. Input received at the remote control 124 (e.g., user voice, button presses, contact motions, etc.) may be transmitted to the media device 104 via the remote control 124. The I / O subsystem 240 may also include one or more other input controllers 244. The one or more other input controllers 244 may be coupled to other input / control devices 248, such as one or more buttons, rocker switches, thumb wheels, infrared ports, USB ports, and / or pointer devices (such as a stylus).

[0113] In some examples, the media device 104 may also include a memory interface 202 coupled to a memory 250. The memory 250 may include any electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device; a portable computer disk (magnetic); a random access memory (RAM) (magnetic); a read-only memory (ROM) (magnetic); an erasable programmable read-only memory (EPROM) (magnetic); a portable optical disk such as a CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW; or a flash memory such as a compact flash card, a secure digital card, a USB storage device, a memory stick, etc. In some examples, the non-transitory computer-readable storage medium of the memory 250 may be used to store instructions (e.g., for performing part or all of the various processes described herein) for use by or in conjunction with an instruction execution system, apparatus, or device such as a computer-based system, a system containing a processor, or other system that can retrieve instructions from an instruction execution system, apparatus, or device and can execute the instructions. In other examples, instructions (e.g., for performing part or all of the various processes described herein) may be stored on a non-transitory computer-readable storage medium of server system 108, or may be divided between a non-transitory computer-readable storage medium of memory 250 and a non-transitory computer-readable storage medium of server system 108. In the context of this document, a "non-transitory computer-readable storage medium" may be any medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0114] In some examples, the memory 250 may store an operating system 252, a communication module 254, a graphical user interface (GUI) module 256, a device built-in media module 258, a device external media module 260, and an application module 262. The operating system 252 may include instructions for processing basic system services and for performing hardware-related tasks. The communication module 254 may facilitate communication with one or more additional devices, one or more computers and / or one or more servers. The graphical user interface module 256 may facilitate graphical user interface processing. The device built-in media module 258 may facilitate the storage and playback of media content stored locally on the media device 104. The device external media module 260 may facilitate streaming playback or downloading of media content obtained from an external source (e.g., on a remote server, on a user device 122, etc.). In addition, the device external media module 260 may facilitate receiving broadcast and cable content (e.g., channel tuning). The application module 262 may facilitate various functions of media-related applications, such as web browsing, media processing, games and / or other processes and functions.

[0115] As described herein, the memory 250 may also store client-side digital assistant instructions (e.g., in a digital assistant client module 264) and various user data 266 (e.g., user-specific vocabulary data, preference data, and / or other data such as a user's media search history, media viewing list, recently viewed list, favorite media items, etc.), thereby, for example, providing client-side functionality of the digital assistant. The user data 266 may also be used to perform speech recognition to support the digital assistant or for any other application.

[0116] In various examples, the digital assistant client module 264 may be capable of accepting voice input (e.g., speech input), text input, touch input, and / or gesture input through various user interfaces of the media device 104 (e.g., I / O subsystem 240, etc.). The digital assistant client module 264 may also be capable of providing output in the form of audio (e.g., speech output), visual, and / or tactile. For example, the output may be provided as speech, sound, alarm, text message, menu, graphic, video, animation, vibration, and / or a combination of two or more of the above. During operation, the digital assistant client module 264 may communicate with a digital assistant server (e.g., DA server 106) using the communication subsystem 224.

[0117] In some examples, the digital assistant client module 264 can utilize various subsystems and peripherals to collect additional information related to the media device 104 from the surrounding environment of the media device 104 to establish a context associated with the user, the current user interaction, and / or the current user input. Such context may also include information from other devices, such as information from the user device 122. In some examples, the digital assistant client module 264 may provide context information or a subset thereof to the digital assistant server along with the user input to help infer the user's intent. The digital assistant may also use the context information to determine how to prepare the output and deliver the output to the user. The context information may also be used by the media device 104 or the server system 108 to support accurate speech recognition.

[0118] In some examples, contextual information accompanying user input may include sensor information such as lighting, ambient noise, ambient temperature, distance to another object, etc. Contextual information may also include information associated with the physical state of the media device 104 (e.g., device location, device temperature, power level, etc.) or the software state of the media device 104 (e.g., running processes, installed applications, past and current network activity, background services, error logs, resource usage, etc.). Contextual information may also include information received from the user (e.g., voice input), information requested by the user, and information presented to the user (e.g., information currently or previously displayed by the media device). Contextual information may also include information associated with the state of a connected device or other device associated with the user (e.g., content displayed on the user device 122, playable content on the user device 122, etc.). Any of these types of contextual information may be provided to the DA server 106 (or for the media device 104 itself) as contextual information related to the user input.

[0119] In some examples, the digital assistant client module 264 may selectively provide information stored on the media device 104 (e.g., user data 266) in response to a request from the DA server 106. Additionally or alternatively, the information may be used to perform speech recognition and / or digital assistant functions on the media device 104 itself. The digital assistant client module 264 may also elicit additional input from the user via a natural language dialog or other user interface when requested by the DA server 106. The digital assistant client module 264 may communicate the additional input to the DA server 106 to assist the DA server 106 in inferring intent and / or satisfying the user intent expressed in the user request.

[0120] In various examples, memory 250 may include additional instructions or fewer instructions. Furthermore, various functions of media device 104 may be implemented in hardware and / or firmware, including in one or more signal processing circuits and / or application specific integrated circuits.

[0121] 3. User equipment

[0122] Figure 3 A block diagram of an exemplary user device 122 according to various examples is shown. As shown, the user device 122 may include a memory interface 302, one or more processors 304, and a peripheral device interface 306. Various components in the user device 122 may be coupled together via one or more communication buses or signal lines. The user device 122 may also include various sensors, subsystems, and peripherals coupled to the peripheral device interface 306. The sensors, subsystems, and peripherals may collect information and / or facilitate various functions of the user device 122.

[0123] For example, user device 122 may include motion sensor 310, light sensor 312, and proximity sensor 314, which are coupled to peripheral device interface 306 to facilitate orientation, lighting, and proximity sensing functions. One or more other sensors 316 (such as a positioning system (e.g., a GPS receiver), a temperature sensor, a biometric sensor, a gyroscope, a compass, an accelerometer, etc.) may also be connected to peripheral device interface 306 to facilitate related functions.

[0124] In some examples, camera subsystem 320 and optical sensor 322 may be used to facilitate camera functions, such as taking pictures and recording video clips. Communication functions may be facilitated by one or more wired and / or wireless communication subsystems 324, which may include various communication ports, radio frequency receivers and transmitters, and / or optical (e.g., infrared) receivers and transmitters. Audio subsystem 326 may be coupled to speaker 328 and microphone 330 to facilitate voice-enabled functions, such as voice recognition, voice replication, digital recording, and telephone functions.

[0125] In some examples, the user device 122 may also include an I / O subsystem 340 coupled to the peripheral device interface 306. The I / O subsystem 340 may include a touch screen controller 342 and / or one or more other input controllers 344. The touch screen controller 342 may be coupled to a touch screen 346. The touch screen 346 and the touch screen controller 342 may, for example, use any of a variety of touch sensitivity technologies (such as capacitive technology, resistive technology, infrared technology, surface acoustic wave technology, proximity sensor array, etc.) to detect contact and movement or interruption thereof. One or more other input controllers 344 may be coupled to other input / control devices 348, such as one or more buttons, rocker switches, thumb wheels, infrared ports, USB ports, and / or pointer devices (such as stylus).

[0126] In some examples, user device 122 may also include a memory interface 302 coupled to memory 350. Memory 350 may include any electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device; a portable computer disk (magnetic); a random access memory (RAM) (magnetic); a read-only memory (ROM) (magnetic); an erasable programmable read-only memory (EPROM) (magnetic); a portable optical disk such as a CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW; or a flash memory such as a compact flash card, a secure digital card, a USB memory device, a memory stick, etc. In some examples, the non-transitory computer-readable storage medium of memory 350 may be used to store instructions (e.g., for performing part or all of the various processes described herein) for use by or in conjunction with an instruction execution system, apparatus, or device such as a computer-based system, a system containing a processor, or other system that can retrieve instructions from an instruction execution system, apparatus, or device and can execute the instructions. In other examples, instructions (e.g., for performing part or all of the various processes described herein) may be stored on a non-transitory computer-readable storage medium of server system 108, or may be divided between a non-transitory computer-readable storage medium of memory 350 and a non-transitory computer-readable storage medium of server system 108. In the context of this document, a "non-transitory computer-readable storage medium" may be any medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0127] In some examples, memory 350 may store operating system 352, communication module 354, graphical user interface (GUI) module 356, sensor processing module 358, phone module 360, and application module 362. Operating system 352 may include instructions for processing basic system services and for performing hardware-related tasks. Communication module 354 may facilitate communication with one or more additional devices, one or more computers, and / or one or more servers. Graphical user interface module 356 may facilitate graphical user interface processing. Sensor processing module 358 may facilitate sensor-related processing and functions. Phone module 360 ​​may facilitate phone-related processes and functions. Application module 362 may facilitate various functions of user applications, such as electronic messaging, web browsing, media processing, navigation, imaging, and / or other processes and functions.

[0128] As described herein, the memory 350 may also store client-side digital assistant instructions (e.g., stored in a digital assistant client module 364) and various user data 366 (e.g., vocabulary data, preference data, and / or other data specific to a user such as a user's electronic address book, to-do list, shopping list, television program favorites, etc.), thereby, for example, providing client-side functionality of the digital assistant. The user data 366 may also be used to perform speech recognition to support the digital assistant or for any other application. The digital assistant client module 364 and the user data 366 may be similar or identical to the digital assistant client module 264 and the user data 266, respectively, as described above with reference to FIG. Figure 2 described.

[0129] In various examples, memory 350 may include additional instructions or fewer instructions. Furthermore, various functions of user device 122 may be performed in hardware and / or firmware, including in one or more signal processing and / or application specific integrated circuits.

[0130] In some examples, user device 122 may be configured to control various aspects of media device 104. For example, user device 122 may be used as a remote control (e.g., remote control 124). User input received via user device 122 may be transmitted to media device 104 (e.g., using a communication subsystem) to cause media device 104 to perform a corresponding action. In addition, user device 122 may be configured to receive instructions from media device 104. For example, media device 104 may hand over a task to user device 122 to perform and cause an object (e.g., a selectable affordance) to be displayed on user device 122.

[0131] It should be understood that the system 100 and the media system 128 are not limited to Figure 1 and Figure 2 The components and configurations shown, and the user device 122, media device 104 and remote control 124 are also not limited to Figure 2 and Figure 3 In various configurations according to various examples, system 100, media system 128, user device 122, media device 104, and remote control 124 may all include fewer components, or include other components.

[0132] 4. Digital Assistant System

[0133] Figure 4AA block diagram of a digital assistant system 400 according to various examples is shown. In some examples, the digital assistant system 400 may be implemented on a stand-alone computer system. In some examples, the digital assistant system 400 may be distributed across multiple computers. In some examples, some modules and functions of the digital assistant may be divided into a server portion and a client portion, where the client portion resides on one or more user devices (e.g., device 104 or device 122) and communicates with the server portion (e.g., server system 108) via one or more networks, such as Figure 1 In some examples, the digital assistant system 400 may be Figure 1 Specific implementation of the server system 108 (and / or DA server 106) shown in . It should be noted that the digital assistant system 400 is only one example of a digital assistant system, and the digital assistant system 400 may have more or fewer components than shown, may combine two or more components, or may have a different configuration or layout of components. Figure 4A The various components shown in the drawings may be implemented in hardware, software instructions for execution by one or more processors, firmware (including one or more signal processing integrated circuits and / or application specific integrated circuits), or a combination thereof.

[0134] The digital assistant system 400 may include a memory 402, one or more processors 404, an I / O interface 406, and a network communication interface 408. These components may communicate with each other via one or more communication buses or signal lines 410.

[0135] In some examples, memory 402 may include non-transitory computer-readable media such as high-speed random access memory and / or non-volatile computer-readable storage media (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0136] In some examples, I / O interface 406 may couple I / O devices 416 of digital assistant system 400, such as a display, keyboard, touch screen, and microphone, to user interface module 422. I / O interface 406 in conjunction with user interface module 422 may receive user input (e.g., voice input, keyboard input, touch input, etc.) and process the input accordingly. In some examples, such as when the digital assistant is implemented on a stand-alone user device, digital assistant system 400 may include a plurality of I / O devices 416 and a plurality of I / O devices 416 of digital assistant system 400. Figure 2 or Figure 3 Any of the components and I / O communication interfaces described in the respective device 104 or device 122. In some examples, the digital assistant system 400 may represent a server portion of a specific implementation of a digital assistant and may interact with a user through a client-side portion residing on a client device (e.g., device 104 or device 122).

[0137] In some examples, the network communication interface 408 may include one or more wired communication ports 412, and / or wireless transmission and reception circuits 414. One or more wired communication ports may receive and send communication signals via one or more wired interfaces such as Ethernet, Universal Serial Bus (USB), FireWire, etc. The wireless circuit 414 may receive RF signals and / or optical signals from a communication network and other communication devices, and send the RF signals and / or optical signals to the communication network and other communication devices. Wireless communication may use any of a variety of communication standards, protocols and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol. The network communication interface 408 may enable the digital assistant system 400 to communicate with other devices through a network such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN).

[0138] In some examples, memory 402 or a computer-readable storage medium of memory 402 may store programs, modules, instructions, and data structures, including all or a subset of the following: operating system 418, communication module 420, user interface module 422, one or more applications 424, and digital assistant module 426. Specifically, memory 402 or a computer-readable storage medium of memory 402 may store instructions for performing process 800 described below. One or more processors 404 may execute these programs, modules, and instructions, and may read data from or write data to data structures.

[0139] The operating system 418 (e.g., Darwin, RTXC, LINUX, UNIX, iOS, OS X, WINDOWS, or an embedded operating system such as VxWorks) may include various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitating communication between various hardware, firmware, and software components.

[0140] The communication module 420 can facilitate communication between the digital assistant system 400 and other devices through the network communication interface 408. For example, the communication module 420 can communicate with the communication subsystem (e.g., 224, 324) of the electronic device (e.g., 104, 122). The communication module 420 can also include various components for processing data received by the wireless circuit 414 and / or the wired communication port 412.

[0141] The user interface module 422 may receive commands and / or input from a user (e.g., from a keyboard, touch screen, pointing device, controller, and / or microphone) via the I / O interface 406 and generate user interface objects on the display. The user interface module 422 may also prepare output (e.g., speech, sound, animation, text, icon, vibration, tactile feedback, lighting, etc.) and deliver it to the user via the I / O interface 406 (e.g., through a display, audio channel, speaker, touchpad, etc.).

[0142] Applications 424 may include programs and / or modules configured to be executed by one or more processors 404. For example, if the digital assistant system 400 is implemented on a stand-alone user device, applications 424 may include user applications such as games, calendar applications, navigation applications, or email applications. If the digital assistant system 400 is implemented on a server, applications 424 may include, for example, resource management applications, diagnostic applications, or scheduling applications.

[0143] The memory 402 may also store a digital assistant module 426 (or the server portion of the digital assistant). In some examples, the digital assistant module 426 may include the following submodules, or a subset or superset thereof: an I / O processing module 428, a speech-to-text (STT) processing module 430, a natural language processing module 432, a dialog flow processing module 434, a task flow processing module 436, a service processing module 438, and a speech synthesis module 440. Each of these modules may have access to one or more of the following systems or data and models of the digital assistant module 426, or a subset or superset thereof: a knowledge ontology 460, a vocabulary index 444, user data 448, a task flow model 454, a service model 456, and an automatic speech recognition (ASR) system 431.

[0144] In some examples, using the processing modules, data, and models implemented in the digital assistant module 426, the digital assistant can perform at least some of the following operations: convert voice input into text; recognize user intent expressed in natural language input received from a user; proactively elicit and obtain information needed to fully infer user intent (e.g., by disambiguating words, games, intentions, etc.); determine a task flow for satisfying the inferred intent; and execute the task flow to satisfy the inferred intent.

[0145] In some examples, such as Figure 4B As shown in FIG. , the I / O processing module 428 can be Figure 4A The I / O device 416 in the Figure 4AThe network communication interface 408 in the device 104 or the device 122 interacts with the electronic device (e.g., device 104 or device 122) to obtain user input (e.g., voice input) and provide a response to the user input (e.g., as a voice output). The I / O processing module 428 can optionally obtain context information associated with the user input from the electronic device when receiving the user input or shortly after receiving the user input. The context information may include user-specific data, vocabulary and / or preferences related to the user input. In some examples, the context information also includes the software and hardware state of the electronic device when the user request is received, and / or information related to the user's surrounding environment when the user request is received. In some examples, the I / O processing module 428 may also send follow-up questions related to the user request to the user and receive answers from the user. When the user request is received by the I / O processing module 428 and the user request may include voice input, the I / O processing module 428 may forward the voice input to the STT processing module 430 (or speech recognizer) to perform speech-to-text conversion.

[0146] The STT processing module 430 may include one or more ASR systems (e.g., ASR system 431). One or more ASR systems may process the speech input received by the I / O processing module 428 to produce a recognition result. Each ASR system may include a front-end speech preprocessor. The front-end speech preprocessor may extract representative features from the speech input. For example, the front-end speech preprocessor may perform Fourier transform on the speech input to extract the spectral features characterizing the speech input as a sequence of representative multidimensional vectors. In addition, each ASR system may include one or more speech recognition models (e.g., sound models and / or language models), and may implement one or more speech recognition engines. Examples of speech recognition models may include hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models. Examples of speech recognition engines may include engines based on dynamic time warping and engines based on weighted finite state transformers (WFST). One or more speech recognition models and one or more speech recognition engines may be used to process the extracted representative features of the front-end speech preprocessor to generate intermediate recognition results (e.g., phonemes, phoneme strings, and sub-words), and ultimately generate text recognition results (e.g., words, word strings, or sequences of symbols). In some examples, the speech input may be at least partially processed by a third-party service or processed on an electronic device (e.g., device 104 device 122) to generate a recognition result. Once the STT processing module 430 generates a recognition result containing a text string (e.g., a word, a sequence of words, or a sequence of symbols), the recognition result may be transmitted to the natural language processing module 432 for intent inference.

[0147] In some examples, one or more language models of one or more ASR systems may be configured to favor media-related results. In one example, one or more language models may be trained using a corpus of media-related text. In another example, the ASR system may be configured to favor media-related recognition results. In some examples, one or more ASR systems may include a static language model and a dynamic language model. The static language model may be trained using a general corpus of text, while the dynamic language model may be trained using user-specific text. For example, a dynamic language model may be generated using text with previous voice input received from a user. In some examples, one or more ASR systems may be configured to generate recognition results based on a static language model and / or a dynamic language model. In addition, in some examples, one or more ASR systems may be configured to favor recognition results corresponding to a previous voice input recently received.

[0148] More details about speech-to-text processing are described in U.S. Utility Patent Application Serial No. 13 / 236,942, filed on September 20, 2011, entitled “Consolidating Speech Recognition Results,” the entire disclosure of which is incorporated herein by reference.

[0149] In some examples, the STT processing module 430 may include a vocabulary of recognizable words and / or may access the vocabulary via the phonetic letter conversion module 431. Each vocabulary word may be associated with one or more candidate pronunciations of the word represented in the phonetic letter. Specifically, the vocabulary of recognizable words may include words associated with multiple candidate pronunciations. For example, the vocabulary may include words associated with and In some examples, the candidate pronunciations for a word may be determined based on the spelling of the word and one or more linguistic and / or phonetic rules. In some examples, the candidate pronunciations may be manually generated, for example, based on a known standard pronunciation.

[0150] In some examples, the candidate pronunciations may be ranked based on their prevalence. comparable The candidate pronunciation may be ranked higher because the former is a more common pronunciation (e.g., among all users, for users in a particular geographic region, or for any other suitable subgroup of users). In some examples, the candidate pronunciations may be ranked based on whether the candidate pronunciation is a custom candidate pronunciation associated with the user. For example, a custom candidate pronunciation may be ranked higher than a standard candidate pronunciation. This may be useful for identifying proper nouns that have unique pronunciations that deviate from the standard pronunciation. In some examples, the candidate pronunciation may be associated with one or more speech features, such as geographic origin, country, or ethnicity. For example, the candidate pronunciation may be associated with a person's ethnicity, ethnicity, or geographic origin. can be associated with the United States, while the candidate pronunciation The candidate pronunciations may be associated with the United Kingdom. In addition, the ranking of the candidate pronunciations may be based on one or more characteristics of the user (e.g., geographic origin, country, ethnicity, etc.) in a user profile stored on the device. For example, it may be determined from the user profile that the user is associated with the United States. Based on the user being associated with the United States, the candidate pronunciations may be ranked (associated with the United States) Comparable candidate pronunciations (associated with the United Kingdom) is ranked high. In some examples, one of the ranked candidate pronunciations may be selected as the predicted pronunciation (eg, the most likely pronunciation).

[0151] Upon receiving speech input, the STT processing module 430 may be used to determine (e.g., using an acoustic model) a phoneme corresponding to the speech input, and may then attempt to determine (e.g., using a language model) a word that matches the phoneme. For example, if the STT processing module 430 may first identify a phoneme sequence corresponding to a portion of the speech input, It can then determine based on the vocabulary index 444 that the sequence corresponds to the word "tomato".

[0152] In some examples, the STT processing module 430 may use fuzzy matching techniques to determine the words in the utterance. Thus, for example, the STT processing module 430 may determine the phoneme sequence corresponds to the word "tomato", even though this particular phoneme sequence is not a candidate phoneme sequence for that word.

[0153] The natural language processing module 432 ("natural language processor") of the digital assistant may take a sequence of words or symbols ("symbol sequence") generated by the STT processing module 430 and attempt to associate the symbol sequence with one or more "executable intents" recognized by the digital assistant. An "executable intent" may represent a task that can be executed by the digital assistant and may have an associated task flow implemented in the task flow model 454. The associated task flow may be a series of programmed actions and steps taken by the digital assistant to perform the task. The scope of the digital assistant's capabilities may depend on the number and variety of task flows that have been implemented and stored in the task flow model 454, or in other words, on the number and variety of "executable intents" recognized by the digital assistant. However, the effectiveness of the digital assistant may also depend on the assistant's ability to infer the correct "one or more executable intents" from a user request expressed in natural language.

[0154] In some examples, in addition to the sequence of words or symbols obtained from the STT processing module 430, the natural language processor 432 may also receive context information associated with the user request (e.g., from the I / O processing module 428). The natural language processing module 432 may optionally use the context information to clarify, supplement, and / or further qualify the information contained in the sequence of symbols received from the STT processing module 430. The context information may include, for example: user preferences; hardware and / or software state of the user's device; sensor information collected before, during, or shortly after the user's request; previous interactions (e.g., conversations) between the digital assistant and the user, etc. As described herein, the context information may be dynamic and may change with the time, location, content, and other factors of the conversation.

[0155] In some examples, natural language processing may be based on, for example, knowledge ontology 460. Knowledge ontology 460 may be a hierarchical structure comprising many nodes, each node representing an "executable intent" or an "attribute" related to one or more of an "executable intent" or other "attributes". As described above, an "executable intent" may represent a task that a digital assistant can perform, i.e., the task is "executable" or can be performed. An "attribute" may represent a parameter associated with a sub-aspect of an executable intent or another attribute. The connection between an executable intent node and an attribute node in knowledge ontology 460 may define how the parameter represented by the attribute node is related to the task represented by the executable intent node.

[0156] In some examples, ontology 460 may be composed of executable intent nodes and attribute nodes. Within ontology 460, each executable intent node may be directly connected to or connected to one or more attribute nodes through one or more intermediate attribute nodes. Similarly, each attribute node may be directly connected to or connected to one or more executable intent nodes through one or more intermediate attribute nodes. For example, Figure 4C As shown, ontology 460 may include a "media" node (i.e., an executable intent node). The attribute nodes "one or more actors", "media classification", and "media title" may each be directly connected to the executable intent node (i.e., the "media search" node). In addition, the attribute nodes "name", "age", "Ulmer scale ranking", and "nationality" may be child nodes of the attribute node "actor".

[0157] In another example, Figure 4C As shown, ontology 460 may also include a "weather" node (i.e., another executable intent node). The attribute nodes "date / time" and "location" may each be connected to the "weather search" node. It should be appreciated that in some examples, one or more attribute nodes may be associated with two or more executable intents. In these examples, the one or more attribute nodes may be connected to corresponding nodes corresponding to two or more executable intents in ontology 460.

[0158] The executable intent nodes, together with the concept nodes they connect, can be described as "domains". In this discussion, each domain can be associated with a corresponding executable intent and can refer to a set of nodes (and their relationships to each other) associated with a particular executable intent. For example, Figure 4C The knowledge ontology 460 shown in may include an example of a media domain 462 and an example of a weather domain 464 within the knowledge ontology 460. The media domain 462 may include an executable intent node "media search" and attribute nodes "one or more actors", "media classification", and "media title". The weather domain 464 may include an executable intent node "weather search", and attribute nodes "location" and "date / time". In some examples, the knowledge ontology 460 may be composed of multiple domains. Each domain may share one or more attribute nodes with one or more other domains.

[0159] Although Figure 4C Two exemplary domains within ontology 460 are shown, but other domains may include, for example, "athlete," "stock market," "direction," "media settings," "sports team," "time," and "tell jokes," etc. The domain "athlete" may be associated with an executable intent node "search for athlete information," and may also include attribute nodes such as "athlete name," "athlete's team," and "athlete statistics."

[0160] In some examples, ontology 460 may include all domains (and therefore executable intents) that a digital assistant can understand and act upon. In some examples, ontology 460 may be modified, such as by adding or removing entire domains or nodes or by modifying the relationships between nodes within ontology 460.

[0161] In some examples, each node in the knowledge ontology 460 can be associated with a set of words and / or phrases related to the attribute or executable intent represented by the node. The corresponding set of words and / or phrases associated with each node can be a so-called "vocabulary" associated with the node. The corresponding set of words and / or phrases associated with each node can be stored in a vocabulary index 444, which is associated with the attribute or executable intent represented by the node. For example, returning Figure 4C , the vocabulary associated with the node for the attribute of "actors" may include words such as "A list", "Reese Witherspoon", "Arnold Schwarzenegger", "Brad Pitt", etc. In another example, the vocabulary associated with the node for the executable intent of "weather search" may include words and phrases such as "weather", "what's the weather like", "forecast", etc. The vocabulary index 444 may optionally include words and phrases in different languages.

[0162] The natural language processing module 432 may receive a sequence of symbols (e.g., a text string) from the STT processing module 430 and determine which nodes are involved by the words in the sequence of symbols. In some examples, if a word or phrase in the sequence of symbols is found to be associated with one or more nodes in the knowledge ontology 460 (via the vocabulary index 444), the word or phrase may "trigger" or "activate" these nodes. Based on the number and / or relative importance of the activated nodes, the natural language processing module 432 may select one of the executable intents as the task that the user intends the digital assistant to perform. In some examples, the domain with the most "triggered" nodes may be selected. In some examples, the domain with the highest confidence may be selected (e.g., based on the relative importance of its respective triggered nodes). In some examples, the domain may be selected based on a combination of the number and importance of the triggered nodes. In some examples, additional factors are also considered in the process of selecting nodes, such as whether the digital assistant has previously correctly interpreted similar requests from the user.

[0163] User data 448 may include user-specific information, such as user-specific vocabulary, user preferences, user address, the user's default language and second language, the user's contact list, and other short-term or long-term information for each user. In some examples, natural language processing module 432 may use user-specific information to supplement information contained in the user input to further qualify the user's intent. For example, for a user request, "What's the weather like this week?", natural language processing module 432 may access user data 448 to determine where the user is located, rather than requiring the user to explicitly provide such information in their request.

[0164] Additional details of searching knowledge ontology based on symbolic strings are described in U.S. utility patent application serial number 12 / 341,743, entitled “Method and Apparatus for Searching Using An Active Ontology,” filed on December 22, 2008, the entire disclosure of which is incorporated herein by reference.

[0165] In some examples, once the natural language processing module 432 identifies an executable intent (or domain) based on a user request, the natural language processing module 432 may generate a structured query to represent the identified executable intent. In some examples, the structured query may include parameters for one or more nodes of the executable intent within the domain, and at least some of the parameters are filled with specific information and requirements specified in the user request. For example, a user may say "Find me the other seasons of this TV series." In this case, the natural language processing module 432 may correctly identify the executable intent as "media search" based on the user input. According to the knowledge ontology, a structured query for the "media" domain may include parameters such as {media actor}, {media classification}, {media title}, etc. In some examples, based on voice input and text derived from the voice input using the STT processing module 430, the natural language processing module 432 may generate a partial structured query for a restaurant reservation domain, wherein the partial structured query includes the parameter {media classification = "TV series"}. However, in this example, the user utterance contains insufficient information to complete the structured query associated with the domain. Therefore, based on the currently available information, other necessary parameters, such as {media title}, may not be specified in the structured query. In some examples, the natural language processing module 432 can use the received context information to fill in some parameters of the structured query. For example, the television series "Mad Men" is currently playing on the media device. Based on this context information, the natural language processing module 432 can use "Mad Men" to fill in the {media title} parameter in the structured query.

[0166] In some examples, the natural language processing module 432 may transmit the structured query (including any completed parameters) to the task flow processing module 436 ("task flow processor"). The task flow processing module 436 may be configured to receive the structured query from the natural language processing module 432, complete the structured query if necessary, and perform the actions required to "complete" the user's final request. In some examples, the various processes necessary to complete these tasks may be provided in the task flow model 454. In some examples, the task flow model 454 may include a process for obtaining additional information from the user, and a task flow for performing actions associated with an executable intent.

[0167] As described above, in order to complete the structured query, the task flow processing module 436 may need to initiate additional dialogues with the user in order to obtain additional information and / or disambiguate potentially ambiguous utterances. When it is necessary to conduct such interactions, the task flow processing module 436 may call the dialogue flow processing module 434 to participate in the dialogue with the user. In some examples, the dialogue flow processing module 434 may determine how (and / or when) to request additional information from the user, and may receive and process user responses. Questions may be provided to the user and answers may be received from the user through the I / O processing module 428. In some examples, the dialogue flow processing module 434 may present the dialogue output to the user via audio and / or video output, and may receive input from the user via verbal or physical (e.g., click) responses. For example, a user may ask "What is the weather in Paris?" When the task flow processing module 436 calls the dialogue flow processing module 434 to determine the "location" information of the structured query associated with the domain "weather search", the dialogue flow processing module 434 may generate questions such as "Which Paris?" and transmit them to the user. In addition, dialog flow processing module 434 may cause affordances associated with "Paris, Texas" and "Paris, France" to be presented for selection by the user. Upon receiving a response from the user, dialog flow processing module 434 may fill in the structured query with the missing information, or pass the information to task flow processing module 436 to complete the missing information from the structured query.

[0168] Once the task flow processing module 436 has completed the structured query for the executable intent, the task flow processing module 436 can begin to execute the final task associated with the executable intent. Therefore, the task flow processing module 436 can execute the steps and instructions in the task flow model 454 according to the specific parameters contained in the structured query. For example, the task flow model of the executable intent of "media search" may include steps and instructions for executing a media search query to obtain relevant media items. For example, by using a structured query such as: {media search, media classification = TV series, media title = Mad Men}, the task flow processing module 436 can perform the following steps: (1) using a media database to execute a media search query to obtain relevant media items; (2) ranking the obtained media items according to relevance and / or popularity, and (3) displaying the classified media items according to relevance and / or popularity.

[0169] In some examples, the task flow processing module 436 can complete the task requested in the user input or provide the informational answer requested in the user input with the help of the service processing module 438 ("service processing module"). For example, the service processing module 438 can perform media searches, retrieve weather information, call or interact with applications installed on other user devices, and call or interact with third-party services (e.g., social networking sites, media review sites, media subscription services, etc.) on behalf of the task flow processing module 436. In some examples, the protocols and APIs required for each service can be specified by the corresponding service model in the service model 456. The service processing module 438 can access the appropriate service model for the service and generate a request for the service according to the protocol and API required by the service according to the service model.

[0170] For example, a third-party media search service may submit a service model for specifying necessary parameters for performing a media search, and an API for transmitting the values ​​of the necessary parameters to the media search service. When the task flow processing module 436 issues a request, the service processing module 438 may establish a network connection with the media search service and send the necessary parameters of the media search (e.g., media actors, media types, media titles) to the online booking interface in a format according to the API of the media search service.

[0171] In some examples, the natural language processing module 432, the dialog flow processing module 434, and the task flow processing module 436 can be used together and repeatedly to infer and define the user's intention, obtain information to further clarify and refine the user's intention, and finally generate a response (i.e., output to the user or complete the task) to meet the user's intention. The generated response can be a dialogue response to the voice input that at least partially meets the user's intention. In addition, in some examples, the generated response can be output as a voice output. In these examples, the generated response can be sent to a speech synthesis module 440 (e.g., a speech synthesizer), where the response can be processed to synthesize the dialogue response into a voice form. In other examples, the generated response can be data content related to satisfying the user's request in the voice input.

[0172] The speech synthesis module 440 may be configured to synthesize speech output for presentation to the user. The speech synthesis module 440 synthesizes speech output based on the text provided by the digital assistant. For example, the generated dialogue response may be in the form of a text string. The speech synthesis module 440 may convert the text string into an audible speech output. The speech synthesis module 440 may use any appropriate speech synthesis technology to generate speech output from the text, and any appropriate speech synthesis technology includes but is not limited to: concatenation synthesis, unit selection synthesis, diphone synthesis, domain-specific synthesis, format synthesis, pronunciation synthesis, synthesis based on a hidden Markov model (HMM), and sine wave synthesis. In some examples, the speech synthesis module 440 may be configured to synthesize individual words based on a phoneme string corresponding to the word. For example, a phoneme string may be associated with a word in the generated dialogue response. The phoneme string may be stored in metadata associated with the word. The speech synthesis module 440 may be configured to directly process the phoneme string in the metadata to synthesize the word in speech form.

[0173] In some examples, instead of using speech synthesis module 440 (or in addition), speech synthesis can be performed on a remote device (e.g., server system 108), and the synthesized speech can be sent to the user device for output to the user. For example, this can occur in some implementations where the output of the digital assistant is generated at the server system. And because the server system typically has more processing power or more resources than the user device, it is possible to obtain higher quality speech output than client-side synthesis would achieve.

[0174] More details about the digital assistant can be found in U.S. Utility Patent Application No. 12 / 987,982, filed on January 10, 2011, and entitled “Intelligent Automated Assistant,” and U.S. Utility Patent Application No. 13 / 251,088, filed on September 30, 2011, and entitled “Generating and Processing Task Items That Represent Tasks to Perform,” the entire disclosures of which are incorporated herein by reference.

[0175] 4. Processes for interacting with digital assistants in a media environment

[0176] FIG. 5A to FIG. 5I A process 500 for operating a digital assistant for a media system according to various examples is shown. The process 500 can be performed using one or more electronic devices implementing a digital assistant. For example, the process 500 can be performed using one or more of the system 100, media system 128, media device 104, user device 122, or digital assistant system 400 described above. FIG. 6A to FIG. 6Q 5 shows screenshots displayed by a media device on a display unit at various stages of process 500 according to various examples. FIG. 5A to FIG. 5I and FIG. 6A to FIG. 6Q 5. It should be understood that some operations in process 500 may be combined, the order of some operations may be changed, and some operations may be omitted.

[0177] At block 502 of process 500, content may be displayed on a display unit (e.g., display unit 126). Fig. 6A In the example shown, the displayed content may include media content 602 (e.g., movies, videos, television programs, video games, etc.) played on a media device (e.g., media device 104). In other examples, the displayed content may include other content associated with the media device, such as content associated with an application running on the media device, or a user interface for interacting with a digital assistant of the media device. Specifically, the displayed content may include a main menu user interface or a user interface containing an object or result previously requested by the user (e.g., a second user interface 618 or a third user interface 626).

[0178] At block 504 of process 500, user input may be detected. The user input may be detected while displaying the contents of block 502. In some examples, the user input may be detected on a remote control of the media device (e.g., remote control 124). Specifically, the user input may be an interaction of the user with the remote control, such as pressing a button (e.g., button 274) or contacting a touch-sensitive surface of the remote control (e.g., touch-sensitive surface 278). In some examples, the user input may be detected via a second electronic device (e.g., device 122) configured to interact with the media device. In response to detecting the user input, one or more of blocks 506 to 592 may be executed.

[0179] At block 506 of process 500, it may be determined whether the user input corresponds to a first input type. The first input type may be a predefined input to the media device. In one example, the first input type may include pressing a specific button of a remote control and releasing the button within a predetermined duration of pressing the button (e.g., a short press). The media device may determine whether the user input matches the first input type. Based on determining that the user input corresponds to the first input type, one or more of blocks 508 to 514 may be executed.

[0180] At block 508 of process 500, and with reference to Figure 6B , a text instruction 604 for invoking and interacting with the digital assistant may be displayed. Specifically, the instruction 604 may describe the user input required to invoke and interact with the digital assistant. For example, the instruction 604 may explain how to perform the second input type described below at block 516.

[0181] At block 510 of process 500, as Figure 6B As shown in , a passive visual indicator 606 may be displayed on the display unit. The passive visual indicator 606 may indicate that the digital assistant has not been called. Specifically, the microphone (e.g., microphone 272) of the media device may not be activated in response to detecting user input. Therefore, the passive visual indicator 606 can be used as a visual signal that the digital assistant is not processing audio input. In this example, the visual indicator 606 can be a passive flat waveform that does not respond to the user's voice. In addition, the passive visual indicator 606 may include a neutral color (e.g., black, gray, etc.) to indicate its passive state. It should be appreciated that other visual patterns or images can be expected for the passive visual indicator. The passive visual indicator 606 may be displayed simultaneously with the instruction 604. In addition, the passive visual indicator 606 may be continuously displayed when executing one or more of the boxes in boxes 512 to 514.

[0182] At block 512 of process 500, and with reference to Figure 6C, instructions 608 for performing a typed search may be displayed on the display unit. Specifically, instructions 608 may describe the user input required to display a virtual keyboard interface that can be used to perform a typed search. In some examples, instructions 604 for invoking and interacting with a digital assistant and instructions 608 for performing a typed search may be displayed in sequence at different times. For example, the display of instructions 608 may replace the display of instructions 604, or vice versa. In this example, instructions 604, 608 are in text form. It should be appreciated that in other examples, instructions 604, 608 may be in graphical form (e.g., pictures, symbols, animations, etc.).

[0183] At block 514 of process 500, one or more exemplary natural language requests may be displayed on a display unit. For example, FIG. 6D to FIG. 6E Two different exemplary natural language requests 610, 612 displayed on the display unit are shown. In some examples, the exemplary natural language request may be displayed via a first user interface on the display unit. The first user interface may be overlaid on the displayed content. The exemplary natural language request may provide guidance to the user for interacting with the digital assistant. In addition, the exemplary natural language request may inform the user of the various capabilities of the digital assistant. In response to receiving a user utterance corresponding to an exemplary natural language request in the exemplary natural language request, the digital assistant may cause the corresponding action to be performed. For example, in response to the digital assistant of the media device being called (e.g., by user input of the second input type at box 504) and provided with a user utterance of "jump forward 30 seconds" (e.g., at box 518), the digital assistant may cause the media content played on the media device to jump forward 30 seconds.

[0184] The displayed exemplary natural language request may be contextually related to the content being displayed (e.g., media content 602). For example, a set of exemplary natural language requests may be stored on a media device or on a separate server. Each exemplary natural language request in the set of exemplary natural language requests may be associated with one or more contextual attributes (e.g., media content being played, homepage, iTunes media store, actor, movie, weather, sports, stock market, etc.). In some examples, box 514 may include identifying an exemplary natural language request from the set of exemplary natural language requests having contextual attributes corresponding to the displayed content on the display unit. The identified exemplary natural language request may then be displayed on the display unit. Therefore, different exemplary natural language requests may be displayed for different displayed content on the display unit. Displaying contextually related exemplary natural language requests may be used to conveniently inform the user of the capabilities of the digital assistant that are most relevant to the user's current usage conditions on the media device. This may improve the overall user experience.

[0185] exist FIG. 6D to FIG. 6EIn the example shown, the exemplary natural language request 610 and the exemplary natural language request 612 may each be contextually related to the media content 602 on the display unit. Specifically, the exemplary natural language request 610 and the exemplary natural language request 612 may be requests for modifying or controlling one or more settings associated with the media content played on the media device. Such exemplary natural language requests may include requests for the following operations: turning on / off closed captions, turning on subtitles in a specific language, rewinding / skipping forward, pausing media content, restarting media content, slowing down or speeding up media content, increasing / decreasing the volume of media content (e.g., audio gain), etc. In addition, other exemplary natural language requests contextually related to the media content 602 may include requests for the following operations: adding a media item corresponding to the media content 602 to the user's watch list, displaying information related to the media content 602 (e.g., actor information, plot synopsis, release date, etc.), displaying other media items or content related to the media content 602 (e.g., the same series, the same season, the same actor / director, the same category, etc.), and the like.

[0186] In an example where the displayed content includes content associated with an application of a media device, the context-related exemplary natural language request may include a request to modify one or more settings or states of the application. Specifically, the exemplary natural language request may include a request to open or close the application or to manipulate one or more features of the application.

[0187] In some examples, the displayed content may include a user interface (e.g., second user interface 618 or third user interface 626) for searching, browsing, or selecting items. Specifically, the displayed user interface may include one or more media items. In addition, the focus of the user interface may be on a media item in the one or more media items (e.g., Figure 6G623 highlighted by cursor 624 in the user interface). In these examples, the context-dependent exemplary natural language request may include a request for information about one or more media items in the displayed user interface or other media items. Specifically, the exemplary natural language request may include a request related to the media item that is the focus of the user interface. In these examples, the exemplary natural language request may include multiple requests, such as "What is it about?", "How much is it ranked?", "Who is in it?", "When will the next episode appear?", "Show me more movies like this." And "Show me movies starring the same actor." In a specific example, information related to a media item or a series of media items, such as the TV series Mad Men, can be displayed via the user interface. In this example, the context-dependent exemplary natural language request may include a requirement based on one or more attributes (e.g., lineup, plot, ranking, release date, director, provider, etc.) of a media item or a series of media items (e.g., other programs participated in by January Jones). In addition, the context-dependent exemplary natural language request may include a request to play, select, or obtain the focused media item or another media item displayed in the user interface (e.g., "Rent this.", "Play this.", "Buy this.", or "Play How to Train Your Dragon 2."), or include a request to navigate media items in the user interface (e.g., "Go to comedies." or "Jump to horror movies."). In addition, in these examples, the context-dependent exemplary natural language request may include a request to search for other media items (e.g., "Find new comedies.", "Show free and good movies.", or "What are some shows starring Nicole Kidman?").

[0188] In some examples, the displayed content may include media items organized according to a specific category or theme. In these examples, the context-dependent exemplary natural language request may include a request related to the specific category or theme. For example, in an example where the displayed content includes media items organized according to various actors, the context-dependent exemplary natural language request may include a request for information or media items related to the actors (e.g., "What are the movies starring Jennifer Lawrence?", "How old is Scarlett Johansson?", "What are Brad Pitt's latest movies?"). In another example where the displayed content includes media items organized according to program channels or content providers (e.g., channel pages or TV guide pages), the context-dependent exemplary natural language request may include a request for information or media items related to program channels or content providers (e.g., "What's playing in one hour?", "What's playing on HBO in prime time?", "Tuned to ABC." Or "Which channels are playing basketball games?"). In another example where the displayed content includes media items that the user recently selected (e.g., a “recently played” list) or media items identified as being of interest to the user (e.g., a “watch list”), the context-related exemplary natural language request may include a request to watch or continue watching one of the media items (e.g., “Continue playing where you left off.”, “Continue watching Birdman.”, or “Start playing from the beginning.”).

[0189] In some examples, the displayed content may include a user interface containing results or information corresponding to a specific topic. Specifically, the results may be associated with a previous user request (e.g., a request to a digital assistant), and may include information corresponding to a topic such as weather, stock market, or sports. In these examples, the context-dependent exemplary natural language request may include a request for refining the results or a request for additional information about a specific topic. For example, in an example where the displayed content includes weather information for a specific location, the context-dependent exemplary natural language request may include a request for displaying additional weather information for another location or a different time range (e.g., "How is it in New York City?", "What is it like next week?", "What about Hawaii?", etc.). In another example where the displayed content includes information related to a sports team or athlete, the context-dependent exemplary natural language request may include a request for providing additional information related to a sports team or athlete (e.g., "How tall is Shaquille O'Neal?", "When was Tom Brady born?", "When does the next game of the 49ers start?", "How did Manchester United perform in the last game?", "Who is the point guard of the LALakers?", etc.). In another example where the displayed content includes information related to the stock market, the context-dependent exemplary natural language request may include a request for additional stock market-related information (e.g., "What was the opening price of the S&P 500?", "How is Apple's stock price going?", "What was the closing price of the Dow Jones yesterday?", etc.). In addition, in some examples, the displayed content may include a user interface containing media search results associated with a previous user request. In these examples, the context-dependent exemplary natural language request may include a request to refine the displayed media search results (e.g., "Find only those released since last year," "Find only those rated G," "Find only free"), or a request to perform a different media search (e.g., "Find good action movies," "Show me some movies starring Jackie Chan," etc.).

[0190] In some examples, the displayed content may include a main menu user interface of a media device. The main menu user interface may be, for example, a home screen or a root directory of a media device. In these examples, the context-dependent exemplary natural language requests may include requests representing various capabilities of the digital assistant. Specifically, the digital assistant may have a set of core competencies associated with the media device, and the context-dependent exemplary natural language requests may include requests related to each core competency of the digital assistant (e.g., "show me some good and free movies," "what's the weather like," "play the next episode of Breaking Bad," or "what's Apple's stock price?").

[0191] The exemplary natural language request may be in natural language form. This can be used to inform the user that the digital assistant is able to understand natural language requests. In addition, in some examples, the exemplary natural language request may be context-ambiguous, thereby informing the user that the digital assistant is able to infer the correct user intent associated with the user's request based on the displayed content. Specifically, as shown in the above examples, the exemplary natural language request may include context-ambiguous terms such as "this" or "some", or context-ambiguous phrases such as "only find free ones." or "How about in New York?". These exemplary natural language requests may inform the user that the digital assistant is able to determine the correct context associated with such requests based on the displayed content. This will encourage the user to rely on the context of the displayed content when interacting with the digital assistant, which may promote a more natural interactive experience with the digital assistant, which is desirable.

[0192] In some examples, block 514 may be performed after blocks 508 to 512. Specifically, an exemplary natural language request may be displayed on the display unit a predetermined amount of time after determining at block 506 that the user input corresponds to the first input type. It should be appreciated that in some examples, blocks 508 to 514 may be performed in any order, and in some examples, two or more of blocks 508 to 514 may be performed simultaneously.

[0193] In some examples, the exemplary natural language requests are displayed in turn in a predetermined order. Each exemplary natural language request may be displayed separately at different times. Specifically, the display of the current exemplary natural language request may be replaced by the display of the subsequent exemplary natural language request. For example, Fig.6D As shown in , the exemplary natural language request 610 may be displayed first. After a predetermined amount of time, the display of the exemplary natural language request 610 ("skip forward 30 seconds") may be replaced with the display of the exemplary natural language request 612 ("play next episode"), as shown in FIG. Fig. 6E Thus, in this example, example natural language request 610 and example natural language request 612 are displayed one at a time rather than simultaneously.

[0194] In some examples, the exemplary natural language requests may be divided into multiple lists, where each list includes one or more exemplary natural language requests. In these examples, block 514 may include displaying the list of exemplary natural language requests on a display unit. Each list may be displayed at a different time in a predetermined order. In addition, the lists may be displayed in turn.

[0195] When one or more of blocks 508 to 514 are executed, the displayed content may continue to be displayed on the display unit. FIG. 6B to FIG. 6E As shown, when executing blocks 508 to 512, media content 602 can continue to be played on the media device and displayed on the display unit. In addition, audio associated with the media content can be output by the media device when the media content is played. In some examples, in response to detecting a user input or according to determining that the user input corresponds to a first input type, the audio amplitude is not reduced. It is desirable to reduce interference with the consumption of the media content 602 being played. Therefore, although elements 604 to 612 are being displayed on the display unit, the user can still continue to focus on the media content 602 via the audio output.

[0196] In some examples, such as FIG. 6B to FIG. 6D In response to detecting the user input or according to determining that the user input corresponds to the first input type, the brightness of the displayed content can be reduced (e.g., by 20% to 40%), as represented by the outline font of the media content 602 in the display unit. In these examples, the displayed elements 604 to 612 can be overlaid on the displayed media content 602. Reducing the brightness can be used to highlight the displayed elements 604 to 612. At the same time, the media content 602 is still discernible on the display unit, so that the user can continue to consume the media content 602 when the elements 604 to 612 are displayed.

[0197] When executing one of blocks 508 to 512, the digital assistant may be invoked (e.g., by detecting user input of the second input type at block 504), and a user utterance corresponding to one of the exemplary natural language requests may be received (e.g., at block 518). The digital assistant may then perform a task in response to the received request (e.g., at block 532). FIG. 5B to FIG. 5I Further details about invoking and interacting with a digital assistant are provided. In addition, when executing one of blocks 508 to 512, a virtual keyboard interface may be invoked (e.g., by detecting a fifth user input at block 558) to perform a typed search. Figure 5G Provides more details about invoking the virtual keyboard interface and performing a typed search.

[0198] Referring again to block 506, based on determining that the user input does not correspond to the first input type, the Figure 5BOne or more of boxes 516 to 530 of the present invention. At box 516, it may be determined whether the user input corresponds to a second input type. The second input type may be a predefined input to the media device that is different from the first input type. In some examples, the second input type may include pressing a specific button on a remote control of the media device and holding the button for more than a predetermined duration (e.g., a long press). The second input type may be associated with invoking a digital assistant. In some examples, the first input type and the second input type may be implemented using the same button of the remote control (e.g., a button configured to invoke a digital assistant). It is desirable to intuitively integrate the invocation of the digital assistant and the instructions provided for invoking and interacting with the digital assistant into a single button. In addition, inexperienced users may intuitively implement a short press rather than a long press. Therefore, providing instructions in response to detecting a short press may make the instructions primarily targeted at inexperienced users rather than experienced users. This may improve the user experience by easily displaying instructions to inexperienced users who need guidance the most while allowing experienced users to choose the option to bypass the instructions.

[0199] Based on determining that the user input at block 516 corresponds to the second input type, one or more of blocks 518 to 530 may be executed. In some examples, while executing one or more of blocks 518 to 530, the media content 602 may continue to be played on the media device. Specifically, while sampling the audio data at block 518 and performing the task at block 528, the media content 602 may continue to be played on the media device and displayed on the display unit.

[0200] At block 518 of process 500, audio data may be sampled. Specifically, a first microphone of the media device (e.g., microphone 272) may be activated to begin sampling audio data. In some examples, the sampled audio data may include user utterances from the user. The user utterances may represent a user request to the digital assistant. Further, in some examples, the user request may be a request to perform a task. Specifically, the user request may be a media search request. For example, referring to Fig. 6F , the sampled audio data may include a user utterance "Find romantic comedies starring Reese Witherspoon." In other examples, the user request may be a request to play a media item or provide specific information (e.g., weather, stock market, sports, etc.).

[0201] The user utterance in the sampled audio data may be in natural language form. In some examples, the user utterance may represent a partially specified user request, where not all information required to satisfy the user request is clearly defined by the user utterance. For example, the user utterance may be "Play the next episode." In this example, the user request does not clearly define which media series' next episode is to be played. Furthermore, in some examples, the user utterance may include one or more ambiguous terms.

[0202] The duration for sampling the audio data may be based on the detection of the endpoint. Specifically, the audio data may be sampled from the start time when the user input of the second input type is initially detected to the end time when the endpoint is detected. In some examples, the endpoint may be based on the user input. Specifically, the first microphone may be activated when the user input of the second input type is initially detected (e.g., a button is pressed for more than a predetermined duration). While the user input of the second input type continues to be detected, the first microphone may remain activated to sample the audio data. Once the user input of the second input type is no longer detected (e.g., the button is released), the first microphone may be deactivated. Therefore, in these examples, the endpoint is detected when the end of the user input is detected. Therefore, the audio data is sampled when the user input of the second input type is detected.

[0203] In other examples, detecting the endpoint may be based on one or more audio features of the sampled audio data. Specifically, one or more audio features of the sampled audio data may be monitored, and the endpoint may be detected at a predetermined time after determining that the one or more audio features do not meet one or more predetermined criteria. In other examples, the endpoint may be detected based on a fixed duration. Specifically, the endpoint may be detected at a predetermined duration after initially detecting the user input of the second input type.

[0204] In some examples, when executing box 504 or box 516, audio associated with the displayed content can be output (e.g., using speaker 268). Specifically, the audio can be the audio of the media item played on the media device and displayed on the display unit. The audio can be output via an audio signal from the media device. In these examples, when determining that the user input corresponds to the second input type and when sampling the audio data, the audio associated with the displayed content can be ducked (e.g., reducing the amplitude of the audio). For example, the audio can be ducked by reducing the gain associated with the audio signal. In other examples, when the audio data is sampled at box 518, the output of the audio associated with the media content can be stopped. For example, the audio output can be stopped by blocking or interfering with the audio signal. The ducking or stopping of the audio output can reduce the background noise in the sampled audio data and increase the relative strength of the voice signal associated with the user's speech. In addition, the ducking or stopping of the audio can be used as an audio prompt to prompt the user to start providing voice input to the digital assistant.

[0205] In some examples, background audio data can be sampled when audio data is sampled, thereby eliminating noise. In these examples, a remote controller or a media device may include a second microphone. The second microphone may be oriented in a direction different from the first microphone (e.g., relative to the first microphone). The second microphone may be activated to sample background audio data when audio data is sampled. In some examples, background audio data can be used to eliminate background noise in audio data. In other examples, a media device may generate an audio signal for outputting audio associated with displayed content. The generated audio signal can be used to eliminate background noise from audio data. Eliminating background noise from an audio signal may be particularly suitable for interaction with a digital assistant in a media environment. This may be due to the public nature of consuming media content, where speech from multiple individuals may be mixed in audio data. By eliminating background noise in audio data, a higher signal-to-noise ratio in audio data may be obtained, which is desirable when processing audio data requested by a user.

[0206] At block 520 of process 500 and with reference to Fig. 6F , an active visual indicator 614 may be displayed on the display unit. The active visual indicator 614 may indicate to the user that the digital assistant is invoked and is actively listening. Specifically, the active visual indicator 614 may serve as a visual prompt to prompt the user to begin providing voice input to the digital assistant. In some examples, the active visual indicator 614 may include color and / or visual animation to indicate that the digital assistant is invoked. For example, Fig. 6FAs shown, active visual indicator 614 may include an active waveform responsive to one or more characteristics (e.g., amplitude) of the audio data received by the digital assistant. For example, active visual indicator 614 displays a waveform with a larger amplitude in response to a louder portion of the audio data, and displays a waveform with a smaller amplitude in response to a quieter portion of the audio data. In addition, when passive visual indicator 606 (e.g., Fig. 6E ), the display of the active visual indicator 614 may be used to replace the display of the visual indicator 606. This may provide a FIG. 6B to FIG. 6E The instructive user interface shown in the figure shows how to invoke and interact with the digital assistant. Fig. 6F A natural transition to an active user interface for actively interacting with a digital assistant as shown in FIG.

[0207] At block 522 of process 500, a textual representation of the user utterance in the sampled audio data may be determined. For example, the textual representation may be determined by performing speech-to-text (STT) processing on the sampled audio data. Specifically, the sampled audio data may be processed using an STT processing module (e.g., STT processing module 430) to convert the user utterance in the sampled audio data into a textual representation. The textual representation may be a string of symbols representing a corresponding textual string.

[0208] In some examples, STT processing can be biased toward media-related text results. Biasing can be achieved by utilizing a language model trained using a corpus of media-related text. In addition or as an alternative, biasing can be achieved by weighting media-related candidate text results more heavily. In this way, media-related candidate text results can be ranked higher when biasing is utilized than when there is no bias. Biasing may be desirable to increase the accuracy of STT processing of media-related user utterances (e.g., movie titles, movie actors, etc.). For example, without biasing toward media-related text results, certain media-related words or phrases, such as "JurassicPark," "Arnold Schwarzenegger," and "Shrek," may be rarely found in a typical text corpus, and therefore may not be successfully recognized during STT processing.

[0209] In some examples, a textual representation may be obtained from an independent device (e.g., DA server 106). Specifically, sampled audio data may be transmitted from a media device to an independent device to perform STT processing. In these examples, the media device may indicate to the independent device (e.g., via data transmitted to the independent device having the sampled audio data) that the sampled audio data is associated with a media application. This indication may bias the STT processing toward media-related textual results.

[0210] In some examples, the text representation may be based on previous user utterances received by the media device before sampling the audio data. Specifically, candidate text results for sampled audio data corresponding to one or more portions of previous user utterances may be weighted more heavily. In some examples, previous user utterances may be used to generate a language model, and the generated language model may be used to determine a text representation of the current user utterance in the sampled audio data. The language model may be dynamically updated as additional user utterances are received and processed.

[0211] In addition, in some examples, the text representation may be based on the time at which a previous user utterance was received before the audio data was sampled. Specifically, candidate text results corresponding to previous user utterances received more recently relative to the sampled audio data may be weighted more heavily than candidate text results corresponding to previous user utterances received earlier relative to the sampled audio data.

[0212] At block 524 of process 500, the textual representation may be displayed on a display unit. For example, Fig. 6F A text representation 616 corresponding to the user utterance in the sampled audio data is shown. In some examples, blocks 522 and 524 may be performed while the audio data is being sampled. Specifically, the text representation 616 of the user utterance may be displayed in a streaming manner such that the text representation 616 is displayed in real time while the audio data is being sampled and the STT processing is performed on the sampled audio data. Displaying the text representation 616 may provide the user with confirmation that the digital assistant is processing the user request correctly.

[0213] At block 526 of process 500, a user intent corresponding to the user utterance may be determined. The user intent may be determined by performing natural language processing on the text representation of block 522. Specifically, a natural language processing module (e.g., natural language processing module 432) may be used to process the text representation to obtain the user intent. For example, referring to Fig. 6F , it can be determined from the text representation 616 corresponding to “Find romantic comedies starring Reese Witherspoon” that the user intent is a request to search for media items categorized as romantic comedies and starring actor Reese Witherspoon. In some examples, block 526 may also include using a natural language processing module to generate a structured query representing the determined user intent. In the present example of “Find romantic comedies starring Reese Witherspoon,” a structured query representing a media search query for media items categorized as romantic comedies and starring actor Reese Witherspoon may be generated.

[0214] In some examples, natural language processing used to determine user intent may be biased toward media-related user intent. Specifically, a natural language processing module may be trained to identify media-related words and phrases (e.g., media titles, media classifications, actors, MPAA movie rating labels, etc.) that are used to trigger media-related nodes in a knowledge ontology. For example, a natural language processing module may identify the phrase "Jurassic Park" in a text representation as a movie title, thereby triggering a "media search" node in a knowledge ontology associated with an executable intent to search for media items. In some examples, biasing may be achieved by limiting the nodes in the knowledge ontology to a predetermined group of media-related nodes. For example, the group of media-related nodes may be nodes associated with an application of a media device. In addition, in some examples, biasing may be achieved by weighting candidate user intents related to the media more heavily than candidate user intents that are not related to the media.

[0215] In some examples, user intent may be obtained from an independent device (e.g., DA server 106). Specifically, audio data may be transmitted to the independent device to perform natural language processing. In these examples, the media device may indicate to the independent device (e.g., via data transmitted to the independent device with the sampled audio data) that the sampled audio data is associated with a media application. This indication may bias natural language processing toward media-related user intent.

[0216] At block 528 of process 500, it may be determined whether the sampled audio data includes a user request. This determination may be made based on the determined user intent of block 526. If the user intent includes a user request for performing a task, it may be determined that the sampled audio data includes a user request. Conversely, if the user intent does not include a user request for performing a task, it may be determined that the sampled audio data does not include a user request. In addition, in some examples, if the user intent cannot be determined from the text representation at block 526 or the text representation cannot be determined from the sampled audio data at block 522, it may be determined that the sampled audio data does not include a user request. Based on determining that the audio data does not include a user request, block 530 may be executed.

[0217] At block 530 of process 500, a request for clarification of the user's intent may be displayed on a display unit. In one example, the request for clarification may be a request for the user to repeat the user's request. In another example, the request for clarification may be a statement that the digital assistant could not understand the user's utterance. In yet another example, an error message may be displayed to indicate that the user's intent could not be determined. Furthermore, in some examples, based on determining that the audio data does not include a user request, no response may be provided.

[0218] refer to Figure 5C, according to determining at box 528 that the sampled audio data contains a user request, box 532 may be executed. At box 532 of process 500, a task that at least partially satisfies the user request may be performed. For example, performing a task at box 526 may include performing one or more tasks defined in the generated structured query of box 526. The task flow processing module (e.g., task flow processing module 436) of the digital assistant may be used to perform one or more tasks. In some examples, the task may include changing the state or setting of an application on a media device. More specifically, the task may include, for example, selecting or playing a requested media item, opening or closing a requested application, or navigating in a displayed user interface in a requested manner. In some examples, the task may be performed at box 532 and no voice associated with the task may be output from the media device. Therefore, although in these examples the user may provide a request to the digital assistant in the form of voice, the digital assistant may not provide a response to the user in the form of voice. Instead, the digital assistant may respond only visually by displaying the result on a display unit. It is desirable to retain the public experience of consuming media content in this way.

[0219] In other examples, the task may include retrieving and displaying the requested information. Specifically, performing the task at box 532 may include performing one or more boxes in boxes 534 to 536. At box 534 of process 500, a result that at least partially satisfies the user request may be obtained. The result may be obtained from an external service (e.g., external service 120). In one example, the user request may be a request to perform a media search query, such as "Find romantic comedies starring Reese Witherspoon." In this example, box 534 may include performing the requested media search (e.g., using a media-related database of an external service) to obtain media items classified as romantic comedies and having actor Reese Witherspoon. In other examples, the user request may include requests for other types of information such as weather, sports, and the stock market, and the corresponding information may be obtained at box 534.

[0220] At block 536 of process 500, a second user interface may be displayed on the display unit. The second user interface may include a portion of the results obtained at block 534. For example, Figure 6GAs shown, the second user interface 618 may be displayed on the display unit. The second user interface 618 may include media items 622 that satisfy the user request of "Find me a romantic comedy starring Reese Witherspoon." In this example, the media items 622 may include some media items such as "Legally Blonde," "Legally Blonde 2," "Hot Pursuit," and "This Means War." The second user interface 618 may also include a text header 620 that describes the obtained results. The text header 620 may rephrase a portion of the user request to convey the impression that the user's request has been directly processed. This provides a more humanized interactive experience between the user and the digital assistant. Figure 6G In the present example shown in , the media items 622 are organized in a single row across the second user interface 618. It should be appreciated that in other examples, the organization and presentation of the media items 622 may vary.

[0221] The second user interface 618 may also include a cursor 624 for navigating and selecting media items 622 in the second user interface 618. The location of the cursor may be indicated by visually highlighting the media item where the cursor is located relative to other media items. For example, in this example, the media item 623 where the cursor 624 is located may be larger and bolder than the other media items displayed in the second user interface 618.

[0222] In some examples, while the second user interface is displayed, at least a portion of the displayed content may continue to be displayed. Figure 6G As shown, the second user interface 618 can be a small pane displayed at the base of the display unit, while the media content 602 continues to play on the media device and is displayed on the display unit above the second user interface 618. The second user interface 618 can be overlaid on the media content 602 being played. In this example, the display area of ​​the second user interface 618 on the display unit can be smaller than the display area of ​​the media content 602 on the display unit. This is desirable to reduce the distraction of the results displayed by the digital assistant while the user is consuming media content. It should be appreciated that in other examples, the display area of ​​the second user interface can vary relative to the display area of ​​the displayed content. In addition, as shown in FIG. Figure 6G As shown in the solid font “MEDIAPLAYING” in FIG. 6 , when the second user interface 618 is displayed, the brightness of the media content 602 may be restored to normal brightness (e.g., before detecting the user input). Fig. 6A This can be used to indicate to the user that the interaction with the digital assistant is complete. Thus, the user can continue to consume media content 602 while viewing the requested result (e.g., media item 622).

[0223] In the example where the media items obtained from the media search are displayed on the second user interface, the number of displayed media items can be limited. This allows the user to focus on the most relevant results and prevents the user from facing too many options when making a selection. It is desirable. In these examples, frame 532 may also include determining whether the number of media items in the obtained results is less than or equal to a predetermined number (e.g., 30, 28, or 25). Based on determining that the number of media items in the obtained results is less than or equal to a predetermined number, all media items in the obtained results may be included in the second user interface. Based on determining that the number of media items in the obtained results is greater than a predetermined number, only a predetermined number of media items in the obtained results may be included in the second user interface.

[0224] In addition, in some examples, only the media items in the obtained results that are most relevant to the media search request may be displayed in the second user interface. Specifically, each of the media items in the obtained results may be associated with a relevance score relative to the media search request. The displayed media items may have the highest relevance score in the obtained results. In addition, the media items in the second user interface may be arranged according to the relevance score. For example, referring to Figure 6G , media items with higher relevance scores are more likely to be close to one side of the second user interface 618 (e.g., close to the side of the cursor 624), while media items with lower relevance scores are more likely to be close to the opposite side of the user interface 618 (e.g., away from the side of the cursor 624). In addition, each media item in the obtained results can be associated with a popularity rating. The popularity rating can be based on the ratings of movie critics (e.g., rotten tomatoes ratings) or based on the number of users who have selected to playback the media item. In some examples, the arrangement of media items 622 in the second user interface 618 can be based on popularity ratings. For example, media items with higher popularity ratings are more likely to be positioned on one side of the second user interface 618, while media items with lower popularity ratings are more likely to be close to being positioned on the opposite side of the second user interface 618.

[0225] like Figure 5C The different flows (e.g., D, E, F, and G) after relay block 532 may be performed after block 532. Figure 5D Frame 538, Figure 5E Frame 542, Fig. 5F Box 550 or Fig.5I538, frame 542, frame 550, or frame 570 may be executed when the second user interface is displayed at frame 536. In some examples, process 500 may alternatively include a determination step after frame 536 to determine the appropriate flow to be executed (e.g., D, E, F, or G). Specifically, user input may be detected after frame 536, and it may be determined whether the detected user input corresponds to the second user input (e.g., frame 538), the third user input (e.g., frame 542), the fourth user input (e.g., frame 550), or the sixth user input (e.g., frame 570). For example, based on determining that the user input corresponds to the third user input of frame 542, one or more of frames 544 to 546 may be executed. After frame 546, a similar determination step may also be included.

[0226] At block 538 of process 500 and with reference to Figure 5D , a second user input may be detected. As described above, the second user input may be detected when the second user interface is displayed on the display unit. The second user input may be detected on a remote control of the media device. For example, the second user input may include a first predetermined action pattern on a touch-sensitive surface of the remote control. In one example, the first predetermined action pattern may include continuous contact action in a first direction from a first contact point to a second contact point on the touch-sensitive surface. When the remote control is held in an intended manner, the first direction may be a downward direction or a direction toward the user. It should be appreciated that other forms of input may be expected for the second user input. In response to detecting the second user input, box 540 may be executed.

[0227] At block 540 of process 500, the second user interface may be eliminated so that the second user interface is no longer displayed. Figure 6G In response to detecting the second user input, the second user interface 618 may stop being displayed. In this example, when the second user interface 618 is eliminated, the media content 602 may be displayed in full screen on the display unit. For example, when the second user interface 618 is stopped being displayed, Fig. 6A Media content 602 is shown being displayed.

[0228] At block 542 of process 500 and with reference to Figure 5E, a third user input may be detected. The third user input may be detected when the second user interface is displayed on the display unit. The third user input may be detected on a remote control of the media device. For example, the third user input may include a second predetermined action pattern on the touch-sensitive surface of the remote control. The second predetermined action pattern may include continuous contact actions in a second direction from a third contact point to a fourth contact point on the touch-sensitive surface. The second direction may be opposite to the first direction. Specifically, when the remote control is held in an intended manner, the second direction may be an upward direction or a direction away from the user. In response to detecting the third user input, one or more of boxes 544 to box 546 may be executed. In some examples, such as Figure 6G As shown, the second user interface 618 may include a graphical indicator 621 (e.g., an arrow) to indicate to the user that the second user interface 618 can be expanded by providing a third user input. In addition, the graphical indicator 621 may indicate to the user a second direction associated with a second predetermined action mode on the touch-sensitive surface for the third user input.

[0229] At block 544 of process 500, a second result may be obtained. The obtained second result may be similar to, but different from, the result obtained at block 534. In some examples, the obtained second result may at least partially satisfy the user request. For example, the obtained second result may share one or more characteristics, parameters, or attributes of the result obtained at block 534. FIG. 6F to FIG. 6G In the example shown, block 544 may include performing one or more additional media search queries related to the media search query performed at block 534. For example, the one or more additional media search queries may include searching for media items categorized as romantic comedies or searching for media items starring Reese Witherspoon. Thus, the resulting second results may include media items that are romantic comedies (e.g., media item 634) and / or media items starring Reese Witherspoon (e.g., media item 636).

[0230] In some examples, the obtained second results may be based on a previous user request received before detecting the user input at block 504. Specifically, the obtained second results may include one or more characteristics or parameters of the previous user request. For example, the previous user request may be "Show me movies released in the past 5 years." In this example, the obtained second results may include media items that are romantic comedies starring Reese Witherspoon and released in the past 5 years.

[0231] Additionally, in some examples, block 544 may include obtaining a second result contextually relevant to the item focused on the second user interface when a third user input is detected. Figure 6G, when the third user input is detected, the cursor 624 may be positioned at the media item 623 in the second user interface 618. The media item 623 may be, for example, the movie "Legally Blonde". In this example, the obtained second results may share one or more characteristics, attributes, or parameters associated with the media item "Legally Blonde". Specifically, the obtained second results may include media items like "Legally Blonde" that are related to attending law school or related to professional women in leadership roles.

[0232] At block 546 of process 500, a third user interface may be displayed on the display unit. Specifically, the display of the second user interface at block 536 may be replaced with the display of the third user interface at block 546. In some examples, in response to detecting the third user input, the second user interface may be expanded to the third user interface. The third user interface may occupy at least a majority of the display area of ​​the display unit. The third user interface may include a portion of the obtained results of block 534. In addition, the third user interface may include a portion of the obtained second results of block 544.

[0233] In one example, if Figure 6H As shown, the third user interface 626 can substantially occupy the entire display area of ​​the display unit. In this example, the display of the third user interface 626 can be used to replace the previous display of the media content 602 and the second user interface 618. In response to detecting the third user input, the media content can be paused on the media device. This can prevent the user from losing any part of the media content 602 when browsing the media items in the third user interface 626.

[0234] The third user interface 626 may include media items 622 that satisfy the user request of "Find me romantic comedies starring Reese Witherspoon." In addition, the third user interface 626 may include media items 632 that at least partially satisfy the same user request. The media items 632 may include multiple media item groups that each correspond to different characteristics, attributes, or parameters. In this example, the media items 632 may include media items 634 that are romantic comedies and media items 636 that are starring Reese Witherspoon. Each media item group may be marked using a text header (e.g., text headers 628, 630). The text header may describe one or more attributes or parameters associated with the media items of the corresponding group. In addition, each text header may be an exemplary user utterance that, when provided to the digital assistant by the user, may cause the digital assistant to obtain media items of a similar group. For example, with reference to the text header 628, in response to receiving the user utterance "romantic comedy" from the user, the digital assistant may obtain and display a media item (e.g., media item 634) that is a romantic comedy.

[0235] Although in Figure 6H In the example shown, media item 622 is based on the initial user request "Find romantic comedies starring Reese Witherspoon," but it should be recognized that in other examples, media item 632 may be based on other factors, such as media selection history, media search history, the order in which previous media searches were received, relationships between media-related attributes, popularity of media items, etc.

[0236] In an example where the user request is a media search request, the second result obtained may be based on the number of media items in the obtained result of box 534. Specifically, in response to detecting a third user input, it may be determined whether the number of media items in the obtained result is less than or equal to a predetermined number. Based on determining that the number of media items in the obtained result is less than or equal to a predetermined number, the second result obtained may include media items different from the media items in the second user interface. The second result obtained may at least partially satisfy the media search request performed at box 534. At the same time, the range of the second result obtained may be wider than the range of the previously obtained result, and may be associated with some of the parameters defined in the media search request performed at box 534. It is desirable to provide the user with a wider set of results and more options to choose from.

[0237] In some examples, based on determining that the number of media items in the obtained results is less than or equal to the predetermined number in block 534, it can be determined whether the media search request includes more than one search attribute or parameter. Based on determining that the media search request includes more than one search attribute or parameter, the obtained second results can include media items associated with the more than one search attribute or parameter. In addition, the media items in the obtained second results can be organized in the third user interface according to the more than one search attribute or parameter.

[0238] exist FIG. 6F to FIG. 6H In the example shown, the media search request "Find romantic comedies starring Reese Witherspoon" may be determined to include more than one search attribute or parameter (e.g., "romantic comedies" and "Reese Witherspoon"). Based on the determination that the media search request includes more than one search attribute or parameter, the resulting second results may include media items 634 associated with the search parameter "romantic comedies" and media items 636 associated with the search parameter "movies starring Reese Witherspoon". Figure 6H As shown, media items 634 may be organized in the "Romantic Comedies" category, and media items 636 may be organized in the "Reese Witherspoon" category.

[0239] In some examples, in accordance with determining that the number of media items in the obtained results of block 534 is greater than a predetermined number, the third user interface may include a first portion and a second portion of the obtained results. The first portion of the obtained results may include a predetermined number of media items (e.g., with the highest relevance score). The second portion of the obtained results may be different from the first portion of the obtained results and may include more media items than the first portion of the obtained results. In addition, it may be determined whether the media items in the obtained results include more than one media type (e.g., movies, television programs, music, applications, games, etc.). In response to determining that the media items in the obtained results include more than one media type, the media items in the second portion of the obtained results may be organized according to media type.

[0240] exist Fig.6I In the example shown, the results obtained at box 534 may include media items starring Reese Witherspoon as romantic comedies. Based on determining that the number of media items in the obtained results is greater than a predetermined number, a first portion of the obtained results (media items 622) and a second portion of the obtained results (media items 638) may be displayed in the third user interface 626. In response to determining that the obtained results include more than one media type (e.g., movies and television programs), media items 638 may be organized according to media type. Specifically, media items 640 may be organized by the category of "movies", and media items 642 may be organized by the category of "television programs". In addition, in some examples, each media item group (e.g., media item 640, media item 642) corresponding to a corresponding media type (e.g., movies, television programs) may be sorted according to the most popular classification, actor / director, or release date within the corresponding media item group. It should be appreciated that in other examples, in response to determining that the media items in the obtained results are associated with more than one media attribute or parameter, the media items in the second portion of the obtained results may be organized according to the media attribute or parameter (rather than the media type).

[0241] In some examples, a user input representing a scroll command (e.g., a fourth user input described below at box 550) may be detected. In response to receiving the user input representing the scroll command, the extended user interface (or more specifically, the items in the extended user interface) may be scrolled. While scrolling, it may be determined whether the extended user interface has been scrolled beyond a predetermined position in the extended user interface. In response to determining that the extended user interface has scrolled beyond a predetermined position in the extended user interface, media items in the third portion of the obtained results may be displayed on the extended user interface. The media items in the third portion may be organized according to one or more media content providers (e.g., iTunes, Netflix, HuluPlus, HBO, etc.) associated with the media items in the third portion. It should be appreciated that in other examples, in response to determining that the extended user interface has scrolled beyond a predetermined position in the extended user interface, other media items may be obtained. For example, popular media items or media items related to the obtained results may be obtained.

[0242] As from Figure 5E The different streams (e.g., B, F, G, and H) starting from block 546 in FIG. 540 may be executed after block 532. Fig. 5F Frame 550, Figure 5G Frame 558, Figure 5H Box 566 or Fig.5I Specifically, in some examples, box 550, box 560, box 564, or box 570 can be performed when the third user interface is displayed at box 546.

[0243] At block 550 of process 500 and with reference to Fig. 5F , a fourth user input may be detected. The fourth user input may be detected when a second user interface (e.g., second user interface 618) or a third user interface (e.g., third user interface 626) is displayed on the display unit. In some examples, the fourth user input may be detected on a remote control of the media device. The fourth user input may indicate a direction on the display unit (e.g., up, down, left, right). For example, the fourth user input may be a contact action from a first position on the touch-sensitive surface of the remote control to a second position to the right of the first position on the touch-sensitive surface. The contact action may therefore correspond to a rightward direction on the display unit. In response to detecting the fourth user input, box 552 may be executed.

[0244] At block 552 of process 500, the focus of the second user interface or the third user interface can be switched from the first item to the second item on the second user interface or the third user interface. The second item can be positioned in a direction relative to the first item (e.g., the same direction corresponding to the fourth user input). Figure 6G, the focus of the second user interface 618 may be on the media item 623, wherein the cursor 624 is positioned at the media item 623. In response to detecting a fourth user input corresponding to a right direction on the display unit, the focus of the second user interface 618 may be shifted from Figure 6G Media item 623 in Switch to Figure 6J Specifically, the position of cursor 624 can be changed from media item 623 to media item 625. In another example, referring to Figure 6H , the focus of the third user interface 626 can be located on the media item 623. In response to detecting a fourth user input corresponding to a downward direction on the display unit, the focus of the third user interface 626 can be moved from Figure 6H Media item 623 in Switch to Figure 6K 627 is positioned below the media item 623. Specifically, the position of the cursor 624 can be changed from the media item 623 to the media item 627.

[0245] At block 554 of process 500, a selection of a media item from the one or more media items may be received via the second user interface or the third user interface. Figure 6J , a selection of the media item 625 may be received via the second user interface 618 by detecting a user input corresponding to the user selection when the cursor 624 is positioned at the media item 625. Similarly, referring to Figure 6K A selection of media item 627 may be received via third user interface 626 by detecting user input corresponding to a user selection when cursor 624 is positioned at media item 627. In response to receiving a selection of a media item from the one or more media items, block 556 may be executed.

[0246] At block 556 of process 500, media content associated with the selected media item may be displayed on a display unit. In some examples, the media content may be a movie, video, television program, animation, etc., being played or streamed on a media device. In some examples, the media content may be a video game, an e-book, an application, or a program running on a media device. In addition, in some examples, the media content may be information related to the media item. The information may be product information describing various characteristics of the selected media item (e.g., plot synopsis, actors, director, author, release date, rating, duration, etc.).

[0247] At block 558 of process 500 and with reference to Figure 5G , the fifth user input may be detected. In some examples, the fifth user input may be detected while the third user interface (e.g., third user interface 626) is displayed. In these examples, the fifth user input may be detected while the focus of the third user interface is on a media item in the top row of the third user interface (e.g., Figure 6H In some embodiments, the fifth user input may be detected when the first user interface is displayed. In some embodiments, the fifth user input may be detected when any one of boxes 508 to 514 is executed. In some examples, the fifth user input may be detected on a remote control of the media device. The fifth user input may be similar to or identical to the third user input. For example, the fifth user input may include a continuous contact action in a second direction on the touch-sensitive surface (e.g., an upward sliding contact action). In other examples, the fifth user input may be an activation of an enable representation. The enable representation may be associated with a virtual keyboard interface or a typing search interface. In response to detecting the fifth user input, one or more of boxes 560 to 564 may be executed.

[0248] At block 560 of process 500, a search field configured to receive typed search input may be displayed. Figure 6L As shown, search field 644 may be displayed on the displayed unit. In some examples, the search field may be configured to receive a typed search query. The typed search query may be a media-related search query, such as a search for a media item. In some examples, the search field may be configured to perform a media-related search based on a text string match between the text entered via the search field 644 and the stored text associated with the media item. In addition, in some examples, the digital assistant may not be configured to receive input via the search field 644. This may encourage users to interact with the digital assistant via a voice interface rather than a typing interface to promote a more humane interface between a media device and a user. It should be appreciated that in some examples, the search field may have been displayed in a second user interface (e.g., second user interface 618) or a third user interface (e.g., third user interface 626). In these examples, it is not necessary to execute frame 566.

[0249] At block 562 of process 500, a virtual keyboard interface may be displayed on a display unit. Figure 6L As shown, a virtual keyboard interface 646 may be displayed. The virtual keyboard interface 646 may be configured such that user input received via the virtual keyboard interface 646 results in text input in the search field. In some examples, the virtual keyboard interface is not available for interaction with a digital assistant.

[0250] At block 564 of process 500, the focus of the user interface may be switched to the search field. Figure 6L , the search field 644 may be highlighted at box 568. In addition, a text input cursor may be positioned in the search field 644. In some examples, text may be displayed in the search field to prompt the user to enter a typed search. Figure 6LAs shown, text 648 includes the prompt "Enter to search."

[0251] At block 566 of process 500 and with reference to Figure 5H , a seventh user input may be detected. In some examples, the seventh user input may be detected while the third user interface (e.g., third user interface 626) is displayed. In some examples, the seventh user input may include pressing a button of a remote control of the electronic device. The button may be, for example, a menu button for navigating to a main menu user interface of the electronic device. It should be appreciated that in other examples, the seventh user input may include other forms of user input. In response to detecting the seventh user input, box 568 may be executed.

[0252] At box 568 of process 500, a third user interface may be displayed on the display unit. Specifically, the seventh user input may cause the third user interface to be eliminated. In some examples, the seventh user input may cause the main menu user interface menu to be displayed, thereby replacing the third user interface. Alternatively, in an example where media content (e.g., media content 602) is displayed before displaying the third user interface (e.g., third user interface 626) and playback of the media content on the electronic device is paused (e.g., paused in response to detection of the third user input) while the third user interface is displayed, playback of the media content on the electronic device may be resumed in response to detection of the seventh user input. Therefore, media content may be displayed in response to detection of the seventh user input.

[0253] At block 570 of process 500 and with reference to Fig.5I , the sixth user input can be detected. Figure 6M As shown, the sixth user input may be detected when the third user interface 626 is displayed. However, in other examples, the sixth user input may be alternatively detected when the second user interface (e.g., the second user interface 618) is displayed. When the sixth user input is detected, the second user interface or the third user interface may include a portion of the result that at least partially satisfies the user request. The sixth user input may include an input for invoking a digital assistant of an electronic device. Specifically, the sixth user input may be similar or identical to the user input of the second input type described above with reference to box 516. For example, the sixth user input may include pressing a specific button on a remote control of a media device and holding the button for more than a predetermined duration (e.g., a long press). In response to detecting the sixth user input, one or more of boxes 572 to 592 may be executed.

[0254] At block 572 of process 500, second audio data may be sampled. Block 572 may be similar or identical to block 518 described above. Specifically, the sampled second audio data may include a user utterance from a second user. The second user utterance may represent a second user request to the digital assistant. In some examples, the second user request may be a request to perform a second task. For example, referring to Figure 6M , the sampled second audio data may include a second user utterance "Only those movies starring Luke Wilson.". In this example, the second user utterance may represent a second user request to refine the previous media search to include only media items with the actor Luke Wilson. In this example, the second user utterance is in natural language form. In addition, in the case where the second user utterance does not explicitly specify all information required to define the user request, the second user request may be partially specified. For example, the second user utterance does not explicitly specify what "those" refers to. In other examples, the second user request may be a request to play a media item or provide specific information (e.g., weather, stock market, sports, etc.).

[0255] It should be appreciated that in some examples, the above blocks 520 to 526 may be similarly performed with respect to the sixth user input. Figure 6M As shown, when the sixth user input is detected, an active visual indicator 614 may be displayed on the display unit. A second textual representation 650 of the second user utterance may be determined (e.g., using the STT processing module 430) and displayed on the display unit. A second user intent corresponding to the second user utterance may be determined based on the second textual representation (e.g., using the natural language processing module 432). In some examples, such as Figure 6M As shown, in response to detecting the sixth user input, the content displayed on the display unit when the sixth user input is detected can be faded or reduced in brightness. This can be used to highlight the active visual indicator 614 and the second text representation 650.

[0256] At block 574 of process 500, a determination may be made as to whether the sampled second audio data includes a second user request. Block 574 may be similar or identical to block 528 described above. Specifically, the determination at block 574 may be made based on a second user intent determined from a second textual representation of the second user utterance. Based on determining that the second audio data does not include a user request, block 576 may be performed. Alternatively, based on determining that the second audio data includes a second user request, one or more of blocks 578 to 592 may be performed.

[0257] At block 576 of process 500, a request to clarify the user's intent can be displayed on the display unit. Block 576 can be similar or identical to block 530 described above.

[0258] At block 578 of process 500, a determination may be made as to whether the second user request is a request to refine the result of the user request. In some examples, the determination may be made based on a second user intent corresponding to the second user utterance. Specifically, the second user request may be determined to be a request to refine the result of the user request based on the expressed indication identified in the second user utterance to refine the result of the user request. For example, referring to Figure 6M , the second text representation 650 can be parsed during natural language processing to determine whether the second user utterance includes a predetermined word or phrase corresponding to the explicit intent of refining the media search results. Examples of words or phrases corresponding to the explicit intent of refining the media search results may include "only", "only", "filtered by...", etc. Therefore, it can be determined based on the word "only" in the second text representation 650 that the second user request is a request for refining the media search results associated with the user request "Find romantic comedies starring Reese Witherspoon". It should be recognized that other techniques can be implemented to determine whether the second user request is a request for refining the results of the user request. Based on determining that the second user request is a request for refining the results of the user request, one or more of boxes 580 to box 582 can be executed.

[0259] At box 580 of process 500, a subgroup of results that at least partially satisfy the user request may be obtained. In some examples, a subgroup of results may be obtained by filtering existing results according to additional parameters defined in the second user request. For example, the obtained results at box 534 (e.g., including media item 622) may be filtered so that media items with actor Luke Wilson are identified. In other examples, a new media search query that combines the requirements of the user request and the second user request may be executed. For example, the new media search query may be a search query for media items that are classified as romantic comedies and have actors Reese Witherspoon and Luke Wilson. In this example, the new media search query may result in media items such as "Legally Blonde" and "Legally Blonde 2".

[0260] In the example of detecting the sixth user input while displaying the third user interface, additional results related to the user request and / or the second user request may be obtained. The additional results may include media items having one or more attributes or parameters described in the user request and / or the second user request. In addition, the additional results may not include all attributes or parameters described in the user request and the second user request. For example, referring to Figure 6H and Figure 6MIn the example described in , the additional results may include media items having at least one (but not all) of the following attributes or parameters: romantic comedy, Reese Witherspoon, and Luke Wilson. The additional results may provide the user with a broader set of results and more options to choose from, which is desirable. In addition, the additional results may be related results that are likely to interest the user.

[0261] At block 582, a subset of the results may be displayed on a display unit. Figure 6N As shown, a subgroup of results may include media items 652, which may include movies, such as "Legally Blonde" and "Legally Blonde 2". In this example, media items 652 are displayed in the top row of the third user interface 626. Text headers 656 may describe attributes or parameters associated with the displayed media items 652. Specifically, text headers 656 may include a paraphrase of the user intent associated with the second user utterance. Figure 6G In an example where the sixth user input is detected when the second user interface 618 shown in FIG. 1 is displayed, media item 652 may alternatively be displayed in the second user interface. In these examples, media item 652 may be displayed as a single row across the second user interface. It should be appreciated that the manner in which media item 652 is displayed in the second user interface or the third user interface may vary.

[0262] In the example of detecting a sixth user input while displaying the third user interface, additional results related to the user request and / or the second user request may be displayed in the third user interface. Figure 6N , the additional results may include media items 654 having one or more parameters described in the user request and / or the second user request. Specifically, media items 654 may include media items 658 starring Luke Wilson as romantic comedies, and media items 660 starring Luke Wilson and released in the past 10 years. Each media item group (e.g., media item 658, media item 660) can be marked using a text header (e.g., text header 662, text header 664). The text header may describe one or more parameters associated with the media items of the corresponding group. The text header may be in natural language form. In addition, each text header may be an exemplary user utterance that, when provided to the digital assistant by the user, may enable the digital assistant to obtain media items of a similar group. For example, with reference to text header 662, in response to receiving a user utterance "romantic comedies starring Luke Wilson" from the user, the digital assistant may obtain and display media items (e.g., media item 658) that are romantic comedies starring Luke Wilson.

[0263] Referring again to box 578, it can be determined that the second user request is not a request for refining the results of the user request. Such a determination can be made based on the absence of any explicit indication in the second user utterance that the results of the user request are to be refined. For example, when parsing the second text representation of the second user utterance during natural language processing, a predetermined word or phrase corresponding to the explicit intent for refining media search results may not be identified. This may be because the second user request is a request that is unrelated to a previous user request (e.g., a new request). For example, the second user request may be "Find horror movies," which is a request that is unrelated to a previous user request of "Find romantic comedies starring Reese Witherspoon." Alternatively, the second user request may include ambiguous language that can be interpreted as a request for refining the results of a previous user request or a new request that is unrelated to a previous user request. For example, referring to Figure 6P , the second user utterance may be "Luke Wilson," which may be interpreted as a request to refine the results of a previous user request (e.g., refine to include only media items with the actor Luke Wilson), or may be interpreted as a new request unrelated to a previous user request (e.g., a new media search for media items with the actor Luke Wilson). In these examples, the second user request may be determined not to be a request to refine the results of the user request. Based on determining that the second user request is a request to refine the results of the user request, one or more of blocks 584 to 592 may be performed.

[0264] At block 584 of process 500, a second task that at least partially satisfies the second user request may be performed. Block 584 may be similar to block 532 described above, except that the second task of block 584 may be different from the task of block 532. Block 584 may include one or more of blocks 586 to 588.

[0265] At block 586 of process 500, a third result that at least partially satisfies the second user request may be obtained. Block 586 may be similar to block 534 described above. Figure 6P In the example shown, the second user utterance "Luke Wilson" may be interpreted as a request to perform a new media search query to identify media items with the actor Luke Wilson. Thus, in this example, block 586 may include performing the requested media search to obtain media items with the actor Luke Wilson. It should be appreciated that in other examples, the user request may include a request for other types of information (e.g., weather, sports, stock market, etc.), and the corresponding types of information may be obtained at block 586.

[0266] At block 588 of process 500, a portion of the third result may be displayed on a display unit. Figure 6Q , a third result including media items 670 (e.g., movies such as “Playing It Cool,” “The Skeleton Twins,” and “You Kill Me”) featuring actor Luke Wilson may be displayed in the third user interface 626. In this example, the media items 670 may be displayed in the top row of the third user interface 626. The text header 678 may describe the properties associated with the displayed media items 670. Specifically, the text header 678 may include a paraphrase of the determined user intent associated with the second user utterance. Before displaying the second user interface (e.g., Figure 6G In an example where the sixth user input is detected when the second user interface 618 shown in FIG. 1 is displayed, media items 670 may be displayed in the second user interface. In these examples, media items 670 may be displayed in a single row across the second user interface. It should be appreciated that in other examples, the organization or configuration of media items 670 in the second user interface or the third user interface may vary.

[0267] At block 590 of process 500, a fourth result may be obtained that at least partially satisfies the user request and / or the second user request. Specifically, the fourth result may include a media item having one or more attributes or parameters defined in the user request and / or the second user request. Figure 6P and Figure 6Q In the example shown in , the fourth result may include media items having one or more of the following attributes or parameters: romantic comedy, Reese Witherspoon, and Luke Wilson. For example, the fourth result may include media item 676 classified as a romantic comedy and starring Luke Wilson. The resulting fourth result may provide the user with a wider set of results and it is therefore desirable to provide more options to choose from. In addition, the fourth result may be associated with an alternative predicted user intent derived from the second user request and one or more previous user requests to increase the likelihood of satisfying the user's actual intent. This can be used to improve the accuracy and relevance of the results returned to the user, thereby improving the user experience.

[0268] In some examples, at least a portion of the fourth result may include media items having all parameters defined in the user request and the second user request. For example, the fourth result may include media item 674 categorized as a romantic comedy and starring Reese Witherspoon and Luke Wilson. Media item 674 may be associated with an alternative intent to refine the results of a previous user request using the second user request. It is desirable to obtain media item 674 in the case where the user actually wants the second request to be a request to refine the obtained results to increase the likelihood of satisfying the user's actual intent.

[0269] In some examples, a portion of the fourth result may be based on the focus of the user interface when the sixth user input is detected. Specifically, when the sixth user input is detected, the focus of the user interface may be on one or more items of the third user interface. In this example, a portion of the fourth result may be contextually related to the one or more items focused on by the user interface. For example, referring to Figure 6K , cursor 624 may be positioned on media item 627, and thus the focus of third user interface 626 may be on media item 627. In this example, a portion of the fourth result may be obtained using the attributes or parameters associated with media item 627. For example, the category of "movies starring Reese Witherspoon" associated with media item 627 may be used to obtain a portion of the fourth result, where the resulting portion may include media items starring Reese Witherspoon and Luke Wilson. In another example, media item 627 may be adventure movies, and thus a portion of the fourth result may include media items that are adventure movies starring Luke Wilson.

[0270] At block 592 of process 500, a portion of the fourth result may be displayed. In an example where a sixth user input is detected while the third user interface is displayed, a portion of the fourth result may be displayed in the third user interface. Figure 6QAs shown, a portion of the fourth result may include a media item 672 displayed in a row after a media item 670. The media item 672 may be associated with one or more attributes or parameters (e.g., romantic comedy, Reese Witherspoon and Luke Wilson) defined in the second user request and / or the user request. For example, the media item 672 may include a media item 676 as a romantic comedy starring Luke Wilson, and a media item 674 as a romantic comedy starring Reese Witherspoon and Luke Wilson. Each media item group (e.g., media item 674, media item 676) may be marked using a text header (e.g., text header 680, text header 682). The text header may describe one or more attributes or parameters associated with the media item of the corresponding group. The text header may be in a natural language form. In addition, each text header may be an exemplary user utterance, which, when provided to a digital assistant by a user, may enable the digital assistant to obtain media items of a similar group with similar attributes.

[0271] As described above, the second user utterance "Luke Wilson" can be associated with two possible user intentions: the first user intention of performing a new media search or the second user intention of refining the results of a previous user request. The displayed media item 670 can satisfy the first user intention, and the displayed media item 674 can satisfy the second user intention. In this example, media items 670 and media items 674 are displayed in the first two rows. In this way, the results of the two most likely user intentions associated with the second user request (e.g., a new search or a refinement of a previous search) can be highlighted (e.g., the first two rows) in the third user interface 626. This can minimize the scrolling or browsing of the user in the third user interface before finding the required media items for consumption. It should be recognized that the way to highlight media items 670 and media items 674 in the third user interface 626 to minimize scrolling and browsing can be varied.

[0272] 7A to 7C A process 700 for operating a digital assistant for a media system according to various examples is shown. The process 700 can be performed using one or more electronic devices implementing a digital assistant. For example, the process 700 can be performed using one or more of the system 100, media system 128, media device 104, user device 122\, or digital assistant system 400 described above. FIG. 8A to FIG. 8W 700 according to various examples. 7A to 7C and FIG. 8A to FIG. 8W A description is given of process 700. It should be understood that some operations in process 700 may be combined, the order of some operations may be changed, and some operations may be omitted.

[0273] At block 702 of process 700, content may be displayed on a display unit (eg, display unit 126). Block 702 may be similar or identical to block 502 described above. Fig. 8A , the displayed content may include media content 802 (e.g., movies, videos, television programs, video games, etc.) playing on a media device (e.g., media device 104). In other examples, the displayed content may include other content, such as content associated with an application running on the media device, or a user interface for interacting with a digital assistant of the media device. Specifically, the displayed content may include a main menu user interface or a user interface containing objects or results previously requested by the user.

[0274] At block 704 of process 700, user input may be detected. Block 704 may be similar or identical to block 504 described above. The user input may be used to invoke a digital assistant of the media device. In some examples, the user input may be detected while the contents of block 702 are displayed. The user input may be detected on a remote control of the media device (e.g., remote control 124). For example, the user input may correspond to the second input type described in block 516 of process 500. Specifically, the user input at block 704 may include pressing a specific button on the remote control of the media device and holding the button for a predetermined duration (e.g., a long press). In response to detecting the user input, one or more of blocks 706 to 746 may be executed.

[0275] At block 706 of process 700, audio data may be sampled. Block 706 may be similar or identical to block 518 described above. The sampled audio data may include user utterances. The user utterances may represent user requests to a digital assistant of a media device. For example, referring to Fig. 8A In the example shown in , the sampled audio data may include the user utterance "What time is it in Paris?". The user utterance may be in an unstructured natural language form. In some examples, the request represented by the user utterance may be partially specified, where the information required to perform the request is missing or not clearly defined in the user utterance (e.g., "Play this"). In other examples, the user utterance may not be an explicit request, but an indirect question or statement from which the request is inferred (e.g., "What did he say?"). In addition, as described in more detail below in box 712, the user utterance may include one or more ambiguous terms.

[0276] At block 708 of process 700, a text representation of the user utterance in the sampled audio data may be determined. Block 708 may be similar or identical to block 522 described above. Specifically, the text representation may be determined by performing STT processing on the user utterance in the sampled audio data. For example, referring to Fig. 8A, a text representation 804 "What time is it in Paris?" can be determined from the user's speech in the sampled audio data and displayed on the display unit. As shown, the text representation 804 can be overlaid on the media content 802, and the media content 802 continues to play on the media device.

[0277] In some examples, the STT processing used to determine the text representation may be biased toward media-related text results. In addition or alternatively, the text representation may be based on a previous user utterance received by the media device before the audio data is sampled. Furthermore, in some examples, the text representation may be based on the time at which the previous user utterance was received before the audio data was sampled. In an example of obtaining a text representation from an independent device (e.g., DA server 106), the media device may indicate to the independent device that the sampled audio data is associated with a media application, and the indication may bias the STT processing on the independent device toward media-related text results.

[0278] At block 710 of process 700, a user intent corresponding to the user utterance may be determined. Block 710 may be similar to block 526 described above. Specifically, the text representation of block 708 may be processed using natural language processing (e.g., using natural language processing module 432) to obtain the user intent. For example, referring to Fig. 8A , it can be determined from the text representation 804 "What time is it in Paris?" that the user intent is to request a time for a location named "Paris". Natural language processing used to determine user intent can be biased towards media-related user intent. In an example of obtaining user intent from an independent device (e.g., DA server 106), the media device can indicate to the independent device that the sampled audio data is associated with a media application, and this indication can bias natural language processing on the independent device towards media-related user intent.

[0279] In some examples, user intent may be determined based on prosodic information derived from user speech in sampled audio data. Specifically, prosodic information (e.g., pitch, rhythm, volume, pressure, intonation, speed, etc.) may be derived from user speech to determine the user's attitude, mood, emotion, or mood. The user intent may then be determined based on the user's attitude, mood, emotion, or mood. For example, the sampled audio data may include the user speech "What did he say?" In this example, the user's impatience or frustration may be determined based on the high volume and pressure detected in the user speech. Based on the user speech and the determined user emotion, it may be determined that the user intent includes a request to increase the volume of audio associated with media content being played on a media device.

[0280] like Fig. 7AAs shown, block 710 may include one or more of blocks 712 to 718. Specifically, when two or more user intents are found to be highly probable and the natural language processing module cannot narrow the scope of the two or more user intents into a single user intent, one or more of blocks 712 to 718 may be performed. For example, this may occur when the user utterance contains ambiguous terms that cannot be disambiguated based on available contextual information.

[0281] At block 712 of process 700, it may be determined whether the user utterance (or a textual representation of the user utterance) includes an ambiguous term. The determination may be made during natural language processing (e.g., using natural language processing module 432) to determine the user's intent. An ambiguous term may be a word or phrase that has more than one possible interpretation. For example, referring to Fig. 8A , the term "Paris" in the user utterance "What time is it in Paris?" can be interpreted as "Paris" in France or "Paris" in Texas, U.S. Therefore, the term "Paris" in the user utterance can be determined to be a fuzzy term.

[0282] In some examples, contextual information may be retrieved (e.g., by a digital assistant) to potentially disambiguate the ambiguous term. If disambiguation is successful, it may be determined that the user utterance does not include an ambiguous term. For example, it may be determined that the media content 802 is a movie (e.g., "Ratatouille") that uses "Paris" in France as a setting, and therefore the user is more likely to be referring to "Paris" in France than to "Paris" in Texas. In this example, the term "Paris" may be successfully disambiguated to indicate "Paris" in France, and therefore it may be determined that the user utterance does not include an ambiguous term.

[0283] In another example, the user utterance may be "Play this." In this example, the user utterance does not clearly define the specific media item to be played, so the term "this" interpreted separately may be a vague term that can refer to any media item accessible to the media device. The term may be disambiguated using contextual information displayed by the media device on the display unit. For example, the digital assistant may determine whether the focus of the displayed user interface is on a media item. Based on determining that the focus of the user interface is on the media item, the digital assistant may disambiguate the term "this" and determine that the term refers to the media item focused on by the displayed user interface. Based on this determination, it may be determined at box 712 that the user utterance does not include a vague term. Therefore, the user intent may be determined as a request to play the media item focused on by the displayed user interface.

[0284] In an example where the term cannot be disambiguated, it may be determined at block 712 that the user utterance includes an ambiguous term. In response to determining that the user utterance includes an ambiguous term, one or more of blocks 714 to 718 may be performed. At block 714 of process 700, two or more candidate user intents may be obtained based on the ambiguous term. The two or more candidate user intents may be the most likely candidate user intents determined from the user utterance that cannot be disambiguated. Fig. 8A In the illustrated example, the two or more candidate user intents may include a first candidate user intent to request the time in “Paris”, France, and a second candidate user intent to request the time in “Paris”, Texas.

[0285] At block 716 of process 700, two or more candidate user intents may be displayed on a display unit for user selection. Figure 8B , a first candidate user intent 810 and a second candidate user intent 808 may be displayed. In addition, a text prompt 806 may be provided to prompt the user to indicate the actual user intent corresponding to the user utterance by selecting between the first candidate user intent 810 and the second candidate user intent 808. The text prompt 806, the first candidate user intent 810, and the second candidate user intent 808 are overlaid on the media content 802.

[0286] At block 716 of process 700, a user selection of one of the two or more candidate user intents may be received. In some examples, the user selection may be received by selecting an affordance corresponding to one of the candidate user intents. Specifically, Figure 8B As shown, each of the two or more candidate user intents 810, 808 can be displayed as a selectable affordance on the display unit. The media device can receive input from the user (e.g., via a remote control of the media device) to change the focus of the display to one of the affordances. A user selection of the candidate user intent corresponding to the affordance can then be received (e.g., via a remote control of the media device). For example, Figure 8B As shown, the media device can receive user input to move a cursor 812 over an affordance corresponding to a first candidate user intent 810 (eg, "Paris" for France). A user selection of the first candidate user intent 810 can then be received.

[0287] In other examples, the user selection may be received via voice interaction with the digital assistant. For example, when two or more candidate user intents are displayed, a second user input may be detected. The second user input may be similar or identical to the user input of box 704. Specifically, the second user input may be an input to invoke the digital assistant (e.g., pressing a specific button on a remote control of a media device and holding the button for a predetermined duration). In response to detecting the second user input, second audio data may be sampled. The second audio data may include a second user utterance representing a user selection of one of the two or more interpretations. For example, referring to Figure 8C , the second audio data may include a second user utterance "'Paris' in France". As shown, a text representation 814 of the second user utterance "'Paris' in France" may be displayed on the display unit. In this example, the second user utterance "'Paris' in France" may represent a user selection of the first candidate user intent 810 (e.g., "Paris" in France). Based on the second user utterance "'Paris' in France", it may be determined that the first candidate user intent 810 is the actual user intent corresponding to the user utterance "What time is it in Paris?" Therefore, it may be determined at box 710 that the user intent is to request the time of "Paris" in France. When determining the user intent based on the received user selection, one or more of boxes 720 to box 746 may be executed.

[0288] In some examples, blocks 710 to 718 may be performed without outputting speech from the media device. Specifically, the text prompt 806 and the candidate user intents 808, 810 may be displayed without outputting speech associated with two or more candidate user intents 808, 810. Thus, input may be received from the user in the form of speech, but output from the digital assistant may be presented to the user visually (rather than in the form of audio) on the display unit. This may be desirable to maintain a common experience associated with consuming media content, thereby improving the user experience of the media device.

[0289] Referring again to box 712, in response to determining that the user utterance does not include an ambiguous term, one or more of boxes 720 to 746 may be executed. At box 720 of process 700, it may be determined whether the user intent corresponds to a predetermined core competency in a plurality of core competencies associated with a media device. For example, a media device may be associated with several predetermined core competencies, such as, for example, searching for media items, playing media items, and providing information related to media items, weather, stock market, and sports. If the user intent involves performing a task related to a predetermined core competency in a plurality of predetermined core competencies, it may be determined that the user intent corresponds to a predetermined core competency in a plurality of predetermined core competencies. For example, if the user intent is a request for a media item starring Reese Witherspoon, it may be determined that the user intent corresponds to a predetermined core competency in a plurality of predetermined core competencies. In response to determining that the user intent corresponds to a core competency in a plurality of core competencies associated with an electronic device, one or more of boxes 724 to 746 may be executed.

[0290] In contrast, if the user intent involves performing a task outside of the plurality of predetermined core competencies, it may be determined that the user intent does not correspond to one of the plurality of predetermined core competencies. For example, if the user intent is a request for map directions, it may be determined that the user intent does not correspond to one of the plurality of predetermined core competencies. In response to determining that the user intent does not correspond to one of the plurality of core competencies associated with the electronic device, block 722 may be executed.

[0291] At box 722 of process 700, a second electronic device (e.g., device 122) may be caused to at least partially satisfy the user intent. Specifically, the second electronic device may be caused to perform a task that facilitates satisfying the user intent. In one example, it may be determined that the media device is not configured to satisfy the user intent of requesting map directions, and therefore the user intent may be transmitted to the second electronic device to satisfy the user intent. In this example, the second user device may perform a task for displaying the requested map directions. In other examples, information other than the user intent may be transmitted to the second electronic device so that the second electronic device performs a task for facilitating satisfying the user intent. For example, a digital assistant of a media device may determine (e.g., using a natural language processing module 432 or a task flow processing module 436) a task flow or structured query that satisfies the user intent, and the task flow or structured query may be transmitted to the second electronic device. The second electronic device may then perform the task flow or structured query to facilitate satisfying the user intent.

[0292] As will become apparent in the description provided below, the level of interference associated with satisfying a user's intent may be based on the nature of the user's intent. In some cases, a task associated with satisfying the user's intent may be performed without displaying any additional response or output on the display (e.g., box 726). In other cases, only a text response (e.g., without a corresponding visual or audio output) is provided to satisfy the user's intent (e.g., box 732). In other cases, a user interface containing relevant results may be displayed to satisfy the user's intent (e.g., boxes 738, 742, or 746). The user interface may occupy most or a small portion of the area of ​​the display unit. Therefore, process 700 may intelligently adjust the level of interference output according to the nature of the user's intent. This enables convenient access to the services of the digital assistant while reducing undesirable interference during consumption of media content, thereby improving the overall user experience.

[0293] At block 724 of process 700, it may be determined whether the user intent includes a request to adjust a state or setting of an application on the media device. In response to determining that the user intent includes a request to adjust a state or setting of an application on the media device, block 726 may be performed. At block 726 of process 700, the state or setting of the application may be adjusted to satisfy the user intent.

[0294] In some examples, the state or setting may be associated with the displayed media content being played on the media device. For example, the request for adjusting the state or setting of the application may include a request for controlling the media device to play the media content. Specifically, it may include a request for pausing, resuming, restarting, stopping, rewinding or fast-forwarding the playback of the displayed media content on the media device. It may also include a request for skipping forward or backward (for example, with a specified duration) in the media content to play the required part of the media content. In addition, the request for adjusting the state or setting of the application may include a request for turning on / off the subtitles or closed captions (for example, in a specified language) associated with the displayed media content, a request for increasing / reducing the volume of the audio associated with the displayed media content, a request for muting / unmuting the audio associated with the displayed media content, or a request for speeding up / slowing down the rate of the displayed media content playback.

[0295] FIG. 8E to FIG. 8F An exemplary example of a user intent including a request to control a media device to play media content is shown. In this example, a digital assistant may be invoked when playing media content 802 (e.g., at block 704). The media content may be displayed without initially displaying subtitles. The sampled audio data (e.g., at block 706) may include the user utterance "Turn on English subtitles." As shown in FIG. Fig. 8EAs shown, a text representation 816 of the user utterance may be displayed on the display unit. Based on the user utterance, it may be determined at block 710 that the user intent includes a request to turn on display of English subtitles for media content 802. In addition, at block 724, it may be determined that the user intent is a request to adjust a state or setting of an application of the electronic device. In response to this determination, English subtitles for media content 802 may be turned on. Fig.8F As shown in the tag 817 in , the display of English subtitles associated with the media content 802 can be initiated to meet the user's intention.

[0296] exist Figures 8G to 8H In another exemplary example shown in , the user speech in the sampled audio data can be a natural language expression, thereby indicating that the user did not hear a portion of the audio associated with the media content. Specifically, as Figure 8G As shown in the text representation 820 in , the user utterance may be “What did he say?”. In this example, it may be determined (e.g., at box 710) that the user intent includes a request to replay a portion of the media content corresponding to a portion of the audio that the user did not hear. It may also be determined that the user intent includes a request to turn on closed captions to assist with audio associated with the media content that is difficult to hear. In addition, based on prosodic information in the user utterance, it may be determined that the user is frustrated or impatient, and thus, based on the user's emotions, it may be determined that the user intent includes a request to increase the volume of the audio associated with the media content. At box 724, it may be determined that these user intents are requests to adjust the state or settings of an application of the electronic device. In response to this determination, the media content may be rewound for a predetermined duration (e.g., 15 seconds) to a previous portion of the media content, and the previous portion (e.g., Figure 8H In addition, before resuming playback of the media content from the previous portion, closed captions (e.g., Figure 8H 824 in FIG. 1 ). Additionally, the volume of the audio associated with the media content may be increased before resuming playback of the media content from a previous portion.

[0297] Should be understood that closed captions or subtitles associated with media content can be obtained from service providers (e.g., cable providers or media subscription services). However, in the example of closed captions or subtitles that cannot be obtained from service providers, media devices can generate closed captions or subtitles to assist the audio associated with media content that is difficult to hear. For example, before receiving user speech in sampled audio data and when playing media content, the voice in the audio associated with media content (e.g., using STT processing module 730) can be converted into text continuously and stored in association with media content. In response to a user request for a previous portion of media content that is not heard by the playback user, the text corresponding to the previous portion being played back can be retrieved and displayed when the previous portion of the playback media content is played back.

[0298] In some examples, a state or setting associated with the displayed media content may be adjusted without displaying an additional user interface for performing the adjustment, or without providing any text or graphics to indicate confirmation that the state or setting is being adjusted. FIG. 8E to FIG. 8H In the example shown, subtitles (or closed captions) may be simply turned on without explicitly displaying text such as "Subtitles turned on," or without displaying a user interface for controlling the display of subtitles. Additionally, a state or setting may be adjusted without outputting any audio associated with satisfying the user's intent. For example, in FIG. 8E to FIG. 8H In the process 700, subtitles (or closed captions) can be turned on without outputting audio (e.g., a voice signal or a non-verbal audio signal) for confirming that the subtitles are turned on. Therefore, the requested action can be simply performed without causing additional audio or visual interference with the media content. In this way, process 700 can minimize interference with the user's consumption of media content while providing convenient access to digital assistant services, thereby improving the user experience.

[0299] In other examples, the request to adjust the state or settings of an application on the media device may include a request to navigate in a user interface of the media device (e.g., second user interface 818, third user interface 826, or main menu user interface). In one example, the request to navigate in the user interface may include a request to switch the focus of the user interface from a first object (e.g., a first media item) to a second object (e.g., a second media item) in the user interface. FIG. 8I to FIG. 8K An illustrative example of such a request is shown. Fig.8IAs shown, the displayed content may include a third user interface 826 having a plurality of media items organized into various categories (e.g., “Romantic Comedies,” “Romantic Comedies Starring Reese Witherspoon,” and “Movies Starring Luke Wilson”). As shown by the position of cursor 828, the focus of the third user interface 826 may be on a first media item 830 categorized as “Romantic Comedies.” A second media item 832 may be titled “Legally Blonde” and may be located in the category of “Romantic Comedies Starring Reese Witherspoon.” As shown in FIG. Figure 8J As shown in the text representation 834 in , the user utterance in the sampled audio data (e.g., at box 706) can be "Go to Legally Blonde." Based on the user utterance, it can be determined (e.g., at box 710) that the user intent is a request to switch the focus of the third user interface 826 from the first media item 830 to the second media item 832 titled "Legally Blonde". In response to determining (e.g., at box 724) that the user intent is a request to adjust the state or setting of an application of the electronic device, the focus of the third user interface 826 can be switched from the first media item 830 to the second media item 832. For example, Figure 8K As shown, the position of the cursor 828 may change from the first media item 830 to the second media item 832 .

[0300] In another example, a request to navigate in a user interface may include a request to change the focus of the user interface to a particular category of results displayed in the user interface. Fig.8I Included are media items associated with categories such as “romantic comedies,” “romantic comedies starring Reese Witherspoon,” and “movies starring Luke Wilson.” Instead of “go to Legally Blonde,” the user utterance in the sampled audio data may be “jump to romantic comedies starring Reese Witherspoon.” Based on the user utterance, it may be determined (e.g., at box 710) that “romantic comedies starring Reese Witherspoon” defines a category of media items displayed in the third user interface 826, and therefore it may be determined that the user intent is a request to change the focus of the user interface to one or more media items associated with the category. In response to determining (e.g., at box 724) that the user intent is a request to adjust the state or setting of an application of the electronic device, the focus of the third user interface 826 may be shifted to one or more media items associated with the category. For example, if Figure 8KAs shown, the position of cursor 828 may be transferred to a second media item 832 associated with "Romantic Comedy Starring Reese Witherspoon."

[0301] In other examples, a request to navigate in a user interface of a media device may include a request to select an object in the user interface. Selection of an object may cause an action associated with the object to be performed. For example, Figure 8K As shown, the cursor 828 is located on the second media item 832 titled "Legally Blonde". Figure 8L As shown, a digital assistant may be called (e.g., at box 704), and the user speech in the sampled audio data (e.g., at box 706) may be "play this" (e.g., displayed as a text representation 836). Based on the user speech, it may be determined (e.g., at box 710) that the user intent is a request for playing a specific media item. In this example, the user speech does not clearly define or identify the specific media item to be played. Specifically, the word "this" is vague. However, the digital assistant may obtain contextual information to disambiguate the user intent. For example, it may be determined that the focus of the third user interface 826 is on the second media item 832 when the audio data is sampled. Based on this determination, the second media item 832 may be identified as a media item to be played. In response to (e.g., at box 724) determining that the user intent to play the second media item 832 is a request for adjusting the state or settings of an application of an electronic device, an action for facilitating the playing of the second media item 832 may be performed. For example, preview information about the second media item 832 may be displayed on a display unit. The preview information may include, for example, a brief plot summary, cast list, release date, user ratings, etc. Additionally or alternatively, the second media item 832 may be played on a media device, and the media content associated with the second media item 832 may be displayed on a display unit (e.g., Figure 8M 838 "Playing Legally Blonde"). It should be appreciated that in other examples, the media item to be selected may be explicitly identified. For example, instead of "play this", the user utterance may specifically state "play Legally Blonde", and a similar action for causing the second media item 832 to be played may be performed.

[0302] In other examples, a request for navigating in a user interface of a media device may include a request for viewing a specific user interface or application of the media device. For example, the user utterance in the sampled audio data may be "go to the actor page", where the user intent includes a request to display a user interface associated with browsing media items according to a specific actor. In another example, the user utterance in the sampled audio data may be "return to home page", where the user intent includes a request to display the main menu user interface of the media device. In yet another example, a request for navigating in a user interface of a media device may include a request to launch an application on an electronic device. For example, the user utterance in the sampled audio data may be "go to the iTunes Store", where the user intent includes a request to launch the iTunes Store application. It should be recognized that other requests for adjusting the state or settings of an application on a media device may be expected.

[0303] Referring again to box 724, it may be determined that the user intent does not include a request for adjusting the state or settings of an application on an electronic device. For example, the user intent may be replaced by a request for presenting information related to one or more media items. In response to such a determination, one or more of boxes 728 to 746 may be executed. At box 728 of process 700, it may be determined whether the user intent is a predetermined request type in a plurality of predetermined request types. In some examples, the plurality of predetermined request types may be requests associated with plain text responses. More specifically, the plurality of predetermined request types may be requests for information that is predetermined to require a plain text response. This is in contrast to requests for responses that are predetermined to require a media object (e.g., an image, an animated object, a video, etc.). In some examples, the plurality of predetermined request types may include a request for the current time at a particular location (e.g., "What time is it in Paris?"), a request to present a joke (e.g., "Tell me a funny joke."), or a request for information about the media content currently playing on the electronic device (e.g., "When was this movie released?"). In response to determining that the user intent is a predetermined request type among a plurality of predetermined request types, one or more of blocks 730 through 732 may be performed.

[0304] At block 730 of process 700, a result that at least partially satisfies the user's intent may be obtained. For example, the result may be obtained from an external service (e.g., external service 120) by executing a task flow. At block 732 of process 700, the result obtained at block 730 may be displayed on a display unit in text form. In addition, the result may be displayed in text form without displaying any corresponding graphics or media-related items corresponding to the result.

[0305] Figure 8M to Figure 8P An illustrative example of blocks 728 to 732 is shown. Figure 8M As shown, the movie "Legally Blonde" may initially be played on the media device and displayed on the display unit. While "Legally Blonde" is playing, the digital assistant may be invoked (e.g., at block 704), and the user utterance in the sampled audio data may be "Who is the female lead?" For example, Figure 8N As shown, a text representation 840 of the user utterance may be displayed on a display unit. Based on the user utterance, it may be determined (e.g., at box 710) that the user intent includes a request to identify the leading actress of a particular media item. Since the user utterance does not specify any particular media item, the user intent may be ambiguous. However, based on the movie "Legally Blonde" displayed when the audio data is sampled, it may be determined that the media item associated with the user intent is "Legally Blonde." In this example, it may be determined (e.g., at box 728) that the user intent is one of a plurality of predetermined request types. Specifically, it may be determined that a plain text response may be provided to satisfy the user intent of identifying the leading actress in Legally Blonde. In response to determining that the user intent is one of a plurality of predetermined request types, a search may be performed (e.g., at box 730) in a media-related database to obtain a result that the leading actress in the movie "Legally Blonde" is "Reese Witherspoon." As Figure 8P As shown, the plain text result 842 "Reese Witherspoon" may be displayed on the display unit to satisfy the user's intent. The plain text result 842 may be overlaid on the displayed media content of "Legally Blonde". In addition, when the plain text result 842 is displayed, the media content of "Legally Blonde" may continue to play. By displaying the plain text result 842 (for example, without displaying a graphical result or an additional user interface to satisfy the user's intent), the user's intent may be satisfied in an unobtrusive manner and minimize interference with the user's consumption of media content. At the same time, the user is provided with access to a digital assistant service. This is desirable for improving the user experience.

[0306] Referring again to box 728, it can be determined that the user intent is not a predetermined request type among multiple predetermined request types. Specifically, the user intent can be a request type that is predetermined to require more than text results to satisfy. For example, the user intent can be a request for executing a media search query and displaying media items corresponding to the media search query. In other examples, the user intent can be a request for information other than media items. For example, the user intent can be a request for information associated with the following items, including: sports teams (e.g., "How did the LALakers perform in their last game?"), athletes (e.g., "How tall is LeBron James?"), stock markets (e.g., "What was the closing price of the Dow Jones Index yesterday?"), or weather (e.g., "What is the weather forecast for 'Paris', France next week?"). In response to determining that the user intent is not a predetermined request type among multiple predetermined request types, one or more boxes in boxes 734 to 746 can be executed.

[0307] At block 734 of process 700, a second result that at least partially satisfies the user intent may be obtained. Block 734 may be similar or identical to block 534 described above. In one example, the user intent may include a request to perform a media search query. In this example, the media search query may be performed at block 734 to obtain a second result. Specifically, the second result may include media items corresponding to the media search query.

[0308] In some examples, the user intent may not be a media search query. For example, the user intent may be a request for providing a weather forecast for "Paris", France (e.g., what is the weather forecast for "Paris", France?). In this example, the second result obtained at box 734 may include a weather forecast for "Paris", France for the next 7 days. The second result may include non-media data that at least partially satisfies the user intent. Specifically, the weather forecast for "Paris", France for the next 7 days may include text data (e.g., a brief description of the date, temperature, and weather conditions) and a graphic image (e.g., an image of a sunny, cloudy, windy, or rainy day). In addition, in some examples, the scope of the user intent may be expanded at box 710 to include a request for a media item that at least partially satisfies the user intent. In these examples, the second result obtained at box 734 may also include one or more media items with media content that at least partially satisfies the user intent. For example, a media search query may be performed at box 734 for a weather forecast for "Paris", France, during a relevant time period, and one or more media items related to the weather forecast for "Paris", France, may be obtained. The one or more media items may include, for example, a video clip from a weather channel presenting a weather forecast for “Paris,” France. In these examples, the non-media data and / or the one or more media items may be displayed in a user interface on the displayed unit (e.g., at blocks 738, 742, or 746 described below).

[0309] At box 736 of process 700, it may be determined whether the displayed content includes media content played on the electronic device. In some examples, it may be determined that the displayed content does not include media content played on the electronic device. For example, the displayed content may alternatively include a user interface, such as a main menu user interface or a third user interface (e.g., third user interface 826). The third user interface may occupy at least a majority of the display area of ​​the display unit. In addition, the third user interface may include previous results related to previous user requests received before detecting user input at box 704. Based on determining that the displayed content does not include media content, box 738 may be executed.

[0310] At block 738 of process 700, a portion of the second result may be displayed in a third user interface on the display unit. In examples where the displayed content already includes a third user interface when the user input at block 704 is received, the display of a portion of the second result in the third user interface may be used to replace the display of a previous result associated with a previous user request. In examples where the displayed content does not include a third user interface when the user input at block 704 is received (e.g., the displayed content includes a main menu user interface), the third user interface may be displayed and the second result may be included in the displayed third user interface.

[0311] In some examples, it may be determined whether the second result includes a result of a predetermined type. A result of a predetermined type may be associated with a small portion of the display area of ​​the display unit. A result of a predetermined type may include, for example, a result related to the stock market or the weather. It should be appreciated that in other examples, the result of the predetermined type may vary. In response to determining that the second result includes a result of a predetermined type, a portion of the second result may be displayed in a second user interface on the display unit. The second user interface may occupy a small portion of the display area of ​​the display unit. In these examples, although it is determined at box 736 that the displayed content does not include media content, a portion of the second result may be displayed in the second user interface.

[0312] Figures 8Q to 8S An illustrative example of blocks 734 to 738 is shown. In this example, Figure 8Q As shown, the displayed content may initially include a third user interface 826. The third user interface 826 may include previous results from a previous user request. Specifically, the third user interface 826 includes media items 844 from a previously requested media search query. Figure 8R As shown, when the third user interface 826 is displayed, the digital assistant may be called (e.g., at box 704). The user utterance in the sampled audio data may include "Show me movies starring Luke Wilson." The text representation 846 of the user utterance may be displayed on the display unit. In this example, it may be determined (e.g., at box 710) that the user intent is a request to perform a media search query for movies starring Luke Wilson. The media search query may be performed (e.g., at box 734) to obtain a second result. Specifically, the second result may include a media item 848 corresponding to a movie starring Luke Wilson. In addition, additional results (e.g., media item 850) related to the user intent or a previous user intent may be obtained. These additional results may be obtained in a manner similar to the manner in which the second result is obtained as described in box 544.

[0313] exist Figures 8Q to 8S In this example, the displayed content includes only the third user interface 826, so it can be determined (e.g., at block 736) that the displayed content does not include media content played on the electronic device. In response to this determination, the second result can be displayed in the third user interface 826. Specifically, Figure 8S As shown, the display of media item 844 in the third user interface 826 can be replaced by the display of media item 848 in the third user interface 826. In addition, media item 850 can be displayed in the third user interface 826.

[0314] As shown in this example, the second result can be presented in the third user interface only after it is determined that the media content is not displayed on the display unit. This allows a wider range of results to be displayed in a larger area, thereby increasing the probability of satisfying the user's actual intention. At the same time, by ensuring that there is no media content being displayed on the display unit before presenting the second result in the third user interface, interference with the user's consumption of media content is avoided.

[0315] Referring again to block 736, the displayed content may include media content being played on a media device. In these examples, it may be determined that the displayed content includes media content being played on a media device. Based on this determination, one or more of blocks 740 to 746 may be executed.

[0316] At block 740 of process 700, it may be determined whether the media content being played can be paused. Examples of media content that can be paused may include on-demand media items, such as on-demand movies and television programs. Examples of media content that cannot be paused may include media programs of broadcast or streaming services and live media programs (e.g., sports events, concerts, etc.). Therefore, on-demand media items may not include broadcast or live programs. Based on determining at block 740 that the media content being played cannot be paused, block 742 may be executed. At block 742 of process 700, a second user interface having a portion of the second result may be displayed on a display unit. Block 742 may be similar to block 536 described above. A second user interface may be displayed when the media content is displayed. The display area occupied by the second user interface on the display unit may be smaller than the display area occupied by the media content on the display unit. Based on determining that the media content being played can be paused, one or more of blocks 744 to 746 may be executed. At block 744 of process 700, the media content being played may be paused on a media device. At block 746 of process 700, a third user interface can be displayed with a portion of the second result. The third user interface can be displayed while the media content is paused.

[0317] Figures 8T to 8W An illustrative example of blocks 740 to 746 is shown. Figure 8TAs shown, media content 802 played on a media device can be displayed on a display unit. While the media content 802 is being displayed, a digital assistant can be activated (e.g., at box 704). The user utterance in the sampled audio data can be "Show me movies starring Luke Wilson." A text c representation 846 of the user utterance can be displayed on the display unit. As described above, it can be determined (e.g., at box 710) that the user intent is a request to obtain a media item for a movie starring Luke Wilson. A corresponding media search query can be executed (e.g., at box 734) to obtain a second result. The second result may include a media item 848 that is a movie starring Luke Wilson. In an example where it is determined (e.g., at box 744) that the media content 802 cannot be paused, the media item 848 can be displayed in the second user interface 818 while the media content 802 continues to be displayed on the display unit (e.g., Figure 8U ). It is desirable to display the media item 848 in the second user interface 818 so that the media content 802 can continue to be available for user consumption while the media item 848 is displayed, thereby satisfying the user's intent. This prevents the user from missing any portion of the media content 802 that cannot be paused or replayed. Alternatively, in an example where it is determined (e.g., at block 744) that the media content 802 can be paused, playback of the media content 802 on the media device can be paused, and the media item 848 can be displayed in the third user interface 826 on the display unit (e.g., Figure 8S ). It is desirable to display the third user interface 826 so that a wider range of media items associated with various alternative user intentions (e.g., media item 850) can be displayed together with the requested media item (e.g., media item 848), thereby increasing the possibility of satisfying the actual intention of the user. At the same time, the media content 802 is paused so that the user does not miss any part of the media content 802. By changing the user interface for displaying media items 848 based on whether the media content 802 can be paused, the user intention associated with the user's speech can be fully realized while reducing interference with the user's consumption of the media content 802. This can improve the overall user experience.

[0318] In some examples, such as Figure 8V As shown, in addition to the media content 802 playing on the media device, the displayed content may also include a second user interface 818. In these examples, the second user interface 818 may include media items 852 related to a previous user request (e.g., a request for a romantic comedy starring Reese Witherspoon). While displaying the media content 802 and the second user interface 818, the digital assistant may be invoked (e.g., at block 704). Figure 8WAs shown, the sampled audio data may include the user utterance "Show me movies starring Luke Wilson." A text representation 846 of the user utterance may be displayed on the display unit. Based on the user utterance, it may be determined (e.g., at box 710) that the user intent is a request to obtain media items that are movies starring Luke Wilson. A corresponding media search query may be executed (e.g., at box 734) to obtain a second result (e.g., media item 848). In these examples, the display of media item 852 in the second user interface 818 may be replaced with the display of media item 848 (e.g., Figure 8U ).

[0319] Fig. 9 A process 900 for interacting with a digital assistant of a media system according to various examples is shown. The process 900 can be performed using one or more electronic devices implementing a digital assistant. For example, the process 900 can be performed using one or more of the system 100, the media system 128, the media device 104, the user device 122, or the digital assistant system 400 described above. It should be understood that some operations in the process 900 can be combined, the order of some operations can be changed, and some operations can be omitted.

[0320] At block 902 of process 900, content may be displayed on a display unit. Block 902 may be similar or identical to block 502 described above. In some examples, the displayed content may include media content (e.g., movies, videos, television programs, video games, etc.). Additionally or alternatively, the displayed content may include a user interface. For example, the displayed content may include a user interface having one or more exemplary natural language requests (e.g., FIG. 6D to FIG. 6E In other examples, the displayed content may include a third user interface (e.g., third user interface 626) with results from a previous user request (e.g., a previously requested media item). The third user interface may occupy at least a majority of the display area of ​​the display unit.

[0321] At block 904 of process 900, a user input may be detected while displaying the contents of block 902. The user input may be similar or identical to the fifth user input described at block 558. Specifically, the user input may be detected on a remote control of the media device. For example, the user input may include a predetermined action pattern on a touch-sensitive surface of the remote control device. In some examples, the user input may be detected via a second electronic device (e.g., device 122) that is different from the media device. The second electronic device may be configured to control the media device wirelessly. In response to detecting the user input, one or more of blocks 906 to 914 may be executed.

[0322] At block 906 of process 900, a virtual keyboard interface (e.g., virtual keyboard interface 646) may be displayed on the display unit. Block 906 may be similar or identical to block 562 described above. The virtual keyboard interface may be overlaid on at least a portion of the first user interface or the third user interface. Additionally, a search field (e.g., search field 644) may be displayed on the display unit. The virtual keyboard interface may be configured such that user input received via the virtual keyboard interface results in text entry in the search field.

[0323] At box 908 of process 900, a selectable affordance may be displayed on a second electronic device (e.g., displayed on touch screen 346 of device 122). The second electronic device may be a device different from a remote control for the media device. Selection of the affordance may enable text input to be received by the media device via a keyboard of the second electronic device. For example, selection of the affordance may cause a virtual keyboard interface (e.g., similar to virtual keyboard interface 646) to be displayed on the second electronic device. Input to the virtual keyboard interface of the second electronic device may cause corresponding text to be entered in a search field (e.g., search field 644).

[0324] At block 910 of process 900, text input may be received via a keyboard (e.g., a virtual keyboard interface) of a second electronic device. Specifically, a user may enter text via the keyboard of the second electronic device, and the text input may be transmitted to and received by the media device. The text input may represent a user request. For example, the text input may be "Jurassic Park," which may represent a request to perform a search for media items associated with the search string "Jurassic Park."

[0325] At block 912 of process 900, results that at least partially satisfy the user request may be obtained. For example, a media search may be performed using the text input, and corresponding media items may be obtained. In a specific example where the text input is "Jurassic Park," media items titled "Jurassic Park" or having the same actors or directors as the movie "Jurassic Park" may be obtained. In another example where the text input is "Reese Witherspoon," media items having the actress Reese Witherspoon in them may be obtained.

[0326] At block 914 of process 900, a user interface can be displayed on a display unit. The user interface can include at least a portion of the results. For example, the user interface can include media items obtained as a result of the media search performed at block 912.

[0327] Although certain blocks of process 500, process 700, and process 900 are described above as being performed by a device or system (e.g., media device 104, user device 122, or digital assistant system 400), it should be appreciated that in some examples, more than one device may be used to perform the blocks. For example, in a block where a determination is made, a first device (e.g., media device 104) may obtain a determination from a second device (e.g., server system 108). Similarly, in a block where content, an object, text, or a user interface is displayed, a first device (e.g., media device 104) may cause the content, object, text, or user interface to be displayed on a second device (e.g., display unit 126).

[0328] 5. Electronic devices

[0329] According to some examples, Fig.10 1 shows a functional block diagram of an electronic device 1000 that is configured according to the principles of the various examples described, such as to voice control media playback and update the knowledge of a virtual assistant in real time. The functional blocks of the device can be implemented by hardware, software, or a combination of hardware and software that implements the principles of the various examples described. It should be understood by those skilled in the art that Fig.10 The functional blocks described in the invention may be combined or separated into sub-blocks to implement the principles of various described examples. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0330] like Fig.10 As shown, the electronic device 1000 may include: an input unit 1003 (e.g., remote control 124, etc.) configured to receive user input such as tactile input, gesture input; an audio input unit 1004 (e.g., microphone 272, etc.) configured to receive audio data; a speaker unit 106 (e.g., speaker 268, etc.) configured to output audio; and a communication unit 1007 (e.g., communication subsystem 224, etc.) configured to send and receive information from an external device via a network. In some examples, the electronic device 1000 may optionally include a display unit 1002 (e.g., display unit 126, etc.) configured to display media, interfaces, and other content. The electronic device 1000 may also include a processing unit 1008 coupled to the input unit 1003, the audio input unit 1004, the speaker unit 1006, the communication unit 1007, and the optional display unit 1002. In some examples, the processing unit 1008 may include a display enabling unit 1010 , a detecting unit 1012 , a determining unit 1014 , a sampling unit 1016 , an output unit 1018 , an executing unit 1020 , an acquiring unit 1022 , and a switching unit 1024 .

[0331] According to some embodiments, the processing unit 1008 is configured to display content on a display unit (e.g., display unit 1002 or an independent display unit) (e.g., using display enabling unit 1010). The processing unit 1008 is further configured to detect user input (e.g., using detection unit 1012). The processing unit 1008 is further configured to determine whether the user input corresponds to a first input type (e.g., using determination unit 1014). The processing unit 1008 is further configured to display multiple exemplary natural language requests on the display unit based on determining that the user input corresponds to the first input type (e.g., display enabling unit 1010). The multiple exemplary natural language requests are contextually related to the displayed content, wherein receiving a user utterance corresponding to an exemplary natural language request among the multiple exemplary natural language requests causes the digital assistant to perform a corresponding action.

[0332] In some examples, user input is detected on a remote controller of the electronic device. In some examples, the first input type includes pressing a button of the remote controller and releasing the button within a predetermined duration. In some examples, multiple exemplary natural language requests are displayed on a display unit via a first user interface, and the first user interface is overlaid on the displayed content. In some examples, the displayed content includes media content, and the media content continues to play while the multiple exemplary natural language requests are displayed.

[0333] In some examples, the processing unit 1008 is further configured to display a visual indicator on the display unit (e.g., using the display enabling unit 1010) based on determining that the user input corresponds to the first input type, thereby indicating that the digital assistant is not processing the audio input.

[0334] In some examples, upon determining that the user input corresponds to the first input type, multiple exemplary natural language requests are displayed on the display unit after a predetermined amount of time. In some examples, each of the multiple exemplary natural language requests is displayed separately at different times in a predetermined order.

[0335] In some examples, the processing unit 1008 is further configured to display multiple lists (eg, using the display enabling unit 1010) of exemplary natural language requests, wherein each list is displayed in turn at a different time.

[0336] In some examples, the processing unit 1008 is further configured to determine whether the user input corresponds to a second input type based on determining that the user input does not correspond to the first input type (e.g., using the determination unit 1014). The processing unit 1008 is further configured to sample the audio data based on determining that the user input corresponds to the second input type (e.g., using the sampling unit 1016 and the audio input unit 1004). The processing unit 1008 is further configured to determine whether the audio data contains a user request (e.g., using the determination unit 1014). The processing unit 1008 is further configured to perform a task that at least partially satisfies the user request based on determining that the audio data contains the user request (e.g., using the execution unit 1020).

[0337] In some examples, the second input type includes pressing a button of an electronic device remote control and holding the button for more than a predetermined duration.

[0338] In some examples, the processing unit 1008 is further configured to display a request for clarifying the user's intent on a display unit (eg, using the display enabling unit 1010 ) based on determining that the audio data does not include a user request.

[0339] In some examples, the displayed content includes media content, and the media content continues to play on the electronic device while the audio data is sampled and the task is performed.

[0340] In some examples, the processing unit 1008 is further configured to output (e.g., using the output unit 1018) audio associated with the media content (e.g., using the speaker unit 1006). The processing unit 1008 is further configured to reduce the audio amplitude (e.g., using the output unit 1018) based on determining that the user input corresponds to the second input type.

[0341] In some examples, the task is performed without outputting a task-related voice from the electronic device. In some examples, the audio data is sampled when the user input is detected. In some examples, the audio data is sampled for a predetermined duration after the user input is detected.

[0342] In some examples, the audio data is sampled via a first microphone on the electronic device remote control (e.g., audio input unit 1004). The processing unit 1008 is further configured to sample background audio data via a second microphone on the remote control (e.g., a second audio input unit of the electronic device 1000) when sampling the audio data (e.g., using sampling unit 1016 and audio input unit 1004). The processing unit 1008 is further configured to use the background audio data to eliminate background noise in the audio data (e.g., using output unit 1018).

[0343] In some examples, audio associated with the displayed content is output via an audio signal from the electronic device. The processing unit 1008 is further configured to use the audio signal to remove background noise in the audio data (eg, using the output unit 1018).

[0344] In some examples, the processing unit 1008 is further configured to display a visual prompt on the display unit (eg, using the display enabling unit 1010 ) prompting the user to provide a verbal request in response to detecting the user input.

[0345] In some examples, the processing unit 1008 is further configured to obtain (e.g., using the acquisition unit 1022) a result that at least partially satisfies the user request. The processing unit 1008 is further configured to display a second user interface on the display unit (e.g., using the display enabling unit 1010). The second user interface includes a portion of the result, wherein at least a portion of the content continues to be displayed when the second user interface is displayed, and wherein a display area of ​​the second user interface on the display unit is smaller than a display area of ​​at least a portion of the content on the display unit. In some examples, the second user interface is overlaid on the displayed content.

[0346] In some examples, the portion of the results includes one or more media items. The processing unit 1008 is further configured to receive a selection of a media item from the one or more media items via a second user interface (e.g., using the detection unit 1012). The processing unit 1008 is further configured to display media content associated with the selected media item on a display unit (e.g., using the display enabling unit 1010).

[0347] In some examples, the processing unit 1008 is further configured to detect a second user input while displaying the second user interface (e.g., using the detection unit 1012). The processing unit 1008 is further configured to stop displaying the second user interface in response to detecting the second user input (e.g., using the display enabling unit 1010).

[0348] In some examples, a second user input is detected on a remote control of the electronic device. The second user input includes a first predetermined motion pattern on a touch-sensitive surface of the remote control.

[0349] In some examples, the processing unit 1008 is further configured to detect a third user input while displaying the second user interface (e.g., using the detection unit 1012). The processing unit 1008 is further configured to replace the display of the second user interface on the display unit with the display of the third user interface in response to detecting the third user input (e.g., using the display enabling unit 1010). The third user interface includes at least a portion of the result, and the third user interface occupies at least a majority of the display area of ​​the display unit.

[0350] In some examples, a third user input is detected on a remote control for the electronic device, and the third user input includes a second predetermined pattern of motions on a touch-sensitive surface of the remote control.

[0351] In some examples, the processing unit 1008 is further configured to obtain a second result different from the result in response to detecting the third user input (e.g., using the obtaining unit 1022). The second result at least partially satisfies the user request, and the third user interface includes at least a portion of the second result.

[0352] In some examples, the second result is based on a user request received before detecting the user input. In some examples, when the third user input is detected, the focus of the second user interface is on an item that is part of the result, and the second result is contextually related to the item.

[0353] In some examples, the displayed content includes media content.The processing unit 1008 is further configured to pause playing of the media content on the electronic device in response to detecting the third user input (eg, using the execution unit 1020 ).

[0354] In some examples, at least a portion of the results include one or more media items. The processing unit 1008 is further configured to receive a selection of a media item from the one or more media items via a third user interface (e.g., using the detection unit 1012). The processing unit 1008 is further configured to display media content associated with the media item on a display unit (e.g., using the display enabling unit 1010).

[0355] In some examples, the processing unit 1008 is further configured to detect a fourth user input associated with a direction on the display unit while displaying the third user interface (e.g., using the detection unit 1012). The processing unit 1008 is further configured to switch the focus of the third user interface from the first item to the second item on the third user interface in response to detecting the fourth user input (e.g., using the switching unit 1024). The second item can be positioned in a direction relative to the first item.

[0356] In some examples, the processing unit 1008 is further configured to detect a fifth user input while displaying the third user interface (e.g., using the detection unit 1012). The processing unit 1008 is further configured to display a search field in response to detecting the fifth user input (e.g., using the display enabling unit 1010). The processing unit 1008 is further configured to display a virtual keyboard interface on the display unit (e.g., using the display enabling unit 1010), wherein the input received via the virtual keyboard interface results in text input in the search field.

[0357] In some examples, the processing unit 1008 is further configured to detect a sixth user input while displaying the third user interface (e.g., using the detection unit 1012). The processing unit 1008 is further configured to sample the second audio data in response to detecting the sixth user input (e.g., using the sampling unit 1016 and the audio input unit 1004). The second audio data includes a second user request. The processing unit 1008 is further configured to determine whether the second user request is a request for refining the results of the user request (e.g., using the determination unit 1014). The processing unit 1008 is further configured to display a subset of the results via the third user interface (e.g., using the display enabling unit 1010) based on determining that the second user request is a request for refining the results of the user request.

[0358] In some examples, the subgroup of results is displayed at the top row of the third user interface. The processing unit 1008 is further configured to obtain a third result that at least partially satisfies the second user request (e.g., using the acquisition unit 1018) based on determining that the second user request is not a request for refining the result of the user request. The processing unit 1008 is further configured to display a portion of the third result via the third user interface (e.g., using the display enabling unit 101). In some examples, the portion of the third result is displayed at the top row of the third user interface.

[0359] In some examples, the processing unit 1008 is further configured to obtain a fourth result that at least partially satisfies the user request or the second user request (e.g., using the obtaining unit 1022). The processing unit 1008 is further configured to display a portion of the fourth result via a third user interface (e.g., using the display enabling unit 1010).

[0360] In some examples, a portion of the fourth result is displayed at a row following a top row of the third user interface.

[0361] In some examples, when the sixth user input is detected, the focus of the third user interface is on one or more items of the third user interface, and the fourth result is contextually related to the one or more items.

[0362] In some examples, the processing unit 1008 is further configured to detect a seventh user input while displaying the third user interface (e.g., using the detection unit 1012). The processing unit 1008 is further configured to stop displaying the third user interface in response to detecting the seventh user input (e.g., using the display enabling unit 1010).

[0363] In some examples, the displayed content is media content, and playback of the media content on the electronic device is paused in response to detecting a third user input. The processing unit 1008 is further configured to resume playback of the media content on the electronic device in response to detecting a seventh user input (e.g., using the execution unit 1020). In some examples, the seventh user input includes pressing a menu button of a remote control of the electronic device.

[0364] According to some embodiments, the processing unit 1008 is further configured to display the content on the display unit (e.g., using the display enabling unit 1010). The processing unit 1008 is further configured to detect user input when displaying the content (e.g., using the detection unit 1012). The processing unit 1008 is further configured to display a user interface on the display unit in response to detecting the user input (e.g., using the display enabling unit 1010). The user interface includes multiple exemplary natural language requests related to the displayed content context, wherein receiving a user utterance corresponding to an exemplary natural language request in the multiple exemplary natural language requests causes the digital assistant to perform a corresponding action.

[0365] In some examples, the displayed content includes media content. In some examples, the plurality of exemplary natural language requests include a natural language request for modifying one or more settings associated with the media content. In some examples, the media content continues to play while the user interface is displayed.

[0366] In some examples, the processing unit 1008 is further configured to output audio associated with the media content. In response to detecting the user input (e.g., using the output unit 1018), the amplitude of the audio is not reduced. In some examples, the displayed content includes a main menu user interface.

[0367] In some examples, the plurality of exemplary natural language requests include exemplary natural language requests associated with each of a plurality of core competencies of the digital assistant. In some examples, the displayed content includes a second user interface with results associated with a previous user request. In some examples, the plurality of exemplary natural language requests include a natural language request for refining the results. In some examples, the user interface includes text instructions for invoking and interacting with the digital assistant. In some examples, the user interface includes a visual indicator indicating that the digital assistant is not receiving audio input. In some examples, the user interface is overlaid on the displayed content.

[0368] In some examples, the processing unit 1008 is further configured to reduce the brightness of the displayed content to highlight the user interface in response to detecting the user input (e.g., using the display enabling unit 1010).

[0369] In some examples, user input is detected on a remote controller of the electronic device. In some examples, the user input includes pressing a button of the remote control device and releasing the button within a predetermined duration after pressing the button. In some examples, the button is configured to invoke a digital assistant. In some examples, the user interface includes text instructions for displaying a virtual keyboard interface.

[0370] In some examples, the processing unit 1008 is further configured to detect a second user input after displaying the user interface (e.g., using the detection unit 1012). The processing unit 1008 is further configured to display a virtual keyboard interface on the display unit (e.g., using the display unit 1012) in response to detecting the second user input.

[0371] In some examples, the processing unit 1008 is further configured to change the focus of the user interface to a search field on the user interface (e.g., using the display enabling unit 1010). In some examples, the search field is configured to receive a text search query via a virtual keyboard interface. In some examples, the virtual keyboard interface is not available for interacting with a digital assistant. In some examples, the second user input includes a predetermined action pattern on a touch-sensitive surface of a remote control device of the electronic device.

[0372] In some examples, the plurality of exemplary natural language requests are displayed a predetermined amount of time after detecting the user input. In some examples, the processing unit 1008 is further configured to display each of the plurality of exemplary natural language requests one at a time in a predetermined order (e.g., using the display enabling unit 1010). In some examples, the processing unit 1008 is further configured to replace display of a previously displayed exemplary natural language request in the plurality of exemplary natural language requests with a subsequent exemplary natural language request in the plurality of exemplary natural language requests (e.g., using the display enabling unit 1010).

[0373] In some examples, the content includes a second user interface having one or more items. When user input is detected, the focus of the second user interface is located on an item in the one or more items. Multiple exemplary natural language requests are contextually related to the item in the one or more items.

[0374] According to some embodiments, the processing unit 1008 is further configured to display content on a display unit (e.g., using display enabling unit 1010). The processing unit 1008 is further configured to detect user input (e.g., using detection unit 1012). The processing unit 1008 is further configured to display one or more suggested examples of natural language utterances in response to detecting the user input (e.g., using display enabling unit 1010). The one or more suggested examples are contextually relevant to the displayed content and, when spoken by the user, cause the digital assistant to perform a corresponding action.

[0375] In some examples, the processing unit 1008 is further configured to detect a second user input (e.g., using the detection unit 1012). The processing unit 1008 is further configured to sample the audio data in response to detecting the second user input (e.g., using the sampling unit 1016). The processing unit 1008 is further configured to determine (e.g., using the determination unit 1014) whether the sampled audio data contains one of the one or more suggested examples of the natural language utterance. The processing unit 1008 is further configured to perform a corresponding action for the utterance (e.g., using the execution unit 1020) based on determining that the sampled audio data contains one of the one or more suggested examples of the natural language utterance.

[0376] According to some embodiments, the processing unit 1008 is further configured to display the content on the display unit (e.g., using the display enabling unit 1010). The processing unit 1008 is further configured to detect user input when displaying the content (e.g., using the detection unit 1012). The processing unit 1008 is further configured to sample audio data in response to detecting the user input (e.g., using the sampling unit 1016). The audio data includes user utterances representing the media search request. The processing unit 1008 is further configured to obtain multiple media items that satisfy the media search request (e.g., using the acquisition unit 1022). The processing unit 1008 is further configured to display at least a portion of the multiple media items on the display unit via the user interface (e.g., using the display enabling unit 1010).

[0377] In some examples, the content continues to be displayed on the display unit while at least a portion of the plurality of media items is displayed.The display area occupied by the user interface is smaller than the display area occupied by the content.

[0378] In some examples, the processing unit 1008 is further configured to determine whether the number of media items in the plurality of media items is less than or equal to a predetermined number (e.g., using the determining unit 1014). Based on determining that the number of media items in the plurality of media items is less than or equal to the predetermined number, at least a portion of the plurality of media items includes the plurality of media items.

[0379] In some examples, based on determining that the number of media items in the plurality of media items is greater than the predetermined number, the number of media items in at least a portion of the plurality of media items is equal to the predetermined number.

[0380] In some examples, each media item in the plurality of media items is associated with a relevance score relative to the media search request, and the relevance score of at least a portion of the plurality of media items is highest among the plurality of media items.

[0381] In some examples, each media item of at least a portion of the plurality of media items is associated with a popularity rating, and at least a portion of the plurality of media items are arranged in the user interface based on the popularity ratings.

[0382] In some examples, the processing unit 1008 is further configured to detect a second user input while displaying at least a portion of the plurality of media items (e.g., using the detection unit 1012). The processing unit 1008 is further configured to expand the user interface (e.g., using the display enabling unit 1010) in response to detecting the second user input so as to occupy at least a majority of the display area of ​​the display unit.

[0383] In some examples, the processing unit 1008 is further configured to determine whether the number of media items in the plurality of media items is less than or equal to a predetermined number in response to detecting the second user input (e.g., using the determination unit 1014). The processing unit 1008 is further configured to obtain a second plurality of media items that at least partially satisfy the media search request based on determining that the number of media items in the plurality of media items is less than or equal to the predetermined number, the second plurality of media items being different from at least a portion of the media items. The processing unit 1008 is further configured to display the second plurality of media items on the display unit via the expanded user interface (e.g., using the display enabling unit 101).

[0384] In some examples, the processing unit 1008 is further configured to determine whether the media search request includes more than one search parameter (e.g., using the determining unit 1014). Based on determining that the media search request includes more than one search parameter, the second plurality of media items are organized in the expanded user interface according to the more than one search parameter of the media search request.

[0385] In some examples, the processing unit 1008 is further configured to display at least a second portion of the plurality of media items via the expanded user interface (e.g., using the display enabling unit 1010) based on determining that the number of media items in the plurality of media items is greater than a predetermined number. The at least second portion of the plurality of media items is different from at least a portion of the plurality of media items.

[0386] In some examples, at least a second portion of the plurality of media items includes two or more media types, and at least a second portion of the plurality of media items is organized in the expanded user interface according to each of the two or more media types.

[0387] In some examples, the processing unit 1008 is further configured to detect a third user input (e.g., using the detection unit 1012). The processing unit 1008 is further configured to cause the extended user interface to scroll (e.g., using the display enabling unit 1010) in response to detecting the third user input. The processing unit 1008 is further configured to determine whether the extended user interface has scrolled beyond a predetermined position on the extended user interface (e.g., using the determination unit 1014). The processing unit 1008 is further configured to display at least a third portion of the plurality of media items on the extended user interface in response to determining that the extended user interface has scrolled beyond a predetermined position on the extended user interface (e.g., using the display enabling unit 1010). At least a third portion of the plurality of media items is organized on the extended user interface according to one or more media content providers associated with the third plurality of media items.

[0388] References FIG. 5A to FIG. 5I The operation is optionally performed by Figures 1 to 3 and FIG. 4A to FIG. 4B For example, display operations 502, 508-514, 520, 524, 530, 536, 546, 556, 560, 562, 576, 582, 588, 592, detection operations 504, 538, 542, 550, 558, 566, 570, determination operations 506, 516, 522, 526, 528, 574, 578, sampling operations 518, 572, execution operations 532, 584, acquisition operations 534, 544, 580, 586, 590, pause operations 540, 568, receiving operations 554, and switching operations 552, 564 may be implemented by one or more of the operating system 252, the GUI module 256, the application module 262, the digital assistant module 426, and one or more processors 204, 404. Those skilled in the art will clearly know how to Figures 1 to 3 and FIG. 4A to FIG. 4B The components shown in FIG. 8 are used to implement other processes.

[0389] According to some examples, Fig.11 1 shows a functional block diagram of an electronic device 1100 that is configured according to the principles of the various examples described, such as to voice control media playback and update the knowledge of a virtual assistant in real time. The functional blocks of the device can be implemented by hardware, software, or a combination of hardware and software that implements the principles of the various examples described. It should be understood by those skilled in the art that Fig.11 The functional blocks described in the invention may be combined or separated into sub-blocks to implement the principles of various described examples. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0390] like Fig.11As shown, the electronic device 1100 may include: an input unit 1103 (e.g., remote control 124, etc.) configured to receive user input such as tactile input, gesture input; an audio input unit 1104 (e.g., microphone 272, etc.) configured to receive audio data; a speaker unit 116 (e.g., speaker 268, etc.) configured to output audio; and a communication unit 1107 (e.g., communication subsystem 224, etc.) configured to send and receive information from an external device via a network. In some examples, the electronic device 1100 may optionally include a display unit 1102 (e.g., display unit 126, etc.) configured to display media, interfaces, and other content. The electronic device 1100 may also include a processing unit 1108 coupled to the input unit 1103, the audio input unit 1104, the speaker unit 1106, the communication unit 1107, and the optional display unit 1102. In some examples, processing unit 1108 may include a display enabling unit 1110 , a detecting unit 1112 , a determining unit 1114 , a sampling unit 1116 , an output unit 1118 , an executing unit 1120 , an acquiring unit 1122 , an identifying unit 1124 , and a transmitting unit 1126 .

[0391] According to some embodiments, the processing unit 1108 is configured to display content on a display unit (e.g., display unit 1102 or an independent display unit) (e.g., using display enabling unit 1110). The processing unit 1108 is further configured to detect user input when displaying content (e.g., using detection unit 1112). The processing unit 1108 is further configured to sample audio data in response to detecting user input (e.g., using sampling unit 1016 and audio input unit 1104). The audio data includes user speech. The processing unit 1108 is further configured to obtain a determination of a user intent corresponding to the user speech (e.g., using acquisition unit 1122). The processing unit 1108 is further configured to obtain a determination of whether the user intent includes a request to adjust the state or settings of an application on the electronic device (e.g., using acquisition unit 1122). The processing unit 1108 is further configured to adjust the state or settings of the application to meet the user intent (e.g., using task execution unit 1120) in response to obtaining a determination that the user intent includes a request to adjust the state or settings of an application on the electronic device.

[0392] In some examples, the request to adjust a state or setting of an application on the electronic device includes a request to play a specific media item. Adjusting the state or setting of the application to satisfy the user's intent includes playing the specific media item.

[0393] In some examples, the displayed content includes a user interface with media items, and the user utterance does not explicitly define a specific media item to be played. The processing unit 1108 is further configured to determine whether the focus of the user interface is on the media item (e.g., using the determination unit 1114). The processing unit 1108 is further configured to identify the media item as a specific media item to be played based on determining that the focus of the user interface is on the media item (e.g., using the identification unit 1124).

[0394] In some examples, a request to adjust the state or settings of an application on an electronic device includes a request to start an application on the electronic device. In some examples, the displayed content includes media content played on the electronic device, and the state or setting is related to the media content played on the electronic device. In some examples, a request to adjust the state or settings of an application on an electronic device includes a request to fast forward or rewind the media content played on the electronic device. In some examples, a request to adjust the state or settings of an application on an electronic device includes a request to jump forward or backward in the media content to play a specific portion of the media content. In some examples, a request to adjust the state or settings of an application on an electronic device includes a request to pause the media content played on the electronic device. In some examples, a request to adjust the state or settings of an application on an electronic device includes a request to turn on or off subtitles of the media content.

[0395] In some examples, the displayed content includes a user interface having a first media item and a second media item.

[0396] In some examples, the request to adjust the state or setting of an application on the electronic device includes a request to switch the focus of the user interface from the first media item to the second media item. Adjusting the state or setting of the application to satisfy the user's intent includes switching the focus of the user interface from the first media item to the second media item.

[0397] In some examples, the displayed content includes media content being played on a media device. User utterances are natural language expressions indicating that a user did not hear a portion of audio associated with the media content. A request for adjusting the state or settings of an application on an electronic device includes a request for replaying a portion of the media content, the portion of the media content corresponding to a portion of the audio that the user did not hear. The processing unit 1108 is further configured to rewind the media content to a previous portion of the media content by a predetermined amount (e.g., using a task execution unit 1120); and restart playing the media content from a previous portion (e.g., using a task execution unit 1120).

[0398] In some examples, the processing unit 1108 is further configured to turn on closed captions (eg, using the task execution unit 1120 ) before resuming playback of the media content from a previous portion.

[0399] In some examples, the request to adjust the state or setting of an application on the electronic device also includes a request to increase the volume of audio associated with the media content. Adjusting the state or setting of the application also includes increasing the volume of the audio associated with the media content before resuming playback of the media content from a previous portion.

[0400] In some examples, speech in audio associated with the media content is converted to text. Adjusting the state or setting of the application also includes displaying a portion of the text when playback of the media content is restarted from a previous portion.

[0401] In some examples, the processing unit 1108 is further configured to obtain a determination of a user emotion associated with the user utterance (eg, using the obtaining unit 1122 ) and determine the user intent based on the determined user emotion.

[0402] In some examples, the processing unit 1108 is further configured to obtain a determination of whether the user intent is a predetermined request type among multiple predetermined request types in response to obtaining a determination that the user intent does not include a request to adjust the state or setting of an application on the electronic device (e.g., using the obtaining unit 1122). The processing unit 1108 is further configured to obtain a result that at least partially satisfies the user intent in response to obtaining a determination that the user intent is a predetermined request type among multiple predetermined request types (e.g., using the obtaining unit 1122), and display the result in text form on the display unit (e.g., using the display enabling unit 1110).

[0403] In some examples, the plurality of predetermined request types include a request for a current time at a particular location. In some examples, the plurality of predetermined request types include a request for presenting a joke. In some examples, the plurality of predetermined request types include a request for information about media content played on an electronic device. In some examples, the textual results are overlaid on the displayed content. In some examples, the displayed content includes media content played on the electronic device, and the media content continues to play while the textual results are displayed.

[0404] In some examples, the processing unit 1108 is further configured to, in response to obtaining a determination that the user intent is not a predetermined request type among a plurality of predetermined request types, obtain media content that at least partially satisfies the user intent (e.g., using the acquisition unit 1122), and determine whether the displayed content includes media content played on the electronic device (e.g., using the determination unit 1114). The processing unit 1108 is further configured to determine whether the media content can be paused based on determining that the displayed content includes the media content (e.g., the determination unit 1114). The processing unit 1108 is further configured to display a second user interface including a portion of the second result on the display unit based on determining that the media content cannot be paused (e.g., the display enabling unit 1110). The display area occupied by the second user interface on the display unit is smaller than the display area occupied by the media content on the display unit.

[0405] In some examples, the user intent includes a request for a weather forecast for a particular location. The user intent includes a request for information associated with a sports team or athlete. In some examples, the user intent is not a media search query, and wherein the second result includes one or more media items having media content that at least partially satisfies the user intent. In some examples, the second result also includes non-media data that at least partially satisfies the user intent. In some examples, the user intent is a media search query, and the second result includes a plurality of media items corresponding to the media search query.

[0406] In some examples, the processing unit 1108 is further configured to display a third user interface including a portion of the second result on the display unit (e.g., using display enabling unit 1110) based on determining that the displayed content does not include media content played on the electronic device, wherein the third user interface occupies a majority of the display area of ​​the display unit.

[0407] In some examples, the display content includes a main menu user interface.

[0408] In some examples, the displayed content includes a third user interface having a previous result, the previous result being related to a previous user request received before detecting the user input. Based on determining that the displayed content does not include media content played on the electronic device, the display of the previous result in the third user interface is replaced with the display of the second result.

[0409] In some examples, the processing unit 1108 is further configured to determine whether the displayed content includes a second user interface having a previous result from a previous user request (e.g., using the determination unit 1114) based on determining that the displayed content includes media content played on the electronic device. Based on determining that the displayed content includes a second user interface having a previous result from a previous user request, the previous result is replaced with the second result.

[0410] In some examples, the processing unit 1108 is further configured to pause playback of the media content on the electronic device (e.g., using the task execution unit 1120) based on determining that the media content can be paused, and display a third user interface including a portion of the second result on the display unit (e.g., using the display enabling unit 1110), wherein the third user interface occupies most of the display area of ​​the display unit.

[0411] In some examples, the processing unit 1108 is further configured to transmit the audio data to a server to perform natural language processing (e.g., using the transmission unit 1126 and using the communication unit 1107), and indicate to the server that the audio data is associated with a media application (e.g., using the transmission unit 1126). The indication biases the natural language processing toward the user's intent related to the media.

[0412] In some examples, the processing unit 1108 is further configured to transmit the audio data to a server to perform speech-to-text processing (e.g., transmission unit 1126).

[0413] In some examples, the processing unit 1108 is further configured to indicate to the server that the audio data is associated with a media application (eg, using the transmission unit 1126). The indication biases the speech-to-text processing toward text results related to the media.

[0414] In some examples, the processing unit 1108 is further configured to obtain a textual representation of the user utterance (e.g., using the obtaining unit 1122), where the textual representation is based on a previous user utterance received prior to sampling the audio data.

[0415] In some examples, the textual representation is based on the time at which a previous user utterance was received before the audio data was sampled.

[0416] In some examples, the processing unit 1108 is further configured to obtain (e.g., using the acquisition unit 1122) a determination that the user intent does not correspond to a core competency of a plurality of core competencies associated with the electronic device. The processing unit 1108 is further configured to cause the second electronic device to perform a task that facilitates satisfying the user intent (e.g., using the task execution unit 1120).

[0417] In some examples, the processing unit 1108 is further configured to obtain a determination of whether the user utterance includes a vague term (e.g., using the obtaining unit 1122). The processing unit 1108 is further configured to obtain two or more candidate user intents based on the vague term in response to obtaining a determination that the user utterance includes a vague term (e.g., using the obtaining unit 1122); and display the two or more candidate user intents on the display unit (e.g., using the display enabling unit 1110).

[0418] In some examples, the processing unit 1108 is further configured to receive a user selection of one of the two or more candidate user intents when the two or more candidate user intents are displayed (eg, using the detection unit 1112 ) and determine the user intent based on the user selection.

[0419] In some examples, the processing unit 1108 is further configured to detect a second user input (e.g., using a detection unit). The processing unit 1108 is further configured to sample second audio data in response to detecting the second user input (e.g., using a sampling unit 1116). The second audio data includes a second user utterance representing a user selection.

[0420] In some examples, two or more interpretations are displayed without outputting speech associated with the two or more candidate user intents.

[0421] According to some embodiments, the processing unit 1108 is further configured to display content on a display unit (e.g., display unit 1102 or an independent display unit) (e.g., using display enabling unit 1110). The processing unit 1108 is further configured to detect user input when displaying content (e.g., using detection unit 1112). The processing unit 1108 is further configured to display a virtual keyboard interface on the display unit in response to detecting user input (e.g., using display enabling unit 1110). The processing unit 1108 is further configured to cause a selectable enabling representation to appear on the display of the second electronic device (e.g., using task execution unit 1120). Selection of the enabling representation causes the electronic device to receive text input via the keyboard of the second electronic device (e.g., using communication unit 1107).

[0422] In some examples, the processing unit 1108 is further configured to receive text input via a keyboard of the second electronic device (e.g., using the detection unit 1112), wherein the text input represents a user request. The processing unit 1108 is further configured to obtain a result that at least partially satisfies the user request (e.g., using the acquisition unit 1122), and display a user interface on the display unit (e.g., using the display enabling unit 1110), wherein the user interface includes at least a portion of the result.

[0423] In some examples, the displayed content includes a second user interface having multiple exemplary natural language requests. In some examples, the displayed content includes media content. In some examples, the displayed content includes a third user interface having results from a previous user request, wherein the third user interface occupies at least a majority of the display area of ​​the display unit. In some examples, the virtual keyboard interface is overlaid on at least a portion of the third user interface. In some examples, user input is detected via a remote control of the electronic device, and the remote control and the second electronic device are different devices. In some examples, the user input includes a predetermined action pattern on a touch-sensitive surface of the remote control device. In some examples, user input is detected via a second electronic device.

[0424] References 7A to 7C and Fig. 9 The operation is optionally performed by Figures 1 to 3 and Figure 4A The above reference 7A to 7C and Fig. 9 The operation is optionally performed by Figures 1 to 3 and FIG. 4A to FIG. 4B For example, display operations 702, 716, 732, 736, 738, 742, 746, 902, 906, 914, detection operations 704, 718, 904, 910, determination operations 708, 710, 712, 714, 720, 724, 728, 736, 740, sampling operations 706, execution operations 722, 726, 744, 908, acquisition operations 730, 734, 912, and switching operations 552, 564 may be implemented by one or more of the operating system 252, 352, GUI module 256, 356, application module 262, 362, digital assistant module 426, and one or more processors 204, 304, 404. Those skilled in the art will clearly know how to implement the system based on the system. Figures 1 to 3 and FIG. 4A to FIG. 4B The components shown in FIG. 8 are used to implement other processes.

[0425] According to some specific implementations, a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) is provided, which stores one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for performing any of the methods described herein.

[0426] According to some implementations, an electronic device (eg, a portable electronic device) is provided that includes means for performing any of the methods described herein.

[0427] According to some implementations, an electronic device (e.g., a portable electronic device) is provided that includes a processing unit configured to perform any of the methods described herein.

[0428] According to some specific implementations, an electronic device (e.g., a portable electronic device) is provided, which includes one or more processors and a memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for performing any of the methods described herein.

[0429] Exemplary methods, non-transitory computer-readable storage media, systems, and electronic devices are set forth in the following items:

[0430] 1. A method for operating a digital assistant of a media system, the method comprising:

[0431] At an electronic device having a memory and one or more processors:

[0432] displaying the content on the display unit;

[0433] Detect user input;

[0434] determining whether the user input corresponds to a first input type; and based on determining that the user input corresponds to the first input type:

[0435] Multiple exemplary natural language requests are displayed on a display unit, the multiple exemplary natural language requests are contextually relevant to the displayed content, wherein receiving a user utterance corresponding to an exemplary natural language request among the multiple exemplary natural language requests causes the digital assistant to perform a corresponding action.

[0436] 2. The method of claim 1, wherein user input is detected on a remote controller of the electronic device.

[0437] 3. The method according to item 2, wherein the first input type includes pressing a button of the remote control and releasing the button within a predetermined duration.

[0438] 4. A method according to any one of items 1 to 3, wherein a plurality of exemplary natural language requests are displayed on a display unit via a first user interface, and wherein the first user interface is overlaid on the displayed content.

[0439] 5. A method according to any one of items 1 to 4, wherein the displayed content includes media content, and wherein the media content continues to play while the plurality of exemplary natural language requests are displayed.

[0440] 6. The method according to any one of items 1 to 5, further comprising:

[0441] Based on determining that the user input corresponds to the first input type, a visual indicator is displayed on the display unit to indicate that the digital assistant is not processing the audio input.

[0442] 7. A method according to any one of items 1 to 6, wherein after determining that the user input corresponds to the first input type, a plurality of exemplary natural language requests are displayed on the display unit after a predetermined amount of time.

[0443] 8. A method according to any one of items 1 to 7, wherein each of the multiple exemplary natural language requests is displayed separately in a predetermined order and at different times.

[0444] 9. The method according to any one of items 1 to 8, wherein displaying a plurality of exemplary natural language requests comprises:

[0445] Multiple lists of exemplary natural language requests are displayed, with each list being displayed at a different time and in turn.

[0446] 10. The method according to any one of items 1 to 9, further comprising:

[0447] Based on determining that the user input does not correspond to the first input type:

[0448] determining whether the user input corresponds to a second input type; and

[0449] According to determining that the user input corresponds to the second input type:

[0450] Sampling audio data;

[0451] Determining whether the audio data includes a user request; and performing a task that at least partially satisfies the user request based on determining that the audio data includes the user request.

[0452] 11. The method of item 10, wherein the second input type comprises pressing a button of a remote control of the electronic device and holding the button for a predetermined duration.

[0453] 12. The method according to any one of items 10 to 11, further comprising:

[0454] Based on determining that the audio data does not include a user request, a request for clarifying the user's intention is displayed on the display unit.

[0455] 13. A method according to any one of items 10 to 12, wherein the displayed content includes media content, and wherein the media content continues to play on the electronic device while the audio data is sampled and while the task is performed.

[0456] 14. The method according to item 13, further comprising:

[0457] outputting audio associated with the media content; and

[0458] Based on determining that the user input corresponds to the second input type, the amplitude of the audio is reduced.

[0459] 15. A method according to any one of items 10 to 14, wherein the task is performed without outputting a voice related to the task from the electronic device.

[0460] 16. A method according to any one of items 10 to 15, wherein the audio data is sampled when user input is detected.

[0461] 17. A method according to any one of items 10 to 15, wherein the audio data is sampled for a predetermined duration after detecting the user input.

[0462] 18. The method according to any one of items 10 to 17, wherein the audio data is sampled via a first microphone on a remote controller of the electronic device, and further comprising:

[0463] While sampling the audio data, sampling background audio data via a second microphone on the remote controller; and

[0464] Background audio data is used to remove background noise from the audio data.

[0465] 19. The method of any one of items 10 to 18, wherein audio associated with the displayed content is output via an audio signal from an electronic device, and further comprising:

[0466] Use an audio signal to remove background noise from audio data.

[0467] 20. The method according to any one of items 10 to 19, further comprising:

[0468] In response to detecting the user input, a visual prompt is displayed on the display unit prompting the user to provide a verbal request.

[0469] 21. The method according to any one of items 10 to 20, wherein the tasks performed include:

[0470] Obtaining a result that at least partially satisfies the user's request; and

[0471] A second user interface is displayed on the display unit, the second user interface including a portion of the result, wherein at least a portion of the content continues to be displayed while the second user interface is displayed, and wherein a display area of ​​the second user interface on the display unit is smaller than a display area of ​​at least a portion of the content on the display unit.

[0472] 22. The method of item 21, wherein the second user interface is overlaid on the displayed content.

[0473] 23. The method of any one of clauses 21 to 22, wherein a portion of the results includes one or more media items, and further comprising:

[0474] receiving, via a second user interface, a selection of a media item from among the one or more media items; and

[0475] Media content associated with the selected media item is displayed on the display unit.

[0476] 24. The method according to any one of items 21 to 22, further comprising:

[0477] While displaying the second user interface, detecting a second user input; and

[0478] In response to detecting the second user input, ceasing to display the second user interface.

[0479] 25. The method of item 24, wherein a second user input is detected on a remote control of the electronic device, and wherein the second user input comprises a first predetermined pattern of motions on a touch-sensitive surface of the remote control.

[0480] 26. The method according to any one of items 21 to 22, further comprising:

[0481] While displaying the second user interface, detecting a third user input; and

[0482] In response to detecting a third user input, display of the second user interface is replaced with display of a third user interface on the display unit, the third user interface including at least a portion of the results, wherein the third user interface occupies at least a majority of a display area of ​​the display unit.

[0483] 27. The method of item 26, wherein a third user input is detected on a remote control of the electronic device, and wherein the third user input comprises a second predetermined pattern of actions on the touch-sensitive surface of the remote control.

[0484] 28. The method according to any one of items 26 to 27, further comprising:

[0485] In response to detecting a third user input:

[0486] A second result different from the result is obtained, wherein the second result at least partially satisfies the user request, and wherein the third user interface includes at least a portion of the second result.

[0487] 29. The method of clause 28, wherein the second result is based on a user request received before the user input is detected.

[0488] 30. The method of any one of items 28 to 29, wherein upon detecting the third user input, the focus of the second user interface is on an item that is a portion of the results, and wherein the second result is contextually related to the item.

[0489] 31. A method according to any one of items 26 to 30, wherein the displayed content includes media content, and wherein in response to detecting a third user input, playback of the media content on the electronic device is paused.

[0490] 32. The method of any one of clauses 26 to 31, wherein at least a portion of the results include one or more media items, and further comprising:

[0491] receiving, via a third user interface, a selection of a media item from among the one or more media items; and

[0492] Media content associated with the media item is displayed on the display unit.

[0493] 33. The method according to any one of items 26 to 32, further comprising:

[0494] while displaying the third user interface, detecting a fourth user input associated with a direction on the display unit;

[0495] In response to detecting a fourth user input:

[0496] A focus of the third user interface is switched from the first item to a second item on the third user interface, the second item being positioned in a direction relative to the first item.

[0497] 34. The method according to any one of items 26 to 33, further comprising:

[0498] While displaying the third user interface, detecting a fifth user input; and

[0499] In response to detecting a fifth user input:

[0500] Display the search field; and

[0501] A virtual keyboard interface is displayed on the display unit, wherein input received via the virtual keyboard interface results in text entry in the search field.

[0502] 35. The method according to any one of items 26 to 34, further comprising:

[0503] While displaying the third user interface, detecting a sixth user input; and

[0504] In response to detecting a sixth user input:

[0505] Sampling second audio data, where the second audio data includes a second user request;

[0506] determining whether the second user request is a request for refining the result of the user request; and

[0507] According to determining that the second user request is a request for refining the result of the user request:

[0508] A subset of the results is displayed via a third user interface.

[0509] 36. A method according to item 35, wherein the subgroup of results is displayed at the top row of the third user interface.

[0510] 37. The method according to any one of items 35 to 36, further comprising:

[0511] Based on determining that the second user request is not a request for refining the result of the user request:

[0512] Obtaining a third result that at least partially satisfies the second user request; and

[0513] A portion of the third result is displayed via a third user interface.

[0514] 38. The method of item 37, wherein a portion of the third results are displayed at a top row of the third user interface.

[0515] 39. The method according to any one of items 35 to 38, further comprising:

[0516] Obtaining a fourth result that at least partially satisfies the user request or the second user request; and

[0517] A portion of the fourth result is displayed via the third user interface.

[0518] 40. The method of item 39, wherein a portion of the fourth result is displayed at a row subsequent to a top row of the third user interface.

[0519] 41. A method according to any one of items 39 to 40, wherein when the sixth user input is detected, the focus of the third user interface is on one or more items of the third user interface, and wherein the fourth result is contextually related to the one or more items.

[0520] 42. The method according to any one of items 26 to 41, further comprising:

[0521] When displaying the third user interface, detecting a seventh user input;

[0522] In response to detecting the seventh user input, ceasing to display the third user interface.

[0523] 43. A method according to item 42, wherein the displayed content is media content, wherein playback of the media content on the electronic device is paused in response to detecting a third user input, and wherein playback of the media content on the electronic device is resumed in response to detecting a seventh user input.

[0524] 44. A method according to any one of items 42 to 43, wherein the seventh user input comprises pressing a menu button of a remote control of the electronic device.

[0525] 45. A method for operating a digital assistant of a media system, the method comprising:

[0526] At an electronic device having a memory and one or more processors:

[0527] displaying the content on the display unit;

[0528] While displaying content, detect user input;

[0529] In response to detecting the user input:

[0530] A user interface is displayed on a display unit, the user interface comprising a plurality of exemplary natural language requests contextually relevant to the displayed content, wherein receiving a user utterance corresponding to an exemplary natural language request among the plurality of exemplary natural language requests causes the digital assistant to perform a corresponding action.

[0531] 46. ​​A method according to item 45, wherein the displayed content includes media content.

[0532] 47. The method of item 46, wherein the plurality of exemplary natural language requests comprises a natural language request for modifying one or more settings associated with the media content.

[0533] 48. A method according to any one of items 46 to 47, wherein the media content continues to play while the user interface is displayed.

[0534] 49. The method according to any one of items 46 to 41, further comprising:

[0535] Audio associated with the media content is output, wherein an amplitude of the audio is not reduced in response to detecting the user input.

[0536] 50. The method of item 45, wherein the displayed content includes a main menu user interface.

[0537] 51. A method according to item 50, wherein the plurality of exemplary natural language requests includes an exemplary natural language request related to each of a plurality of core competencies of the digital assistant.

[0538] 52. The method of item 45, wherein the displayed content includes a second user interface having results associated with a previous user request.

[0539] 53. A method according to item 52, wherein the plurality of exemplary natural language requests includes a natural language request for refining the results.

[0540] 54. A method according to any one of items 45 to 53, wherein the user interface includes text instructions for invoking and interacting with the digital assistant.

[0541] 55. A method according to any one of items 45 to 54, wherein the user interface includes a visual indicator indicating that the digital assistant is not receiving audio input.

[0542] 56. A method according to any one of items 45 to 55, wherein a user interface is overlaid on the displayed content.

[0543] 57. The method according to any one of items 45 to 56, further comprising:

[0544] In response to detecting the user input, the brightness of the displayed content is reduced to highlight the user interface.

[0545] 58. A method according to any one of items 45 to 57, wherein the user input is detected on a remote control of the electronic device.

[0546] 59. The method of item 58, wherein the user input comprises pressing a button of the remote control device and releasing the button within a predetermined duration after pressing the button.

[0547] 60. The method of item 59, wherein the button is configured to invoke a digital assistant.

[0548] 61. A method according to any one of items 45 to 60, wherein the user interface includes text instructions for displaying a virtual keyboard interface.

[0549] 62. The method according to any one of items 45 to 61, further comprising:

[0550] After displaying the user interface, detecting a second user input; and

[0551] In response to detecting the second user input, a virtual keyboard interface is displayed on the display unit.

[0552] 63. The method according to item 62, further comprising:

[0553] Changes the focus of the user interface to a search field in the user interface.

[0554] 64. A method according to item 63, wherein the search field is configured to receive a text search query via a virtual keyboard interface.

[0555] 65. A method according to any one of items 45 to 64, wherein the virtual keyboard interface cannot be used to interact with a digital assistant.

[0556] 66. A method according to any one of items 45 to 65, wherein the second user input comprises a predetermined pattern of actions on a touch-sensitive surface of a remote control device of the electronic device.

[0557] 67. A method according to any one of items 45 to 66, wherein a plurality of exemplary natural language requests are displayed a predetermined amount of time after detecting user input.

[0558] 68. The method according to any one of items 45 to 67, wherein displaying a plurality of exemplary natural language requests further comprises:

[0559] Each exemplary natural language request in the plurality of exemplary natural language requests is displayed one at a time in a predetermined order.

[0560] 69. The method of item 68, wherein displaying in sequence further comprises:

[0561] Display of a previously displayed exemplary natural language request in the plurality of exemplary natural language requests is replaced with a subsequent exemplary natural lan...

Claims

1. A computer-implemented method comprising: At a first electronic device having one or more first processors and a first memory: detecting a user input comprising a contact motion in a first direction; In response to detecting the user input including the contact action along the first direction, displaying a first search interface on a display of the first electronic device; as well as detecting text input via the first search interface; as well as At a second electronic device having one or more second processors and a second memory: A second search interface is displayed on a display associated with the second electronic device, wherein: The second search interface includes a search field, and The focus of the second search interface is not on the search field; In response to detecting the user input including the contact action along the first direction, adjusting the focus of the second search interface to the search field; receiving the text input from the first electronic device; and The received text input is displayed in the search field.

2. The method according to claim 1, comprising: receiving text input via a keyboard of the first electronic device, the text input representing a user request; Obtaining a result that at least partially satisfies the user request; as well as A user interface is displayed on a display unit of the second electronic device, the user interface including at least a portion of the result.

3. The method according to any one of claims 1 to 2, comprising: Content is caused to be displayed on a display unit of the second electronic device, wherein the displayed content includes a first user interface having a plurality of exemplary natural language requests. 4 . The method of claim 3 , wherein the displayed content includes a second user interface having results from a previous user request, the second user interface occupying at least a majority of a display area of ​​the display unit. The method of claim 4 , wherein the second search interface is overlaid on at least a portion of a third user interface.

6. The method of any one of claims 1 to 2, wherein the user input is detected via a remote control associated with the second electronic device, and wherein the remote control and the first electronic device are different devices.

7. The method of claim 6, wherein the user input comprises a predetermined pattern of motions on a touch-sensitive surface of a remote control device.

8. A computer-readable storage medium storing one or more programs, wherein the one or more programs are configured to be executed by one or more processors of a first electronic device having a display device, the one or more programs comprising instructions for: detecting a user input comprising a contact motion in a first direction; In response to detecting the user input including the contact action along the first direction, displaying a first search interface on a display of the first electronic device; as well as detecting text input via the first search interface; as well as At a second electronic device having one or more second processors and a second memory: A second search interface is displayed on a display associated with the second electronic device, wherein The second search interface includes a search field, and The focus of the second search interface is not on the search field; In response to detecting the user input including the contact action along the first direction, adjusting the focus of the second search interface to the search field; receiving the text input from the first electronic device; and The received text input is displayed in the search field.

9. The computer-readable storage medium according to claim 8, comprising: receiving text input via a keyboard of the first electronic device, the text input representing a user request; Obtaining a result that at least partially satisfies the user request; as well as A user interface is displayed on a display unit of the second electronic device, the user interface including at least a portion of the result.

10. The computer-readable storage medium according to any one of claims 8 to 9, comprising: Content is caused to be displayed on a display unit of the second electronic device, wherein the displayed content includes a first user interface having a plurality of exemplary natural language requests.

11. The computer-readable storage medium of claim 10, wherein the displayed content includes a second user interface having results from a previous user request, the second user interface occupying at least a majority of a display area of ​​the display unit.

12. The computer-readable storage medium of claim 11, wherein the second search interface is overlaid on at least a portion of a third user interface.

13. The computer-readable storage medium of any one of claims 8 to 9, wherein the user input is detected via a remote control associated with the second electronic device, and wherein the remote control and the first electronic device are different devices.

14. The computer-readable storage medium of claim 13, wherein the user input comprises a predetermined pattern of motions on a touch-sensitive surface of a remote control device.

15. A system comprising: A first electronic device comprising: one or more first processors; and a first memory storing one or more programs, the one or more programs being configured to be executed by the one or more first processors, the one or more programs comprising first instructions, the first instructions, when executed by the one or more first processors, causing the first electronic device to: detecting a user input comprising a contact motion in a first direction; In response to detecting the user input including the contact action along the first direction, displaying a first search interface on a display of the first electronic device; and detecting text input via the first search interface; and A second electronic device comprising: one or more second processors; and a second memory storing one or more second programs, the one or more second programs being configured to be executed by the one or more second processors, the one or more second programs comprising second instructions, which when executed by the one or more second processors cause the second electronic device to: A second search interface is displayed on a display associated with the second electronic device, wherein The second search interface includes a search field, and The focus of the second search interface is not on the search field; In response to detecting the user input including the contact action along the first direction, adjusting the focus of the second search interface to the search field; receiving the text input from the first electronic device; and The received text input is displayed in the search field.

16. The system of claim 15, wherein the second instruction causes the second electronic device to: receiving text input via a keyboard of the first electronic device, the text input representing a user request; Obtaining a result that at least partially satisfies the user request; and A user interface is displayed on a display unit of the second electronic device, the user interface including at least a portion of the result.

17. The system according to any one of claims 15 to 16, wherein the second instruction causes the second electronic device to: Content is caused to be displayed on a display unit of the second electronic device, wherein the displayed content includes a first user interface having a plurality of exemplary natural language requests.

18. The system of claim 17, wherein the displayed content includes a second user interface having results from a previous user request, the second user interface occupying at least a majority of a display area of ​​the display unit.

19. The system of claim 18, wherein the second search interface is overlaid on at least a portion of a third user interface.

20. The system of any one of claims 15 to 16, wherein the user input is detected via a remote control associated with the second electronic device, and wherein the remote control and the first electronic device are different devices.

21. The system of claim 20, wherein the user input comprises a predetermined pattern of motions on a touch-sensitive surface of a remote control device.

22. An electronic device comprising: Means for performing the method according to any one of claims 1 to 2.

23. An electronic device, comprising: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 2.

Images