Coordinated Processing of Overlapping Audio Queries

Abstract

To provide a system and method for coordinating overlapping processing of audio queries.SOLUTION: In the method performed by a data processing system for coordinating parallel processing of audio queries across a plurality of devices, an audio signal including speech of a user is received from a client device, the client device detects the audio signal by using a sensor such as a microphone or a transducer, the audio signal is converted to a text in order to generate a query and is transmitted to a first digital assistant server, a ranking determination logic is used to not only override a response of a first digital assistant but also select a response of a second digital assistant, the ranking determination logic is selected on the basis of a signal in the query and is provided to the client device to present the selected response from the selected digital assistant.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority under 35 U.S.C. § 119 of U.S. Provisional Patent Application No. 62 / 666,681, entitled "COORDINATION OF OVERLAPPING PROCESSING OF AUDIO QUERIES," filed May 3, 2018, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Excessive network transmissions, packet-based or otherwise, of network traffic data between computing devices may prevent the computing devices from correctly processing the network traffic data, completing operations related to the network traffic data, or responding to the network traffic data in a timely manner. Excessive network transmissions of network traffic data may also complicate data routing or degrade the quality of responses if the responding computing device is overwhelmed, which may result in inefficient bandwidth utilization. A portion of the excessive network transmissions may include transmissions for requests that are not valid requests. Summary of the Invention [Means for solving the problem]

[0003] At least one aspect is directed to a system for coordinating overlapping processing of audio queries. The system may include a data processing system. The data processing system may include at least one processor and a memory. The data processing system may receive one or more data packets via an interface, the data packets including an audio signal detected by a sensor of a computing device. The data processing system may generate a query based on the audio signal received via the one or more data packets. The data processing system may transmit the query to a plurality of digital assistant components. The data processing system may receive a first response to the query from a first digital assistant component of the plurality of digital assistant components. The data processing system may receive a second response to the query from a second digital assistant component of the plurality of digital assistant components. The first digital assistant component may be permitted to access a database to which the second digital assistant component is prohibited. The data processing system may determine, based on a ranking determination function, that the second response from the second digital assistant component should be selected over the first response from the first digital assistant component. The data processing system may provide the second response from the second digital assistant component to the computing device in response to the selection.

[0004] Optionally, the data processing system may be configured to select a second response from the second digital assistant component based on pre-configured preferences established for the second digital assistant component. Optionally, the data processing system may be configured to select a second response from the second digital assistant component based on a confidence score for the second response from the second digital assistant component. Optionally, the data processing system may be configured to determine that the first digital assistant component is a preferred digital assistant. The data processing system may be configured to determine that the first response from the first digital assistant component is flagged as a fallback response. The data processing system may be configured to override the first digital assistant component in response to determining that the first response from the first digital assistant component is flagged as a fallback response. The data processing system may be configured to select the second response from the second digital assistant component. The data processing system may be configured to retain the first digital assistant component as a preferred digital assistant. Optionally, the data processing system may be configured to determine a signal related to the query and select a ranking determination function from a plurality of ranking determination functions based on the signal. Optionally, the data processing system may be configured to determine a signal related to the query and select a response from the second digital assistant component based on the signal. Optionally, the data processing system may be configured, based on the ranking determination function, to select the second response based on the first response being an audio-only response. Optionally, the data processing system may be configured, based on the ranking determination function, to select the second response based on the second digital assistant component accessing data stored in the memory of the computing device. Optionally, the first response may comprise one of calendar information or media content.Optionally, the first digital assistant component may be configured to execute independently of the second digital assistant component to generate the first response. Optionally, the data processing system may be configured to send a query to each of the multiple digital assistants via an independent branch to have the multiple digital assistants process queries in an overlapping manner. Optionally, the first digital assistant component may be established by a different administrator than the second digital assistant component. Optionally, the data processing system may be configured to receive one or more data packets from the computing device via a streaming audio server configured to convert audio to text. Optionally, the data processing system may be configured to receive a second response from the second digital assistant component before the first response from the first digital assistant component. The data processing system may be configured to determine that the second digital assistant component is not a preferred digital assistant component. The data processing system may be configured to determine that the first digital assistant is a preferred digital assistant. The data processing system may be configured to wait for a first response from the first digital assistant component in response to determining that the second digital assistant component is not a preferred digital assistant and that the first digital assistant is a preferred digital assistant. Upon receiving the first response from the first digital assistant component, the data processing system may be configured to determine that the first response is flagged as a fallback response. The data processing system may be configured to select a second response based on a determination that the second digital assistant component is not a preferred digital assistant and that the first response is flagged as a fallback response.

[0005] At least one aspect is directed to a method for coordinating overlapping processing of audio queries. The method may be executed by a data processing system including at least one processor and a memory. The method may include the data processing system receiving one or more data packets including an audio signal detected by a sensor of a computing device. The method may include the data processing system generating a query based on the audio signal received via the one or more data packets. The method may include the data processing system transmitting the query to a plurality of digital assistant components. The method may include the data processing system receiving a first response to the query from a first digital assistant component of the plurality of digital assistant components. The method may include the data processing system receiving a second response to the query from a second digital assistant component of the plurality of digital assistant components. The first digital assistant component may be permitted to access a database to which the second digital assistant component is prohibited. The method may include the data processing system determining, based on a ranking determination function, that the data processing system should select the second response from the second digital assistant component over the first response from the first digital assistant component. The method can include the data processing system providing a second response from the second digital assistant component to the computing device in response to the selection.

[0006] Optionally, the method may include selecting a second response from the second digital assistant component based on a confidence score for the second response from the second digital assistant component. Optionally, the method may include determining that the first digital assistant component is a preferred digital assistant. The method may include determining that the first response from the first digital assistant component is flagged as a fallback response. The method may include overriding the first digital assistant component in response to determining that the first response from the first digital assistant component is flagged as a fallback response. The method may include selecting a second response from the second digital assistant component. The method may include retaining the first digital assistant component as a preferred digital assistant. Optionally, the method may include determining, based on a ranking determination function, that the first digital assistant component should select the first response based on accessing data stored in a memory of the computing device. Optionally, the method may include sending a query to each of a plurality of digital assistants via independent branches to have the plurality of digital assistants process the query in an overlapping manner. Optionally, the method may include receiving a second response from a second digital assistant component before a first response from the first digital assistant component. The method may include determining that the second digital assistant component is not a preferred digital assistant component. The method may include determining that the first digital assistant is a preferred digital assistant. The method may include waiting for a first response from the first digital assistant component in response to determining that the second digital assistant component is not a preferred digital assistant and that the first digital assistant is a preferred digital assistant.The method may include, upon receiving a first response from the first digital assistant component, determining that the first response is flagged as a fallback response. The method may include selecting a second response based on a determination that the second digital assistant component is not a preferred digital assistant and the first response is flagged as a fallback response.

[0007] At least one aspect is directed to a computer program product configured, when executed on a data processing system comprising a processor and a memory, to cause the data processing system to perform the method of the above aspect. The computer program product may be comprised in or provided on a tangible, non-transitory computer-readable medium.

[0008] These and other aspects and implementations are described in detail below. The above information and the following detailed description, including illustrative examples of various aspects and implementations, provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations, and are incorporated into and constitute a part of this specification.

[0009] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For clarity, not every component may be labeled in every drawing. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example system for coordinating overlapping processing of audio queries. [Figure 2] FIG. 1 illustrates an example operation of a system for coordinating overlapping processing of audio queries. [Figure 3]FIG. 3 is a flow diagram for coordinating overlapping processing of audio queries using the exemplary system shown in FIG. 1 or FIG. 2. [Figure 4] FIG. 3 illustrates a method for coordinating overlapping processing of audio queries using the exemplary system shown in FIG. 1 or FIG. 2. [Figure 5] FIG. 1 is a block diagram illustrating a schematic architecture for a computer system that may be employed to implement elements of the systems and methods described and illustrated herein, including, for example, the systems shown in FIGS. 1 and 2, the flow shown in FIG. 3, or the method shown in FIG. 4. DETAILED DESCRIPTION OF THE INVENTION

[0011] What follows is a more detailed description of various concepts related to methods, apparatus, and systems and their implementations for coordinating parallel processing of audio queries across multiple devices, such as data processing systems, display devices, and client devices. The various concepts introduced above and described in further detail below may be implemented in any of numerous ways.

[0012] An audio query may be processed by multiple digital assistant components. However, different digital assistant components may generate different results for the audio query based on the processing techniques used by each component or based on database access controls established for the different digital assistant components. Furthermore, different digital assistant components may process and return results at different times. Therefore, if different digital assistant components can process audio queries in parallel or in an overlapping manner, it may be difficult to coordinate the processing of the audio query.

[0013] The system and method of the present technical solution coordinate overlapping processing of audio queries. For example, a computing device can receive an audio query detected by a microphone of the computing device. The computing device, or its interface, can forward the audio query to a first digital assistant agent for processing. The interface can forward the audio query to a second digital assistant agent for processing. The first digital assistant can generate at least one first result, and the second digital assistant can generate a second result. The second digital assistant can generate a reliability score for the second result. In response to the reliability score, the second digital assistant can override the first digital assistant and determine that the second result should be returned. The second digital assistant can determine that the first result is not yet available and can override the first digital assistant to provide the second result.

[0014] For example, multiple digital assistant components (e.g., digital assistants, digital assistant agents, or voice agents) may be configured to process, parse, service, or respond to an audio query entered by a user via a computing device (e.g., a networked television, a networked speaker, or other networked computing device). The system and method of the present technical solution provide a framework for arbitrating and prioritizing among the responding digital assistant agents. For example, after a voice query is transcribed into text by a preprocessor component, the query may be passed to a first digital assistant (e.g., a third-party cloud configured with a third-party digital assistant component) to process the query, while a second digital assistant may process the query in a parallel or overlapping manner.

[0015] One of the digital assistant components, or a separate coordinator or response handler component, can determine which voice agent to use to provide a response—or which response to use. The coordinator component can be configured with rules, policies, or quality assessment techniques to determine which digital assistant component to use. The second digital assistant component can determine to override the first digital assistant component. The second digital assistant component can determine to override the first digital assistant component based on a reliability score, a quality assessment technique, a signal generated from parsing the query, a database used to provide results, or a domain related to the query.

[0016] For example, a first digital assistant component may be configured to process and respond to audio queries related to media-domain queries such as "Play Movie_A on Streaming_Service_A," while a second digital assistant component may be configured to process and respond to search-domain queries such as "When is my next appointment?" If the search query signal is ambiguous or falls into multiple categories, the coordinator component may score or rank responses from multiple digital assistant components and select a response from one of the digital assistant components. For example, the coordinator component may score or rank responses or queries based on confidence level or other factors.

[0017] Therefore, the system and method of the technical solution can reduce latency by allowing the first digital assistant component and the second digital assistant component to operate in parallel or overlapping fashion rather than serially. Latency can be further reduced by selecting results from the digital assistant that completes processing first. Reliability can be increased by using a ranking decision function to select a response or result from one of the digital assistants. The system and method of the technical solution can reduce dependency on third-party cloud services to obtain a preferred domain, which may result in more robust performance. For example, if a response from the second digital assistant component indicates that the query belongs to a domain that is available and preferred by the second digital assistant or its administrator, the second digital assistant component can provide a response without even waiting for the first digital assistant component to return a response, thereby reducing latency or delay when the first digital assistant is slower or experiencing malfunctions, errors, failures, excessive processing loads, or other processing issues. The system and method of the present technical solution can fine-tune results based on the reliability score from either the first digital assistant component or the second digital assistant component. The first digital assistant component can report "fallback" results along with their reliability scores. If the second digital assistant component has a lower reliability score compared to a threshold or the reliability score of the first digital assistant component, the system can determine to return the "fallback" results from the first digital assistant component. If the first digital assistant component indicates that it does not have an answer, the second digital assistant component can provide results regardless of the domain or signal associated with the query.

[0018] The systems and methods of the present disclosure may provide a way to enable third-party televisions with their own digital assistant agents to integrate with a digital assistant agent provided by a data processing system. The systems and methods may include registering a device model to specify common metadata for a type of device, such as third-party voice agent endpoints with which the digital assistant component of the data processing system can interact, optional voice bias phrases, and device capabilities, such as a browser. The systems and methods may include providing a field in the device model to specify which query fields are preferred for servicing media queries, etc. The systems and methods may include integrating with a digital assistant software development kit ("SDK") application programming interface ("API") extension for visual results on the television. The systems and methods may include integrating with a digital assistant SDK out-of-box experience ("OOBE") flow to guide a user to sign in to the device. The device restriction solution for supporting OOBE and authentication may be secure.

[0019] A model registration API can enable television or other computing device manufacturers to specify their cloud natural language understanding (“NLU”) endpoints and other independent broadcast program-specific metadata. The system can provide contextual voice bias phrases registered along with device models for the data processing system's automatic speech recognition to correctly recognize common queries uttered on the device. The system can trigger third-party cloud NLU in parallel with building candidate results for the data processing system to obtain optimal latency. The system can provide a ranking layer that takes into account ranking protocols or ranking policies for fulfilling queries based on domain. The system can provide a new processing flow to enable responses from a digital assistant component without requiring sign-in. The system can block content selection for queries for which responses are provided by a third-party digital assistant and can perform post-processing to scrub such queries from other logs. The system can provide a flow for agent capabilities since multiple agents are available on the platform. The system can provide a scalable visual solution for rendering HTML5 on third-party devices so that any device with a browser can show visual results and support interactive elements such as suggested clicks or links.

[0020] 1 shows an exemplary system 100 for coordinating parallel processing of audio queries across multiple devices. The system 100 can include at least one data processing system 102, at least one first digital assistant server 106, one or more client devices 104, one or more display devices 108, and a network 110. The display device 108 can be coupled to the client device 104 via a wired or wireless connection (e.g., via the network 110, short-range communication, or pairing). The client device 104 can be part of the display device 108 or can be otherwise integrated into the display device. The at least one data processing system 102, the one or more client devices 104, and the one or more display devices 108 can be communicatively coupled to each other via the network 110.

[0021] The data processing system 102 may include an interface 122. The data processing system 102 may include a second digital assistant application 120. The data processing system 102 may include a natural language processor (NLP) component 124 for parsing audio-based input. The data processing system 102 may include an interface management component 128 for detecting and managing interfaces of other devices in the system 100. The data processing system 102 may include an audio signal generator component 130 for generating an audio-based signal. The data processing system 102 may include a direct action application programming interface (API) 126 for fulfilling requests parsed from the audio-based input. The data processing system 102 may include the audio signal generator component 130 for selecting a response to the audio-based input signal. The data processing system 102 may include a data repository 140. The data processing system 102 may include a search coordinator component 132, a content indexer component 134, a ranking engine 136, and a response handler component 138. The NLP component 124, the interface management component 128, the audio signal generator component 130, the data repository 140, the direct action API 126, the interface management component 128, the search coordinator component 132, the content indexer component 134, the ranking engine 136, and the response handler component 138 can form the digital assistant application 120.

[0022] The functionality of data processing system 102, such as digital assistant application 120, may be included in or otherwise accessible from one or more client devices 104 and one or more display devices 108. The functionality of data processing system 102 may correspond to functions or interfaces with digital assistant application 120 executing on client device 104 or display device 108. Client device 104 and display device 108 may each include and execute separate instances of one or more components of digital assistant application 120. Client device 104 and display device 108 may otherwise have access to the functionality of the components of digital assistant application 120 on remote data processing system 102 via network 110. For example, display device 108 may include functionality of NLP component 124 and can access the remainder of the components of digital assistant application 120 via network 110 to data processing system 102. Data processing system 102, client device 104, and display device 108 can include and execute separate instances of components of digital assistant application 120. The digital assistant applications 120 accessible to or executing on client device 104 and display device 108 may each have different functionality. Client device 104 or display device 108 can execute a second digital assistant application 120 that interfaces with data processing system 102. Data processing system 102 can interface with first digital assistant server 106.

[0023] The client device 104 and the display device 108 each include at least one logical device, such as a computing device having a processor for communicating with the data processing system 102 and each other over the network 110. The client device 104 and the display device 108 may include instances of any of the components described with respect to the data processing system 102. The client device 104 may include a desktop computer, a laptop, a tablet computer, a personal digital assistant, a smartphone, a mobile device, a portable computer, a thin client computer, a virtual server, a speaker-based digital assistant, or other computing device. The display device 108 may include a smart television, a hybrid television, a connected television, a set-top box connected to a television, a digital video recorder, a monitor, a screen, or other computing device with display capabilities. The client device 104 may be communicatively coupled to the display device 108. For example, once paired, the client device 104 may function as a remote control for controlling various functions on the display device 108. The client device 104 may be part of or integrated with the display device 108.

[0024] The components of the system 100 can communicate via a network 110. The network 110 can include, for example, a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, an SDH (Synchronous Digital Hierarchy) network, an NFC (Near Field Communication) network, a local area network (LAN), a wireless network, or a wireline network, and combinations thereof. The network 110 can include wireless links, such as infrared channels or satellite bands. The topology of the network 110 can include a bus, star, or ring network topology. The network 110 can include a cellular network using any protocol used to communicate between mobile devices, including advanced mobile phone protocol (AMPS), time division multiple access (TDMA), code division multiple access (CDMA), global system for mobile communications (GSM), general packet radio services (GPRS), or universal mobile telecommunications system (UMTS). Different types of data may be transmitted over different protocols, or the same type of data may be transmitted over different protocols.

[0025] Network 110 is Bluetooth (registered trademark) , Bluetooth Low Energy (Bluetooth Low Energy)The communication link may include a short-range communication link (e.g., extending up to 30 meters) established between client device 104 and display device 108 using a cellular phone, a dedicated short-range communications (DSRC), or a near-field communications (NFC) protocol. Using such protocols, data processing system 102 can establish a communication link with one or more client devices 104 via interface 122. Data processing system 102 can establish a communication link with one or more display devices 108 via interface 122. The short-range communication link may be established between client device 104 and display device 108 via a pairing protocol.

[0026] The client device 104 may include a sensor 152, a speaker 150, an interface 122, and a transducer 154. The display device 108 may include a sensor 152, a speaker 150, an interface 122, and a transducer 154. The client device 104 and the display device 108 may include at least one sensor 152, at least one transducer 154, at least one audio driver, and at least one speaker 150. The sensor 152 may include a microphone or an audio input sensor. The sensor 152 may include at least one of a GPS sensor, a proximity sensor, an ambient light sensor, a temperature sensor, a motion sensor, an accelerometer, or a gyroscope. The sensor may include an occupancy sensor or a weight sensor. The transducer 154 may convert audio input into an electronic signal. The audio driver may include a script or program executed by one or more processors of the client device 104 or the display device 108 to control the speaker 150. The speaker 150 can render audio signals by converting electrical signals into audible waves. The client device 104 and the display device 108 can each include peripheral devices such as a keyboard, a pointing device, a monitor (built-in or separate), and headphones, among other devices.

[0027] Client device 104 and display device 108 can each run an instance of multimedia content application 160. Multimedia content application 160 can be associated with a particular first digital assistant server 106. Multimedia content application 160 can include a script or program installed on client device 104 or display device 108. Multimedia content application 160 can include a script or program executable via another application installed on client device 104 or display device 108, such as a web browser. Multimedia content application 160 can include an application programming interface (API) that can enable other applications (e.g., digital assistant application 120) on client device 104 or display device 108 to interface with multimedia content application 160. Multimedia content application 160 can play, stream, download, or otherwise access multimedia content. The multimedia content may correspond to one or more audiovisual content files in any format, such as MP3, ACC, OPUS, RTMP, RTP, MP4, FLV, WebM, ASF, ISMA, HEVC, H.264, VP8, VP9, ​​HLS, HDS, and SMIL, among others. The one or more audiovisual content files may include audio and / or visual content for playback on client device 104 or display device 108. Upon loading the one or more audiovisual content files, multimedia content application 160 can play or stream the multimedia content on client device 104 or display device 108.

[0028] The multimedia content application 160 can perform an authentication process with the first digital assistant server 106. The authentication process can include, via input or retrieval from storage, the identification of the account identifier and access code on the client device 104 or the display device 108. Upon receipt, the multimedia content application 160 can send the account identifier and access code to the first digital assistant server 106. The first digital assistant server 106 can then compare its own stored access code for the account identifier with the access code received from the client device 104 or the display device 108. In response to determining a match between the two access codes, the first digital assistant server 106 can send a successful authentication indicator to the client device 104 or the display device 108.

[0029] In response to receiving a successful authentication indicator, the multimedia content application 160 can retrieve a catalog of available multimedia content made available by the first digital assistant server 106 for the account identifier. The multimedia content application 160 can display the catalog of available multimedia content in a graphical user interface, with each element corresponding to each available multimedia content. Upon interaction with one of the elements of the graphical user interface, the multimedia content application 160 can send a request to the first digital assistant server 106 for the selected multimedia content. The first digital assistant server 106 can identify and provide one or more audiovisual content files corresponding to the selected multimedia content. The multimedia content application 160 can then store, play, or stream one or more audiovisual content files corresponding to the selected multimedia content on the client device 104 or the display device 108. Additional details regarding the functionality of the first digital assistant server 106 and the multimedia content application 160 in the context of the system 100 are described herein.

[0030] Applications, scripts, programs, or other components associated with data processing system 102 may be installed on client device 104 or display device 108. The applications may enable client device 104 or display device 108 to communicate input audio signals (and other data) to interface 122 of data processing system 102. The applications may enable client device 104 and display device 108 to drive components of client device 104 and display device 108 to render output audio signals.

[0031] The client device 104 and the display device 108 may be associated with an end user who inputs voice queries into the client device 104 or the display device 108 (via the sensors 152) as input audio signals and receives audio output in the form of computer-generated speech, which may be provided from the data processing system 102. In response to the input audio signals, the client device 104 and the display device 108 may receive action data structures for performing predetermined functions or actions. The interface 122 may receive data messages or provide data messages to the direct action API 126 of the data processing system 102 and enable communication between components of the system 100. The client device 104 and the display device 108 may include a user interface that enables a user to interact with components of the system 100.

[0032] The data processing system 102 may include at least one server having at least one processor. For example, the data processing system 102 may include multiple servers located in at least one data center or server farm. The data processing system 102 may determine the request and a trigger keyword associated with the request from the audio input signal. Based on the request and the trigger keyword, the data processing system 102 may generate or select response data. The response data may be audio-based or text-based. For example, the response data may include one or more audio files that, when rendered, provide an audio output or sound waves. The data in the response data may be referred to as a content item. The response data may include other content in addition to audio content (e.g., text content, video content, or image content). The response data may include various types of content, such as calendar information, messages, weather information, news information, or ticket information.

[0033] The data processing system 102 may include multiple servers logically grouped to facilitate distributed computing techniques. A logical group of servers may be referred to as a data center, server farm, or machine farm. The servers may be geographically distributed. A data center or machine farm may be governed as a single entity, or a machine farm may include multiple machine farms. The servers within each machine farm may be heterogeneous—one or more of the servers or machines may operate according to one or more types of operating system platforms. The data processing system 102 may include, for example, servers in a data center housed with associated storage systems in one or more high-density rack systems located within an enterprise data center. In this manner, a data processing system 102 with consolidated servers may improve system manageability, data security, system physical security, and system performance by locating the servers and high-performance storage systems on a localized, high-performance network. Centralizing all or some of the data processing system 102 components, including the servers and storage systems, and combining them with advanced system management tools enables more efficient use of server resources, which conserves power and processing requirements and reduces bandwidth usage. Each of the components of the data processing system 102 may include at least one processing unit, server, virtual server, circuit, engine, agent, appliance, or other logic device such as a programmable logic array configured to communicate with the data repository 140 and other computing devices.

[0034] The data processing system 102 may include a data repository 140. The data repository 140 may include one or more local or distributed databases and may include a database management system. The data repository 140 may include computer data storage or memory and may store one or more application identifiers 142, one or more entity metadata 144, one or more ranking policies 146, and at least one reference address 148. Each application identifier 142 may be an alphanumeric value corresponding to a multimedia content application 160 to be executed on the client device 104 or the display device 108. Each entity metadata 144 may correspond to an entity. Each reference address 148 may be identified or referenced. Details of the use and functionality of the one or more application identifiers 142, one or more entity metadata 144, one or more ranking policies 146, and reference address 148 are provided below.

[0035] Data repository 140 can include computer data storage or memory and can store, among other data, one or more parameters, one or more policies, response data, and templates. The parameters, policies, and templates can include information such as rules for voice-based sessions between client device 104, data processing system 102, and display device 108. The parameters, policies, and templates can include information for another digital assistant application 120 received via interface 122 from another source (e.g., data processing system 102, client device 104, and display device 108). For example, parameters, policies, and templates stored in data repository 140 of a digital assistant application 120 hosted on client device 104 or display device 108 can include parameters, policies, and templates from data repository 140 of a digital assistant application 120 accessible via client device 104 and display device 108, and vice versa. In this way, parameters, policies, and templates of different digital assistant applications 120 can be shared and used by each other. The response data may include content items for audio output or associated metadata, as well as input audio messages that may be part of one or more communication sessions with the client device 104 and the display device 108 .

[0036] The NLP component 124 may receive an input audio signal. The data processing system 102 may receive an input audio signal from the client device 104 or the display device 108 (e.g., via a transducer 154 or a sensor 152) contained in a data packet. The data packet may include a device identifier associated with the client device 104 or the display device 108. The data packet may include an application identifier of a multimedia content application 160 executable on the client device 104 or the display device 108. The functionality of the NLP component 124 may be divided among the data processing system 102, the client device 104, and the display device 108. For example, the NLP component 124 executing on the client device 104 or the display device 108 may package the input audio signal detected at the sensor 152 into a data packet and send the data packet to the data processing system 102 for further processing by the NLP component 124 executing on the data processing system 102.

[0037] The NLP component 124 can convert the input audio signal into recognized text by comparing the input audio signal against a stored exemplar set of audio waveforms and choosing the closest match. The exemplar waveforms can be generated across a large set of input audio signals. Once the input audio signal is converted into recognized text, the NLP component 124 can, for example, go through a training phase to match the text to actions or words associated with the output audio signal.

[0038] From the input audio signal, the NLP component 124 can identify at least one request. The request can indicate the intent or subject of the input audio signal. The request can indicate a type of action that may be taken. For example, the NLP component 124 can parse the input audio signal to identify at least one request to play multimedia content (e.g., "Okay, play the 2015 Bat Movie"). The request can include at least one word, phrase, root or part word, or derivative word that indicates the action to be taken. The request can include a trigger keyword, such as "Okay" or "Get started." The NLP component 124 can detect the trigger keyword in the request. In response to detecting the trigger keyword, the NLP component 124 can identify the intent, subject, and type of action to be taken from the input audio signal.

[0039] The NLP component 124 can identify at least one entity from the request. The at least one entity can correspond to at least one word, phrase, root or subword, or derivative in the request. The at least one entity may include descriptors for the multimedia content, such as title, author, distributor, original publication date, language, genre, duration, and other relevant keywords (e.g., actor names). For example, an entity for the "Bat Movie I" film released in 2005 may include, among other information, "Bat Movie I" as the title, "Ms. Director" as the author, "June 15, 2005" as the original publication date, "English" as the language, "action" as the genre, and "140 minutes" as the duration. The NLP component 124 can determine that at least one word, phrase, root or subword, or derivative in the request corresponds to the at least one entity. To perform the determination, the NLP component 124 can access a semantic knowledge graph. A semantic knowledge graph can specify a set of words, phrases, roots or sub-words, or derivatives that relate to at least one entity. A semantic knowledge graph can include nodes connected to each other via edges. A node may be associated with a word, phrase, root or sub-word, or derivative and may be marked as related or unrelated to at least one entity. Each edge connecting two nodes can represent a relationship between the two. Continuing with the previous example, a node for "Bat Movie" in the semantic knowledge graph may be marked as related to the entity and may have an edge to a node labeled "Bat Movie I," also labeled as related to the entity, indicating a relationship between the two terms. Using the semantic knowledge graph, the NLP component 124 can determine that at least one word, phrase, root or sub-word, or derivative in the request corresponds to at least one entity.

[0040] The NLP component 124 can determine whether at least one request corresponds to a query for multimedia content based on the entities. In response to determining that at least one word, phrase, root or subword, or derivative in the request corresponds to at least one entity, the NLP component 124 can determine that at least one request corresponds to a query for multimedia content. In response to determining that none of the words, phrases, root or subwords, or derivatives in the request correspond to at least one entity, the NLP component 124 can determine that at least one request does not correspond to a query for multimedia content. A determination that at least one request corresponds to a query for multimedia content can trigger the functions of the search coordinator component 132, the content indexer component 134, the ranking 136, and the response handler component 138, as described in more detail below.

[0041] The audio signal generator component 130 can retrieve information from a data repository 140, where the information can be stored as part of the response data. The audio signal generator component 130 can query the data repository 140, for example, to select or otherwise identify a response phrase or content item from the response data. The audio signal generator component 130 can generate or otherwise retrieve an output signal that includes the content item. The data processing system 102 can execute the audio signal generator component 130 to generate or create an output signal that corresponds to the content item or request. For example, when a request is fulfilled, the audio signal generator component 130 can generate an audio output signal that includes the phrase "action completed."

[0042] Interface 122 may be a data interface or a network interface that allows components of system 100 to communicate with each other. Interface 122 of data processing system 102 may provide or transmit one or more data packets containing action data structures, audio signals, or other data to client device 104 or display device 108 over network 110. For example, data processing system 102 may provide an output signal from data repository 140 or audio signal generator component 130 to client device 104. Data processing system 102 may instruct client device 104 or display device 108 via data packet transmission to perform a function indicated in the action data structure. The output signal may be obtained, generated, converted into one or more data packets (or other communication protocol), or transmitted as such from data processing system 102 (or other computing device) to client device 104 or display device 108.

[0043] The direct action API 126 of the data processing system 102 can, for example, generate an action data structure based on the request. The action data structure can include data or instructions for performing a specified action to fulfill the request. In some implementations, the action data structure can be a JSON-formatted data structure or an XML-formatted data structure.

[0044] Depending on the action specified in the request, the direct action API 126 can execute code or an interaction script that specifies the parameters needed to fulfill the request. An action data structure can be generated in response to the request. The action data structure can be included in a message sent to or received by the client device 104 or the display device 108. The direct action API 126 can package the request into the action data structure for transmission to the display device 108. The direct action API 126 can access the device identifier from the response data to determine which client device 104 or display device 108 is associated with the user who generated the request. Once received, the display device 108 can process the action data structure and perform the indicated action. The direct action API 126 can package the request into the action data structure for execution by the client device 104 or the display device 108. Once received, the client device 104 can process the action data structure using the digital assistant application 120 or one or more applications running on the client device 104.

[0045] The action data structure may include information for completing the request. For example, the action data structure may be an XML or JSON formatted data structure that includes attributes used in completing or otherwise fulfilling the request. The attributes may include the location of the display device 108, the location of the client device 104, the permission level of a user associated with the client device 104, a device identifier, an interface identifier, a vehicle state, or a request state. In some implementations, the request state includes one or more attributes that must be satisfied before the action is fulfilled. For example, with the request "Okay, change the song," the request state may have the attribute {requestor:[authorized, passenger]}, which may indicate that the request should be made by an explicitly authorized user.

[0046] The direct action API 126 can retrieve templates from the data repository 140 to determine which fields or attributes should be included in the action data structure. The direct action API 126 can determine the required parameters and package the information into the action data structure. The direct action API 126 can retrieve content from the data repository 140 to obtain information for the attributes of the data structure.

[0047] The direct action API 126 can fill fields with data from the input audio signal. The direct action API 126 can fill fields with data from the client device 104 or the display device 108, or another source. The direct action API 126 can prompt the user for additional information when filling fields. Templates can be standardized for different types of actions that respond to messages and perform functions within the client device 104 or the display device 108. An action data structure can be initially generated by the direct action API 126 executed by the remote data processing system 102. The remote data processing system 102 can send the action data structure to the client device 104 or the display device 108, and the client device 104 or the display device 108 can add fields and attributes to the action data structure.

[0048] The direct action API 126 can obtain response data (or parameters or policies) from the data repository 140 and data received with end-user consent from the client device 104 or display device 108. The response data (or parameters or policies) can be included in an action data structure. When content included in the action data structure includes end-user data used for authentication, the data can be passed through a hashing function before being stored in the data repository 140.

[0049] The data processing system 102 may include, interface with, or otherwise communicate with a search coordinator component 132. The search coordinator component 132 may generate a query command for sending to the client device 104 or the display device 108. The generation of the query command by the search coordinator component 132 may be in response to a determination in the request that the query corresponds to a query for multimedia content. The query command may include an entity identified from the response. The query command may include instructions for the client device 104 or the display device 108 to process the query command. The instructions may indicate which multimedia content application 160 on the client device 104 or the display device 108 should perform a search for the entity. To generate the instructions, the search coordinator component 132 may access the data repository 140 to identify one or more application identifiers 142 to include in the query command. The data repository 140 may identify the one or more application identifiers 142 according to a device identifier associated with the client device 104 or the display device 108. Each application identifier 142 may be labeled by frequency of use by the client device 104 or display device 108. The search coordinator component 132 may insert one or more application identifiers 142 in the initial data packet into instructions for a query command. The search coordinator component 132 may identify a subset of application identifiers 142 for the client device 104 and display device 108 based on frequency of use by the client device 104 and display device 108.For example, the search coordinator component 132 may determine that the display device 108 uses multimedia content applications 160 "A" and "B" more frequently than multimedia content applications 160 "C," "D," and "E." In this example, the search coordinator component 132 may select application identifiers 142 corresponding to the multimedia content applications 160 for insertion into the query command. Each application identifier 142 in the query command may indicate which multimedia content application 160 on the client device 104 or the display device 108 should perform a search for the entity. After generating the query command, the search coordinator 132 may transmit the query command to the client device 104 or the display device 108.

[0050] Receiving the query command can cause the client device 104 or display device 108 to perform a search for the entity in the multimedia content application 160. In response to receiving the query command, the client device 104 or display device 108 can parse the query command to identify the entity. The client device 104 or display device 108 can parse the query command to identify one or more application identifiers 142. The client device 104 or display device 108 can identify the multimedia content application 160 corresponding to the one or more application identifiers 142 in the query command. The client device 104 or display device 108 can start executing the multimedia content application 160 identified from the one or more application identifiers 142 in the query command. To perform the search, the client device 104 or display device 108 can provide at least one entity in the query command to each multimedia content application 160 via the application programming interface for the multimedia content application 160. Once the entity is provided, each multimedia content application 160 can send a query request to the associated first digital assistant server 106.

[0051] In parallel with generating and sending the query command to the client device 104 or the display device 108, the search coordinator component 132 can generate a query request to send to the first digital assistant server 106. The generation of the query request by the search coordinator component 132 can be in response to a determination in the request that the query corresponds to a query for multimedia content. The generation of the query request by the search coordinator component 132 can be in response to a call by the content indexer component 134, as described below. The query request can include entities identified from the response. The query request can include instructions for the first digital assistant server 106 to process the query request. The search coordinator 132 can identify which multimedia content provider should send the query request from one or more application identifiers 142 to the client device 104 or the display device 108. After generating the query request, the search coordinator 132 can send the query request to the first digital assistant server 106.

[0052] Receiving a query request from the client device 104, the display device 108, or the search coordinator component 132 can cause the first digital assistant server 106 to perform a search for the entity. When the client device 104 or the display device 108 may have initialized the multimedia content application 160 to send a query request, the query request from the client device 104 or the display device 108 may be received after the query request from the search coordinator component 132. The first digital assistant server 106 can access the content data repository 162 in response to the query request. The content data repository 162 can include one or more entity metadata 144 and one or more ranking policies 146. Each entity metadata 144 can correspond to an entity. Each entity metadata 144 on the data repository 162 can include one or more entries for the multimedia content corresponding to the entity, such as title, author, distributor, first publication date and time, language, genre, duration, and other related keywords. For example, one entity metadata 144 can correspond to "Bat Movie I," and another entity metadata 144 can correspond to "Bat Movie II." Each entity metadata 144 can be associated with a reference address 148 (e.g., a uniform resource locator). The reference address 148 can include a host name, a file path name, and a query parameter. The host name can correspond to the first digital assistant server 106. The file path name can correspond to one or more audiovisual files for the multimedia content corresponding to the entity. The query parameter can include an attribute-value pair. The query parameter can be dynamically generated by the first digital assistant server 106 and can vary for each client device 104, display device 108, or search coordinator component 132.

[0053] Based on the entity from the query request, the first digital assistant server 106 can identify a reference address 148 corresponding to the entity. The first digital assistant server 106 can traverse through the content data repository 162. For each entity metadata 144, the first digital assistant server 106 can compare the entity with the entity metadata 144. For example, the first digital assistant server 106 can compare the entity "Bat Movie" with any of the entries for the entity metadata 144, such as the title. The first digital assistant server 106 can determine whether one or more entries in the entity metadata 144 match the entity from the query request. The first digital assistant server 106 can determine that there is no match between the entity and the entity metadata 144. In response to determining that there is no match, the first digital assistant server 106 can continue to traverse the content data repository 162. The first digital assistant server 106 can determine that there is no more entity metadata 144 to traverse. In response to determining that there is no further entity metadata 144 to review, the first digital assistant server 106 can determine that the entity does not exist on the content data repository 162. The first digital assistant server 106 can send a null response. The null response can indicate that the first digital assistant server 106 does not have multimedia content corresponding to the entity.

[0054] Conversely, the first digital assistant server 106 can determine that there is a match between the entity and the entity metadata 144. In response to determining a match, the first digital assistant server 106 can determine that the entity exists on the content data repository 162. The first digital assistant server 106 can identify a reference address 148 corresponding to the entity metadata 144 that matches the entity. The first digital assistant server 106 can generate a search response. The search response can indicate that the first digital assistant server 106 has multimedia content corresponding to the entity. The search response can include a reference address 148 corresponding to the multimedia content for the entity.

[0055] The first digital assistant server 106 can continue to traverse through the content data repository 162 to identify additional matches between the entity of the query request and the entity metadata 144. With additional matches, the first digital assistant server 106 can iteratively identify reference addresses 148 corresponding to the entity metadata 144 that matches the entity and include the reference addresses 148 in the search response. The first digital assistant server 106 can determine that there is no more entity metadata 144 to traverse. In response to determining that there is no more entity metadata 144 to traverse, the first digital assistant server 106 can send a response to the client device 104, display device 108, or search coordinator component 132 that sent the query search.

[0056] The data processing system 102 can include, interface with, or otherwise communicate with a content indexer component 134. The content indexer component 134 can access the data repository 140 of the data processing system 102 or the first digital assistant server 106 to identify a reference address 148 corresponding to the entity. The content indexer component 134 can access the data repository 140 before accessing the first digital assistant server 106 to identify the reference address 148. The content indexer component 134 can traverse through the data repository 140. In addition to the application identifier 142, the data repository 140 can include one or more entity metadata 144 and one or more ranking policies 146. The entity metadata 144 and ranking policies 146 on the data repository 140 can be maintained separately and independently from the entity metadata 144 and ranking policies 146 on the content data repository 162. The entity metadata 144 and ranking policies 146 on the data repository 140 may be from previous search responses from multimedia content providers. The content indexer component 134 may maintain a timer to track the elapsed time for each entity metadata 144 and associated reference address 148 in storage in the data repository 140.

[0057] The content indexer component 134 may traverse through the content data repository 140. For each entity metadata 144, the content indexer component 134 may compare the entity to the entity metadata 144. For example, the content indexer component 134 may compare the entity "Bat Movie" to any of the entries for the entity metadata 144, such as the title. The content indexer component 134 may determine whether one or more entries in the entity metadata 144 match the entity from the query request. The content indexer component 134 may determine that there is no match between the entity and the entity metadata 144. In response to determining that there is no match, the content indexer component 134 may continue traversing the content data repository 140. The content indexer component 134 may determine that there is no more entity metadata 144 to traverse. In response to determining that there is no more entity metadata 144 to traverse, the content indexer component 134 may determine that the entity does not exist on the content data repository 140. In response to determining that the entity does not exist on the content data repository 140, the content indexer component 134 can invoke the search coordinator component 132 to send a query request to the first digital assistant server 106 to retrieve the reference address 148.

[0058] Conversely, the content indexer component 134 may determine that there is a match between the entity and the entity metadata 144. In response to determining a match, the content indexer component 134 may determine that the entity exists on the content data repository 140. The content indexer component 134 may identify a reference address 148 that corresponds to the entity metadata 144 that matches the entity. The content indexer component 134 may store the reference address 148 that corresponds to the entity.

[0059] The content indexer component 134 can identify the elapsed time for the identified reference address 148. The content indexer component 134 can compare the elapsed time with a preset expiration time. Because the first digital assistant server 106 may periodically update or change the catalog of available multimedia content, the preset expiration time can range from one hour to one month. The preset expiration time can correspond to the amount of time the reference address 148 is valid for an entity on the first digital assistant server 106. The content indexer component 134 can determine that the elapsed time is less than the preset expiration time. In response to determining that the elapsed time is less than the preset expiration time, the content indexer component 134 can store the reference address 148 corresponding to the entity. The content indexer component 134 can terminate the access of the first digital assistant server 106 to search for a reference address corresponding to the entity.

[0060] Meanwhile, the content indexer component 134 can determine that the elapsed time is greater than or equal to a preset expiration time. In response to determining that the elapsed time is greater than or equal to the preset expiration time, the content indexer component 134 can replace the reference address 148 from the data repository 140. To update the data repository 140, the content indexer component 134 can invoke the search coordinator component 132 to send a query request to the first digital assistant server 106 to retrieve the reference address 148. The content indexer component 134 can receive a search response from the first digital assistant server 106. The search response can include a new reference address 148 for the entity. In response to receiving the reference address 148 from the first digital assistant server 106, the content indexer component 134 can save the reference address 148 on the data repository 140 and associate the newly saved reference address 148 with the entity metadata 144 that matches the entity. The content indexer component 134 can receive a null response from the first digital assistant server 106. In response to receiving the null response, the content indexer component 134 can remove the reference address 148 from the data repository 140.

[0061] The content indexer component 134 can continue to traverse through the content data repository 140 to identify additional matches between the entities of the query request and the entity metadata 144. With additional matches, the content indexer component 134 can iteratively identify reference addresses 148 corresponding to the entity metadata 144 that match the entities and include the reference addresses 148 in the search response. The content indexer component 134 can determine that there is no more entity metadata 144 to traverse. In response to determining that there is no more entity metadata 144 to traverse, the content indexer component 134 can terminate the search for additional ranking policies 146 in the data repository 140. After traversing, the content indexer component 134 can invoke the search coordinator component 132 to send a query request to the first digital assistant server 106 to retrieve the reference addresses 148. The invocation of the search coordinator component 132 may be independent of the comparison of the elapsed time with the preset expiration time described above.

[0062] After invoking the search coordinator component 132 to send the query request, the content indexer component 134 can receive a search response from the first digital assistant server 106. Receiving a search response from the first digital assistant server 106 can indicate that the first digital assistant server 106 has access to the entity. The search response can include a reference address 148 corresponding to the entity. The content indexer component 134 can parse the search response to identify the reference address 148. In response to identifying a match between the entity and the entity metadata 144 on the data repository 140, the content indexer component 134 can compare the reference address 148 from the data repository 140 with the reference address 148 from the first digital assistant server 106. The content indexer component 134 can determine that the reference address 148 from the data repository 140 is different from the reference address 148 from the first digital assistant server 106. A difference in ranking policy 146 may mean that the first digital assistant server 106 is updating the reference address 148 for the entity. In response to determining that the ranking policy 146 is different, the content indexer component 134 can replace the reference address 148 stored in the data repository 140 with the reference address 148 from the first digital assistant server 106. The content indexer component 134 can determine that the reference address 148 from the data repository 140 is identical to the reference address 148 from the first digital assistant server 106. In response to determining that the ranking policy 146 is identical, the content indexer component 134 can proceed to invoke the response handler component 138 to send the query results to the client device 104 or the display device 108, as described below.

[0063] The content indexer component 134 can receive a null response from the first digital assistant server 106. Receiving a null response from the first digital assistant server 106 can indicate that the first digital assistant server 106 no longer has the entity. For example, the first digital assistant server 106 may have updated its catalog of available multimedia content to exclude the entity. In response to receiving the null response and identifying a match between the entity and the entity metadata 144 on the data repository 140, the content indexer component 134 can remove the reference address 148 from the data repository 140. The content indexer component 134 can proceed to invoke the response handler component 138 to send the query results to the client device 104 or the display device 108, as described below. In some cases, the content indexer component 134 may not receive a response from the first digital assistant server 106 within a predetermined amount of time after sending the query response, for example, due to a problem with the network 110 or an outage in the first digital assistant server 106. In such cases, the content indexer component 134 can maintain the ranking policy 146 in the data repository 140.

[0064] The data processing system 102 can include, interface with, or otherwise communicate with a ranking engine component 136. The ranking engine component 136 can use one or more ranking policies 146 stored in the data repository 140. The ranking engine component 136 can use a ranking decision function, such as the ranking decision functions or blocks shown in FIGS. 2 and 3. The ranking decision function can be based on reliability score, latency, time to completion, flags, priority, preconfigured preferences, protocols, or other factors. For example, the second digital assistant 120 can be prioritized over the first digital assistant.

[0065] The ranking engine component 136 can process or parse a query from the client device 104 or the display device 108 to identify or generate one or more signals for the query. The ranking engine component 136 can select a decision function from multiple decision functions stored in the data repository 140. The decision function may be referred to as or stored as a policy 146. The decision function (or policy 146) can vary based on the signal for the query. For example, a query for media content can generate a signal indicating that a first digital assistant should be selected for the results. A query for search results from a database accessible to the data processing system 102 can generate a signal to prioritize results from a second digital assistant.

[0066] The ranking decision function can take into account the reliability score. For example, the first digital assistant or the second digital assistant can return a reliability score along with the results. For example, if the second digital assistant returns results first and provides a high reliability score (e.g., greater than 0.8 out of 1), the data processing system 102 may not have to wait for any results to come from the first digital assistant and may proceed to send the results from the second digital assistant—thereby overriding the first digital assistant.

[0067] The data processing system 102 may include, interface with, or otherwise communicate with a response handler component 138. In response to identifying a reference address 148 corresponding to an entity in the request, the response handler component 138 may send or provide the reference address 148 to the client device 104 or the display device 108. The response handler component 138 may generate instructions for rendering a content selection interface at the client device 104 or the display device 108. The content selection interface may include one or more selection elements and placeholder elements (sometimes collectively referred to as "display cards"). The one or more selection elements may correspond to the reference address 148 for the entity included in the request of the client device 104 or the display device 108. The search performed by the multimedia content application 160 may not be complete before the response handler component 138 sends the reference address 148 to the client device 104 or the display device 108. Thus, the content selection interface may initially filter out results from a search performed by a multimedia content application 160 executing on the client device 104 or the display device 108. The response handler component 138 may provide instructions to the client device 104 or the display device 108 along with the reference address 148 for rendering the content selection interface.

[0068] After presentation of the content selection interface on the client device 104 or the display device 108, interaction with a placeholder element may cause the client device 104 or the display device 108 to determine whether the search performed by the multimedia content application 160 is complete. The client device 104 or the display device 108 may determine that the search performed by the multimedia content application 160 is complete. In response to determining that the search is complete, the client device 104 or the display device 108 may modify the content selection interface to include additional selection elements. Each additional selection element may correspond to a reference address 148 found during the search performed by the multimedia content application 160 executing on the client device 104 or the display device 108. Interaction with a selection element of a content selection element may cause the client device 104 or the display device 108 to launch the multimedia content application 160 (or another application running on the client device 104 or the display device 108) to play one or more audiovisual files that correspond to the reference address 148 of the associated selection element.

[0069] Meanwhile, the client device 104 or the display device 108 may determine that the search performed by the multimedia content application 160 is not yet complete. In response to determining that the search is not yet complete, the client device 104 or the display device 108 may wait a predetermined amount of time. During the predetermined amount of time, the client device 104 or the display device 108 may determine that the search performed by the multimedia content application 160 is complete. In response to determining that the search is complete, the client device 104 or the display device 108 may modify the content selection interface to include additional selection elements. The client device 104 or the display device 108 may determine that a predetermined amount of time has passed and that the search is not yet complete. In response to determining that a predetermined amount of time has passed and that the search is not yet complete, the client device 104 or the display device 108 may modify the content selection interface to remove the placeholder element.

[0070] FIG. 2 illustrates an exemplary operation of a system for coordinating overlapping processing of audio queries. System 200 can include one or more components or functions shown in FIG. 1 or FIG. 5. System 200 can include data processing system 102. System 200 can include, interface with, or otherwise communicate with or utilize first digital assistant server 106. System 200 can include, interface with, or otherwise communicate with or utilize second digital assistant server 120. First digital assistant server 106 (sometimes referred to as a first digital assistant, a first digital assistant component, or a first digital assistant application) can run independently of second digital assistant application 120 (sometimes referred to as a second digital assistant, a second digital assistant component, or a second digital assistant application). One of the first digital assistant or the second digital assistant can be referred to as a local digital assistant, and the other of the first digital assistant or the second digital assistant can be referred to as a remote digital assistant. For example, the first digital assistant server 106 may be referred to as a remote digital assistant, and the second digital assistant application 120 may be referred to as a local digital assistant. In some cases, the first digital assistant server 106 may be referred to as a local digital assistant, and the second digital assistant application 120 may be referred to as a remote digital assistant.

[0071] In ACT 202, a client device 104 (which may include, be integrated with, or otherwise communicatively coupled to a display device 108) may receive an input audio signal. The client device 104 may be referred to as a computing device or a client computing device. The client device 104 may receive an audio signal from a user. The audio signal may include voice input provided by a user of the client device 104. The voice input may include a request, query, command, instruction, or response. A microphone (e.g., sensor 152 or transducer 154) of the client device 104 may detect or receive the audio input.

[0072] In ACT 222, the client device 104 can transmit or provide the audio input or data corresponding to the audio input to a server. The client device 104 can transmit the audio input or data corresponding to the audio input to a streaming audio server (“S3”) 204 or the data processing system 102. The client device 104 can convert the audio input received in ACT 202 into a format configured for transmission to the server. The client device 104 can, for example, preprocess the audio input and convert the audio input into a digital format configured for transmission over a data network. The client device 104 can transmit the audio input or data corresponding to the audio input in response to identifying a request in the audio input. The request can include a keyword, hotword, wake-up word, token, or other indication in the audio input that indicates to the client device 104 that the user has a request for content or an action to be performed by the client device 104 or one or more servers. The client device 104 can transmit the audio input or data corresponding to the audio input in response to authorization or authentication.

[0073] The client device 104 can send an input audio signal as a request to the streaming audio server 204. The request can include a request to convert the audio, speech, or voice into text or some other format that facilitates downstream transmission or processing of the audio input. The S3 server 204 can transcribe the voice into text. In ACT 224, the S3 server 204 can send the transcription results back to the client device 104. In some cases, the S3 server 204 can send the transcription results to the data processing system 102 for further processing. In some cases, the S3 server can send the transcription results to the device repository 220 for storage.

[0074] In ACT 226, the S3 server 204 can send the transcribed results to the data processing system 102. The transcribed results can include data packets having a query that corresponds to the input audio signal received in ACT 202. The S3 server 204 can forward the text query to the data processing system 102. In some cases, the client device 204 can convert the audio input into data packets and send the data packets to the data processing system 102 without sending the audio input to the S3 server 204.

[0075] The data processing system 102 may receive, via the interface, one or more data packets corresponding to an audio signal detected by the client device 104. The data processing system 102 may receive the one or more data packets from the client device 104 or the S3 server 204. The one or more data packets may include a text query, or the data processing system 102 may process the one or more data packets to identify a query. For example, the data processing system 102 may generate a query based on the received data packets corresponding to the audio signal.

[0076] The data processing system 102 can send queries to one or more digital assistants for processing. The data processing system 102 can provide queries to one or more digital assistants for processing. The data processing system 102 can process queries using one or more digital assistants run or managed by the data processing system 102. The data processing system 102 can provide queries to one or more digital assistants external to the data processing system 102. A digital assistant can be external to the data processing system 102 if it is geographically remote from the data processing system 102 or governed or managed by a different entity than the data processing system 102. A digital assistant may be referred to as external to the data processing system 102 if it is a different type of digital assistant.

[0077] Data processing system 102 can send, provide, or otherwise communicate or communicate queries to multiple digital assistant components for processing. One or more of the multiple digital assistants can process the query and generate a response to the query. Data processing system 102 can include an internal digital assistant, such as second digital assistant application 120. Second digital assistant application 120 can process the query and generate a response to the query. Data processing system 102 (or interface 122) can send the query to second digital assistant application 120 for processing. Data processing system 102 (or interface 122) can send the query to first digital assistant server 106 outside of data processing system 102 for processing. Data processing system 102 (e.g., via interface 122 or second digital assistant application 120) can simultaneously send queries to multiple digital assistants for processing at the same time or in an overlapping manner. For example, the second digital assistant application 120 can receive a query and can send the query to the first digital assistant server 106 to generate a response to the query. The second digital assistant application 120 can send the query to the first digital assistant server 106 before the second digital assistant application 120 processes the query or generates a response to the query, thereby reducing the latency or delay caused by the second digital assistant application 120. In some cases, to further reduce latency or delay, the interface 122 can simultaneously send the query to the second digital assistant application 120 and the first digital assistant server 106 for simultaneous or overlapping processing.

[0078] The data processing system 102 (e.g., via interface 122) can send a query to the first digital assistant server 106 before the second digital assistant application 120 of the data processing system 102. The data processing system 102 can send a query to the first digital assistant server 106 after the query is sent to the second digital assistant application 120. The data processing system 102 can send a query to the first digital assistant server 106 after the second digital assistant application 120 receives the query. The data processing system 102 can utilize an application programming interface (API) to send or write queries, i.e., original queries received by the data processing system 102 from the client device 104, to the first digital assistant server 106. In some cases, the data processing system 102 can provide, send, relay, or forward the original query received from the client device 104 to the first digital assistant server 106. In some cases, the data processing system 102 can process or modify the original query before sending it to the first digital assistant server 106.

[0079] The data processing system 102 can process or parse the text query using the second digital assistant and can generate second digital assistant candidate results 204. The text query can be passed to the first digital assistant server 106, which can be running in a cloud server system, at 208. The first digital assistant server 106 can be separate from the data processing system 102 running the second digital assistant. The data processing system can send the query to each of multiple digital assistants via independent branches to have the multiple digital assistants process the query in an overlapping manner. An independent branch can refer to a logical branch in a logic flow, i.e., a separate processing thread in which the separate digital assistants process the query separately to generate a response but do not communicate with each other. The separate digital assistants can process the query separately to generate results in an overlapping manner. The first digital assistant candidate results 206 can be provided to the data processing system 102 in the response 212. The first digital assistant server 106 can be established by a different administrator who established the second digital assistant application 120. For example, a different company, entity, or manufacturer may be responsible for designing and building the first digital assistant server 106 compared to the second digital assistant application 120. The first digital assistant server 106 and the second digital assistant application 120 can run independently of each other to generate responses or results to queries.

[0080] The first and second digital assistants may have access to different types of information or data. The first digital assistant server 106 may be authorized to access databases or data sources that the second digital application 120 may not be authorized to access or may be prohibited from accessing. For example, the first digital assistant server 106 may have access to a multimedia repository that stores movies or songs. Access to this multimedia repository may include authorization via credentials or subscriptions. The first digital assistant 106 may have access to authentication information via the client device 104 or subscription information associated with the user profile of the client device 104, but these credentials may not be transferred to or available to the second digital assistant application 120.

[0081] The data processing system 102 can receive results for the query. The data processing system 102 can receive candidate results for the query from one or more digital assistants. The data processing system 102 can receive a second digital assistant candidate result 210. In ACT 212, the data processing system 102 can receive a first digital assistant candidate result 206. The first digital assistant candidate result 206 and the second digital assistant candidate result 210 can be in a similar format to each other or in a different format. The first digital assistant candidate result 206 and the second digital assistant candidate result 210 can be from different sources and can be generated using different techniques, functions, parameters, selection criteria, or processes.

[0082] Results from the first or second digital assistant component may include content items, references, links, images, text, video, audio, or other types of results that respond to or are based on a query from an input audio signal received by the client device 104. The results may include data from various sources. The results may include streaming data. The results may include references or links to streaming content, such as streaming audio or streaming video. The results may include suggestions that respond to the query. The results may include deep links to content. The results may include authentication or credential information configured to allow the client device 104 to access content stored or identified in the results.

[0083] At 214, data processing system 102 can perform a ranking decision to select candidate result 204 or 206. The second digital assistant application 120 can perform the ranking decision. The ranking engine component 136 of data processing system 102 or the second digital assistant application 120 can perform the ranking decision. For example, the ranking engine component 136 can receive a candidate result or an indication that candidate results from multiple digital assistant components are available. The ranking engine component 136 can wait to perform the ranking decision until it receives an indication that results have been received from all of the digital assistant components to which data processing system 102 sent a query.

[0084] Data processing system 102 (e.g., via ranking engine component 136) can determine, based on the ranking determination function, that it should select a response or result from one of first digital assistant application server 106 (e.g., first digital assistant component) or second digital assistant application 120 (e.g., second digital assistant component). Data processing system 102 can determine that it should select second digital assistant candidate result 210 or first digital assistant candidate result 206. Data processing system 102 can determine that it should select one of the results from one of second digital assistant candidate result 210 or first digital assistant candidate result 206. For example, data processing system 102 can determine that it should select a response or result from the second digital assistant component rather than a response or result from the first digital assistant component.

[0085] Data processing system 102 can utilize various ranking decision functions or techniques to select a result or response, some examples of which are described in the following paragraphs. Data processing system 102 can select a ranking decision function or technique based on a signal related to the input audio signal or a query corresponding to the input audio signal. Data processing system 102 can select a digital assistant or response based on a signal related to the input audio signal or a query corresponding to the input audio signal. The signal can indicate whether the query is for a certain type of media, i.e., an internet search query, a news request, a weather request, a video request, a movie request, a music request, or another type of request. Data processing system 102 can select a response or result based on information related to the digital assistant that provided the result, based on information related to the result itself, based on information related to client device 104, based on information related to a profile of a user associated with client device 104, based on the type of result, based on a data source used to provide the result, based on a data source to which the result links or points, the data size of the result, or other factors.

[0086] Data processing system 102 can select a result or response based on pre-configured preferences. The pre-configured preferences can be stored or configured on client device 104, data processing system 102, or first digital assistant server 106. For example, second digital assistant application 120 can execute at least partially on client device 104. Second digital assistant application 120 can include an interface located on client device 104. Second digital assistant application 120 can be associated with an administrator, manufacturer, or provider of client device 104. Second digital assistant application 120 can be pre-configured to be a preferred digital assistant. Being a preferred digital assistant for client device 104 can cause data processing system 102 to prioritize selecting results from second digital assistant candidate results 210 over results from first digital assistant candidate results 206. Data processing system 102 can rank results from second digital assistant 120 higher than results from first digital assistant 106. Data processing system 102 can weight results from second digital assistant application 120 more heavily than results from first digital assistant server 106. By ranking, weighting, or otherwise prioritizing results from second digital assistant application 120, data processing system 102 may be more likely to select results from second digital assistant 120 compared to results from first digital assistant server 106. Configuration information related to preferences, rankings, weightings, or other prioritizations may be stored in database 140 (e.g., policy 146 data structure) or on client device 104 or other data storage device.

[0087] The data processing system 102 can select a result or response based on information about or related to the response or result itself. For example, the data processing system 102 can determine, based on a ranking decision function, that the response should be selected based on whether it contains audio-only, video and audio, video-only, text-only, image-only, or other multimedia content. The data processing system 102 can, for example, select a first response based on an attribute or characteristic associated with a second response. For example, the data processing system 102 can select a first response that may include video in response to a second response that only includes audio.

[0088] The data processing system 102 can select a result, response, or digital assistant based on the data source for the result or digital assistant. For example, the data processing system 102 can determine, based on a ranking decision function, that the data processing system 102 should select a second response from a second digital assistant component based on the second digital assistant component accessing data stored in memory of the client device 104. The data processing system 102 can make this determination to reduce network bandwidth usage or latency or data storage as a result of data being stored on the client device 104 that presents or renders the data.

[0089] The data processing system 102 can select a response or result based on information related to how the result was selected. For example, the selection technique or process used to select a result can output a reliability score for the result. The reliability score can reflect, indicate, or communicate a level of reliability related to whether the result is accurate, reliable, or responsive to the query. For example, some results may have a high reliability score associated with the result (e.g., 0.7, 0.8, 0.9, or 1), while another result may have a low reliability score (e.g., 0.4, 0.3, 0.2, or less). The reliability score can indicate the likelihood that the result will match or respond to the query. The reliability score can indicate the likelihood that a user of the client device 104 will interact with the result or the content item corresponding to the result. Thus, the data processing system 102 can select a response or result from the second digital assistant candidate results 210, for example, based on the reliability score for the response from the second digital assistant component.

[0090] A result may have a low or high reliability score based on one or more factors. For example, a digital assistant component may have access to a finite or predetermined set of data or data sources. The digital assistant component may be limited in terms of results based on limited access to data sources. Thus, the digital assistant component may determine that the best matching result from a limited set of data sources may still be associated with a low reliability score.

[0091] In decision block 216, data processing system 102 (or ranking engine component 136) can determine whether the results from the second digital assistant are preferred. If data processing system 102 determines in decision block 216 that the results from the second digital assistant are preferred, the data processing system proceeds to HTML adapter 218. HTML adapter 218 can convert the results from the second digital assistant, which may be coded in HTML, into a format compatible with S3 server 204. HTML adapter 218 of data processing system 102 can transfer the converted results from second digital assistant candidate results 210 to the S3 server.

[0092] However, if in decision block 216 the data processing system 102 determines that the results from the second digital assistant are unfavorable, the data processing system 102 can proceed to the S3 server 204 and provide a response 212 from the first digital assistant server 106.

[0093] The data processing system 102 (e.g., the ranking engine component 136) can be configured with various logic, rules, heuristics, or other functions for selecting a response. For example, the data processing system 102 can determine that a first digital assistant component is a preferred digital assistant. The data processing system 102 can determine that the first digital assistant is a preferred digital assistant based on the type of query, configuration information, profile information, predetermined information, or other factors. The data processing system 102 can receive a first response from the first digital assistant that is set as the preferred digital assistant. This first response from the first digital assistant can be associated with a flag. The first digital assistant can assign, attach, provide, or otherwise indicate a flag for the first response. The flag can indicate that this first response is a fallback response. A fallback response can refer to a response that is an alternative response to be used in instances where there is no other response. The fallback response can indicate that the first response should be lowered in priority, weight, or ranking when entering the response into the selection process being performed by the ranking engine component 136. The first digital assistant or data processing system 102 can determine that the first response should be flagged as a fallback response based on a confidence score associated with the first response or other factors associated with the first response. For example, the first digital assistant component can determine that the quality associated with the data file associated with the first response is low quality or has a quality below a threshold. Quality can refer to or include video quality, image quality, or audio quality. Quality can refer to or include a data streaming rate. Quality can refer to a delay in transmitting the data file or establishing a connection with a data source. Quality can refer to the quality of the content of the response, such as a rating or review associated with the result.The quality can refer to the relevance of the result to the query or profile associated with the client device 104. The first digital assistant component can determine that the confidence score associated with the first response is low or below a threshold.

[0094] In response to determining that the quality or reliability score associated with the result is low or below a threshold, the first digital assistant component can flag the response as a fallback response. That is, although the first digital assistant component (e.g., first digital assistant server 106) is configured to be the preferred digital assistant, the first digital assistant component can flag the response generated or provided by the first digital assistant component as a fallback response. When the data processing system 102 (e.g., ranking engine component 136) receives a result from the first digital assistant server 106 that is the preferred digital assistant, it can override the first digital assistant component. For example, the data processing system 102 can be configured to select a response from the preferred digital assistant component. However, because the result from the preferred digital assistant component is flagged as a fallback response, the data processing system 102 can determine that the response from the first digital assistant component should not be selected, thereby overriding the first digital assistant component. Instead, the data processing system 102 can select a response from a second digital assistant component 120, which may not be the preferred digital assistant component, to provide to the client device 104.

[0095] When overriding the first digital assistant component by selecting a response from the second digital assistant, even though the first digital assistant is the preferred digital assistant component, the data processing system 102 can still retain the first digital assistant component as the preferred digital assistant. For example, the data processing system 102 can override the first digital assistant component for responses that are flagged as fallback responses, but cannot override for other responses that are not flagged as fallback responses.

[0096] The data processing system 102 can provide the selected result(s) or the selected result(s) from the selected digital assistant to the client device 104. The data processing system 102 can provide the results to the client device 104 via another device or server. The data processing system 102 can preprocess or otherwise manipulate or modify the results before providing them to the client device 104. The data processing system 102 can provide the results to a device or server remote from the data processing system 102 for processing before the results are provided to the client device. For example, the data processing system 102 can provide the results to the S3 server 204. The S3 server 204 can convert responses received from either the HTML adapter 218 or the first digital assistant server 106 into audio and can provide the audio responses to the client device 104 or the display device 108 for rendering or presentation. The S3 server 204 can save the responses in the device repository 220 for further processing or as a data log for further processing.

[0097] In some cases, data processing system 102 may provide a selected response or result to client device 104. Client device 104 may be configured to present the response via an output interface of client device 104. Client device 104 may be configured to convert the result or response from one format to another format for presentation via the output interface of client device 104. For example, client device 104 itself may be configured with text-to-speech capabilities for converting text or HTML content into audio or visual output.

[0098] FIG. 3 shows a flow diagram for coordinating overlapping processing of audio queries using the exemplary system shown in FIG. 1 or FIG. 2. Process 300 can be performed by data processing system 102, first digital assistant server 106, client device 104, one or more processors or computing devices 500. At 302, the data processing system can operate a first digital assistant in a first branch and a second digital assistant in a second branch. The two digital assistants can operate in parallel or in an overlapping manner. Overlapping processing can refer to both digital assistants operating simultaneously for at least a partial amount of time. Overlapping processes can refer to operating at least partially in parallel or at least partially simultaneously. One of the digital assistants can start operating first or stop operating first. Processing on separate digital assistants can occur in parallel (e.g., simultaneously). The first digital assistant can operate on a third-party server. The second digital assistant can operate on the data processing system. The first digital assistant may be governed by a third party that is different from the entity that governs the second digital assistant.

[0099] In 304, the data processing system can execute a decision block to determine whether results from the second digital assistant are available. If results from the second digital assistant are not available, in ACT 306, the data processing system can return to decision block 304 and check again to see if results are available. The data processing system can return to decision block 304 to repeatedly check whether results from the second digital assistant are available. In some cases, the data processing system can time out if results from the second digital assistant are not available for a predetermined amount of time (e.g., 10 seconds, 30 seconds, 60 seconds, or longer).

[0100] If in block 304 the data processing system determines that a result from the second digital assistant is available (ACT 308), the data processing system can proceed to decision block 310 to determine whether the result from the second digital assistant is preferred. If the result from the second digital assistant is preferred, the data processing system can proceed to block 312, where the data processing system can select a response from the second digital assistant.

[0101] However, if in block 310 the data processing system determines that the result from the second digital assistant is unfavorable, the data processing system can proceed to block 314 and wait for a response from the first digital assistant. Once the data processing system receives the result from the first digital assistant, the data processing system can proceed to decision block 316, where the data processing system can determine whether the response from the first digital assistant is flagged as a fallback response. A fallback response can indicate that the response should only be used as a fallback, i.e., last resort or last resort. If there is a low (e.g., below a lower threshold, such as 0.2, 0.3, 0.4, etc.) confidence score that the response is accurate, reliable, or appropriate, the response can be tagged or flagged as a fallback response.

[0102] If the data processing system determines in decision block 316 that the response from the first digital assistant server is flagged as a fallback response, the data processing system can proceed to select a response from the second digital assistant in block 312. For example, if the response from the first digital assistant server is flagged as a fallback, the data processing system can override the first digital assistant response and proceed to use the second digital assistant, even though the first digital assistant is the preferred digital assistant.

[0103] However, if the first digital assistant response is not flagged as a fallback, as determined by the data processing system in decision block 316, the data processing system 102 can proceed to block 318 and select the first digital assistant response.

[0104] In some cases, in block 304, the data processing system may determine that results from the second digital assistant component are available or ready before results from the first digital assistant component. In block 310, the data processing system may determine that the second digital assistant component is not a preferred digital assistant component. The data processing system may make this determination based on configuration information, instructions, a display, or other data. The data processing system may determine that the first digital assistant is a preferred digital assistant. For example, a configuration, an instruction, a display, or other data file may indicate which of multiple digital assistants is a preferred digital assistant. In block 310, by determining that the first digital assistant is a preferred digital assistant, the data processing system may determine that the second digital assistant is not a preferred digital assistant.

[0105] The data processing system can take one or more actions in response to determining that the second digital assistant is not the preferred digital assistant or in response to determining that the first digital assistant is the preferred digital assistant. For example, in block 314, the data processing system can wait for a first response from the first digital assistant component in response to determining that the second digital assistant component is not the preferred digital assistant and that the first digital assistant is the preferred digital assistant. Thus, although the data processing system received a response from the second digital assistant before receiving any response from the first digital assistant, the data processing system can wait for a first response from the first digital assistant because the first digital assistant is the preferred digital assistant.

[0106] The data processing system can receive a response from the first digital assistant. The data processing system can receive a response from the first digital assistant after waiting for a response after initially receiving a response from the first digital assistant. In block 316, the data processing system can determine, upon receiving the first response from the first digital assistant component, that the first response is flagged as a fallback response. For example, the first digital assistant is configured as a preferred digital assistant, but the response generated by the first digital assistant can be assigned as a fallback response. In block 312, the data processing system can select a second response based on a determination that the second digital assistant component is not a preferred digital assistant and the first response is flagged as a fallback response. While the data processing system may prioritize the response from the first digital assistant and the data processing system may be waiting for a response from the first digital assistant despite already receiving a response from the second digital assistant, the data processing system may ultimately determine that the response from the second digital assistant should be selected because the response from the first digital assistant was flagged as a fallback response. The response from the second digital assistant may not be flagged as a fallback. If the responses from both the first and second digital assistants are flagged as fallback responses, the data processing system may select the response from the preferred digital assistant, such as the first digital assistant in this example. Thus, the data processing system may rank the response from the preferred digital assistant higher and may select the response from the preferred digital assistant unless the response from the preferred digital provider is flagged as a fallback response and the response from the non-preferred digital assistant is not flagged as a fallback.

[0107] 4 illustrates a method for coordinating parallel processing of audio queries across multiple devices using the example systems shown in FIG. 1 or FIG. 3. For example, method 400 may be performed by data processing system 102. At 405, the data processing system may receive an audio signal from a client device. The client device may detect the audio signal using a sensor, such as a microphone or a transducer. The audio signal may include voice or speech from a user of the client device. The audio signal may include a wake-up word, a keyword, a hot word, or a trigger word to alert the client device to take an action in response to at least a portion of the audio signal.

[0108] In ACT 410, the audio signal may be converted to text to generate a query. For example, the client device itself may convert the detected audio signal to speech, or the client device may send the audio signal to a streaming audio server for converting the audio to text. The client device may preprocess or convert the audio signal to another format to facilitate data transmission over a network. For example, the client device may convert an analog audio input signal to a digital audio signal that facilitates transmitting the audio signal as a payload in a data packet over a network. The data processing system or streaming audio server may convert the audio signal to text upon receiving the data packet.

[0109] The data processing system may identify a query in the text that corresponds to the audio signal. The query may include, for example, a request for content. The query may include a request for data or information. The query may include a request for the data processing system or client device or other device to perform an action, such as a digital or electronic action.

[0110] In ACT415, the data processing system can send the query to a first digital assistant server for processing. The data processing system can send the query to a second digital assistant server for processing. The data processing system can send the query to multiple digital assistants via separate branches. A branch can refer to a processing thread. The data processing system can send the query to multiple digital assistants simultaneously for processing.

[0111] In ACT 420, the data processing system can use ranking decision logic to override the response of the first digital assistant and select the response of the second digital assistant. The data processing system can select the ranking decision logic based on a signal in the query. The signal can indicate or indicate the type of query, such as a movie query, a news query, a music query, an audiobook query, a weather query, or a television show query. The data processing system can identify a signal indicating the type of query and then select the corresponding ranking decision logic to select a preferred digital assistant. For example, if the query is a movie query, the data processing system can set the first digital assistant as the preferred digital assistant. However, if the signal indicates that the query is a search query, the data processing system can, for example, set the second digital assistant as the preferred digital assistant.

[0112] The data processing system can select a ranking decision function from a pool of multiple ranking decision functions. The data processing system can select a ranking decision function based on a signal of a query. For example, the data processing system can select a first ranking decision function for a movie query, a second ranking decision function for a music query, or a third ranking decision function for a news query. For example, the first ranking decision function can assign a first digital assistant as a preferred digital assistant and set a first threshold for a reliability score for the response. The second ranking decision can assign a second digital assistant as a preferred digital assistant and set a second threshold for a reliability score for the response. The first and second thresholds can be set or adjusted based on factors related to the response to the respective query. For example, there can be a lower reliability score threshold for weather queries and a higher reliability score threshold for navigation-related queries or movie-related queries.

[0113] For example, a first digital assistant component may be configured to process and respond to audio queries related to media domain queries, such as "Play Movie_A on Streaming_Service_A," while a second digital assistant component may be configured to process and respond to search domain queries, such as "When is my next appointment?" If a query signal is ambiguous or falls into multiple categories, the data processing system may rank responses from multiple digital assistant components and select a response from one of the digital assistant components. For example, the data processing system may score or rank responses or queries based on confidence level, quality score, or other factors.

[0114] Thus, the data processing system can use a ranking decision logic or function to select one of the digital assistants from which to select a response. In ACT 425, the data processing system can provide the selected response from the selected digital assistant to the client device for presentation. For example, the data processing system can provide a response to a query generated by a second digital assistant to the client device for presentation.

[0115] 5 is a block diagram of an exemplary computer system 500. The computer system or computing device 500 can include or be used to implement system 100 or its components, such as data processing system 102, client device 104, first digital assistant server 106, and display device 108. The computing system 500 includes a bus 505 or other communication component for communicating information and a processor 510 or processing circuit coupled to the bus 505 for processing information. The computing system 500 can include one or more processors 510 or processing circuits coupled to the bus for processing information. The computing system 500 includes a primary memory 515, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 505 for storing information and instructions to be executed by the processor 510. The primary memory 515 can be or include data repository 140. The primary memory 515 can also be used to store location information, temporary variables, or other intermediate information during execution of instructions by the processor 510. Computing system 500 may further include a read-only memory (ROM) 520 or other static storage device coupled to bus 505 for storing static information and instructions for processor 510. A storage device 525, such as a solid-state device, magnetic disk, or optical disk, may be coupled to bus 505 for persistently storing information and instructions. Storage device 525 may include or be part of data repository 140.

[0116] Computing system 500 may be coupled via bus 505 to a display 535, such as a liquid crystal display or active matrix display, for displaying information to a user. An input device 530, such as a keyboard including alphanumeric and other keys, may be coupled to bus 505 for communicating information and command selections to processor 510. Input device 530 may include a touchscreen display 535. Input device 530 may include a cursor control device, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 510 and for controlling cursor movement on display 535. Display 535 may be part of data processing system 102, client device 104, display device 108, or other component of FIG. 1, for example.

[0117] The processes, systems, and methods described herein may be performed by computing system 500 in response to processor 510 executing an arrangement of instructions stored in main memory 515. Such instructions may be read into main memory 515 from another computer-readable medium, such as storage device 525. Execution of the arrangement of instructions stored in main memory 515 causes computing system 500 to perform the example processes described herein. One or more processors in a multi-processing arrangement may be employed to execute the instructions stored in main memory 515. Hardwired circuitry may be used in place of or in combination with software instructions with the systems and methods described herein. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0118] Although an exemplary computing system is illustrated in FIG. 5, the subject matter including the operations described herein may be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, or in combinations of one or more of them, including the structures disclosed herein and their structural equivalents.

[0119] For situations in which the systems described herein may collect or utilize personal information about a user, the user may be provided with an opportunity to control whether a program or function may collect personal information (e.g., information about the user's social network, social behavior, or activities, the user's preferences, or the user's location) or to control whether or how content that may be more important to the user is received from a content server or other data processing system. In addition, some data may be anonymized in one or more ways before being stored or used, such that personally identifiable information is removed when generating parameters. For example, a user's identifying information may be anonymized so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (such as to the city level, zip code level, or state level) where location information is obtained so that the user's specific location cannot be determined. Thus, the user may control how information is collected about him or her and used by content servers.

[0120] The subject matter and operations described herein may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in a combination of one or more of these. The subject matter described herein may be implemented as one or more circuits of one or more computer programs, e.g., computer program instructions encoded on one or more computer-stored multimedia for execution by or to control the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by the data processing device. A computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. Although a computer storage medium is not a propagating signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be, or may be contained within, one or more separate physical components or multimedia (e.g., multiple CDs, disks, or other storage devices). The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0121] The terms “data processing system,” “computing device,” “component,” or “data processing apparatus” encompass various apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system-on-chip, or a plurality or combination of the above. An apparatus may include special-purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program in question, e.g., code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as a web services infrastructure, a distributed computing infrastructure, and a grid computing infrastructure. The components of system 100 may include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0122] A computer program (also referred to as a program, software, software application, app, script, or code) can be written in any type of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including a standalone program or a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored within a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), within a single file dedicated to the program, or within multiple coordinated files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

[0123] The processes and logic flows described herein may be executed by one or more programmable processors executing one or more computer programs (e.g., components of data processing system 102, client device 104, first digital assistant server 106, and display device 108) to perform actions by operating on input data and generating output. The processes and logic flows may also be executed by, and apparatus may be implemented as, special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of, by way of example, semiconductor memory devices, including, for example, EPROM, EEPROM, and flash memory devices, non-volatile memory, non-volatile multimedia, and non-volatile memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be augmented by or incorporated within special-purpose logic circuitry.

[0124] The subject matter described herein may be implemented in a computing system that includes back-end components, e.g., data servers, or middleware components, e.g., application servers, or front-end components, e.g., client computers having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or a combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (“LANs”) and wide area networks (“WANs”), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0125] A computing system such as system 100 or system 500 may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communications network (e.g., network 110). The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server sends data (e.g., data packets representing content items) to client devices (e.g., to display data to and receive user input from users interacting with the client devices). Data generated at a client device (e.g., a result of user interaction) may be received at the server from the client device (e.g., received by data processing system 102 from client device 104 or display device 108).

[0126] Although operations are shown in the figures in a particular order, such operations do not require that they be performed in the particular order or sequential order shown, and not all of the operations shown need be performed. Actions described herein may be performed in various orders.

[0127] The separation of various system components may not be necessary in all implementations, and the described program components may be included within a single hardware or software product. For example, the NLP component 124 and the direct action API 126 may be part of a single component, app, or program, or a logical device having one or more processing circuits, or one or more servers of the data processing system 102.

[0128] While several exemplary implementations have been described above, it should be apparent that the foregoing is presented by way of example and not limitation. In particular, while many of the examples presented herein involve particular combinations of method acts or system elements, those acts and those elements may be combined in other ways to achieve the same purpose. Acts, elements, and features described with respect to one implementation are not excluded from similar roles in other implementations.

[0129] The phraseology and terminology used herein are for purposes of description and should not be considered limiting. The use herein of "including," "comprising," "having," "storing," "involving," "characterized by," "characterized by," and variations thereof, is meant to encompass the subsequently listed items, equivalents thereof, and additional items, as well as alternative implementations consisting of the subsequently listed items exclusively. In one implementation, the systems and methods described herein consist of one, each combination of two or more, or all of the described elements, acts, or components.

[0130] Any reference herein to system and method implementations, elements, or acts in the singular may also encompass implementations that include a plurality of those elements, and any reference herein to any implementation, element, or act in the plural may also encompass implementations that include only a single element. References in the singular or plural are not intended to limit the disclosed systems or methods, their components, acts, or elements to single or multiple configurations. References to any act or element that is based on any information, act, or element may include implementations in which the act or element is based at least in part on the any information, act, or element.

[0131] Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to "one implementation," "several implementations," "one implementation," etc. are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described with respect to an implementation may be included in at least one implementation or embodiment. Terms as used herein do not necessarily all refer to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0132] References to "or" may be construed as inclusive, such that any term described using "or" may refer to either one, more than one, or all of the described terms. A reference to "at least one of 'A' and 'B'" can include "A" only, "B" only, and both "A" and "B." Such references used with "comprising" or other open terminology can include additional items.

[0133] Where a reference sign follows a technical feature in a drawing, the detailed description, or any claim, the reference sign is included to enhance the intelligibility of the drawing, the detailed description, and the claims, and thus neither the reference sign nor the absence of a reference sign has any limiting effect on the scope of any claim element.

[0134] The systems and methods described herein may be embodied in other specific forms without departing from their characteristics. The above implementations are illustrative rather than limiting of the described systems and methods. The scope of the systems and methods described herein is so indicated by the appended claims, rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. [Explanation of symbols]

[0135] 100 systems 102 Data Processing System 104 client devices 106 First Digital Assistant Server 108 Display Devices 110 Network 120 Second Digital Assistant Application 122 Interface 124 Natural Language Processor (NLP) Components 126 Direct Action Application Programming Interface (API) 128 Interface Management Components 130 Audio Signal Generator Components 132 Search Coordinator Component 134 Content Indexer Component 136 Ranking Engine Components 138 Response Handler Configuration Element 140 Data Repositories 142 Application Identifier 144 Entity Metadata 146 Ranking Policy 148 Referral Addresses 150 speakers 152 sensors 154 Transducer 160 Multimedia Content Applications 162 Content Data Repository 500 Computing Systems 505 Bus 510 processor 515 Main storage 520 Read-Only Memory (ROM) 525 Storage Devices 530 Input Devices 535 Display

Claims

1. 1. A system for coordinating overlapping processing of audio queries, comprising: a data processing system comprising at least one processor and a memory, said data processing system comprising: receiving, via an interface, one or more data packets corresponding to an audio signal detected by a sensor of the computing device; generating a query based on the audio signal received via the one or more data packets; Sending the query to a plurality of digital assistant components, including sending the query to a first digital assistant component executing on a remote server and sending the query to a second digital assistant component executing on the computing device, wherein the first digital assistant component and the second digital assistant component are configured to process the query at least partially in parallel; receiving a first response to the query from the second digital assistant component; After receiving the first response to the query from the second digital assistant component based on a ranking determination function, waiting to receive a second response to the query from the first digital assistant component; Selecting the first response from the second digital assistant component based on the second response to the query from the first digital assistant component being flagged as a fallback response; and providing the first response from the second digital assistant component to the computing device in response to the selection. system.

2. The data processing system is configured to wait to receive the second response to the query from the first digital assistant component based on pre-configured preferences established for the first digital assistant component. The system of claim 1.

3. the data processing system is configured to select the first response from the second digital assistant component based on a confidence score for the second response from the first digital assistant component; 3. The system of claim 1 or 2.

4. the data processing system, determining a signal for said query; configured to select the ranking decision function from a plurality of ranking decision functions based on the signal; 4. A system according to any one of claims 1 to 3.

5. the data processing system, determining a signal for said query; configured to select a response based on the signal; 5. A system according to any one of claims 1 to 4.

6. selecting the first response from the second digital assistant component is further based on the second digital assistant component accessing data stored in a memory of the computing device; 6. A system according to any one of claims 1 to 5.

7. The system of claim 1 , wherein the second response comprises one of calendar information or media content.

8. 8. The system of claim 1, wherein the first digital assistant component executes independently of the second digital assistant component to generate the second response.

9. The data processing system is configured to send the query to each of the plurality of digital assistant components via independent branches to have the plurality of digital assistant components process the query in a redundant manner; 9. A system according to any one of claims 1 to 8.

10. 10. The system of claim 1, wherein the first digital assistant component is established by a different administrator than the second digital assistant component.

11. the data processing system, configured to receive the one or more data packets from the computing device via a streaming audio server configured to convert audio to text.

11. A system according to any one of claims 1 to 10.

12. 1. A method for coordinating overlapping processing of audio queries by a data processing system comprising a processor and a memory, the method comprising: receiving one or more data packets corresponding to an audio signal detected by a sensor of the computing device; generating a query based on the audio signal received via the one or more data packets; Sending the query to a plurality of digital assistant components, including sending the query to a first digital assistant component executing on a remote server and sending the query to a second digital assistant component executing on the computing device, wherein the first digital assistant component and the second digital assistant component process the query at least partially in parallel; Receiving a first response to the query from the second digital assistant component; After receiving the first response to the query from the second digital assistant component based on a ranking decision function, waiting to receive a second response to the query from the first digital assistant component; Selecting the first response from the second digital assistant component based on the second response to the query from the first digital assistant component being flagged as a fallback response; providing the first response from the second digital assistant component to the computing device in response to the selection; A method for providing the above.

13. waiting to receive the second response to the query from the first digital assistant component based on pre-configured preferences established for the first digital assistant component. The method of claim 12, comprising:

14. Selecting the first response from the second digital assistant component based on a confidence score for the second response from the first digital assistant component. The method of claim 12 or 13, comprising:

15. determining a signal for the query; selecting the ranking decision function from a plurality of ranking decision functions based on the signal; 15. The method of any one of claims 12 to 14, comprising:

16. determining a signal for the query; selecting a response based on said signal; 16. The method of any one of claims 12 to 15, comprising:

17. selecting the first response from the second digital assistant component is further based on the second digital assistant component accessing data stored in a memory of the computing device; 17. The method of any one of claims 12 to 16.

18. Sending the query to each of the plurality of digital assistant components via independent branches to have the plurality of digital assistant components process the query in an overlapping manner.

18. The method of any one of claims 12 to 17, comprising:

19. 19. The method of claim 12, wherein the first digital assistant component is established by a different administrator than the second digital assistant component.

20. 20. A computer program configured, when executed on a data processing system comprising a processor and a memory, to cause the data processing system to perform the method of any one of claims 12 to 19.

Images