Rendering visual components on the app in response to voice commands

Abstract

The present disclosure generally relates to systems and methods for rendering visual components on applications. At least one action list may indicate rendering a user interface component. A data processing system may identify a request from an input audio signal from a client device. The client device may display a first application in a foreground process. The data processing system may select an action list to perform an action corresponding to the request by a second application. The data processing system may use the action list to generate an action data structure to provide to the second application. The data processing system may determine that the output of the second application is authorized to be presented with the first application. The data processing system may display a user interface component from the second application on the client device together with the first application.

Description

Background Art

[0001] An application can be installed on a computing device. The computing device can execute the application. The application can present digital content. Summary of the invention

[0002] According to a first aspect of the present disclosure, a system for rendering a visual component on an application is provided. The system may include a proxy registry executed on a data processing system having one or more processors. The proxy registry may maintain multiple action lists for multiple applications. At least one action list in the multiple action lists may indicate that a user interface component of one of the multiple applications is rendered when the action is performed. The system may include a natural language processor executed on the data processing system. The natural language processor may receive a data packet including an input audio signal detected by a sensor of a client device. The client device may display a graphical user interface of a first application in a foreground process on the client device. The natural language processor may parse the input audio signal of the data packet to identify a request. The system may include an action handler executed on the data processing system. The action handler may select an action list from multiple action lists to perform an action corresponding to the request by a second application in multiple applications. The second application may be installed on the client device without being in a foreground process. The action handler may generate an action data structure based on the action list. The system may include an action handler executed on the data processing system. The proxy interface may provide the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output. The proxy interface may determine, based on an authorization policy of the second application, that output from the second application performing the action is authorized to be presented with the graphical user interface of the first application. The proxy interface may, in response to determining that the output of the second application is authorized to be presented with the first application, identify a user interface component of the second application for an action manifest selected from a plurality of action manifests. The proxy interface may display the user interface component including the output from the second application on the client device together with the graphical user interface of the first application that is authorized to be presented with the second application.

[0003] At least one of the multiple action lists may have an address template for accessing a user interface component when performing an action, the address template defining a first part and a second part, the first part corresponding to one of the multiple applications, and the second part including input variables when performing the action. The natural language processor may parse the input audio signal of the data packet to identify parameters defining a request to be executed by the second application. The action handler may generate an address for executing an action of the action list based on the address template of the action list, the address including a first substring and a second substring, the first substring corresponding to the second application, and the second substring having parameters for controlling the execution of the action; and generate an action data structure including the address generated by the action handler based on the address template.

[0004] At least one of the multiple action lists may have an address template for accessing a user interface component when executing the action, the address template defining a first part and a second part, the first part corresponding to one of the multiple applications, and the second part including input variables when executing the action. The natural language processor may parse the input audio signal of the data packet to identify parameters defining the request to be executed by the second application. The action handler is used to: generate an address for executing the action of the action list according to the address template of the action list, the address including a first substring and a second substring, the first substring corresponding to the second application, and the second substring having parameters for controlling the execution of the action. The proxy interface may use the address generated according to the address template of the action list to identify the user interface component of the second application for an action list selected from the multiple action lists.

[0005] Each action list may have an agent identifier corresponding to one of the plurality of applications and a request identifier corresponding to the action. The natural language processor may parse the input audio signal of the data packet to identify an agent identifier corresponding to the agent, the agent corresponding to a second application installed on the client device. The action handler may select an action list from the plurality of action lists for performing the action, the action having an agent identifier corresponding to the agent identifier identified from parsing the input audio signal.

[0006] The action handler can determine that an interface mode indicator of the action manifest specifies rendering of the user interface component when performing the action, and in response to determining that the interface mode indicator specifies rendering of the user interface component, identify the user interface component of the second application for the action manifest.

[0007] The proxy interface can determine that the client device has not been authenticated by the second application based on an action list for performing an action corresponding to the second request, and can, in response to determining that the client device has not been authenticated by the second application, present a prompt interface on the client device to authenticate the client device to the second application to perform the action corresponding to the second request.

[0008] The proxy interface can determine, based on the user's authorization policy for the third application, that the output of the third application is not authorized to be presented with the first application, that the third application is installed on the client device but is not in the foreground process, set the third application as the foreground process of the client device, change the first application from the foreground process to a non-foreground process running on the client device, and display a graphical user interface of the third application on the client device, which graphical user interface includes the output generated by the third application.

[0009] The proxy interface can determine that the first application is authorized to present the user interface component of the second application together with the graphical user interface of the first application based on the authorization policy of the first application, which authorization policy allows user interface components from a first subset of multiple applications to be presented together with the graphical user interface and restricts user interface components from a second subset of multiple applications from being presented together with the graphical user interface, and in response to determining that the first application is authorized to present the user interface component of the second application together with the graphical user interface of the first application, display the user interface component including output from the second application on the client device together with the graphical user interface of the first application.

[0010] The proxy interface can provide the action data structure to the second application so that the second application monitors interaction events on a user interface component from the second application that is presented together with the graphical user interface of the first application, and in response to detecting the interaction event, processes the interaction event to update the user interface component.

[0011] The natural language processor may be used to receive a data packet via a digital assistant application, the data packet including an input audio signal detected by a sensor of a client device, the client device displaying a graphical user interface of the digital assistant application corresponding to the first application in a foreground process. The agent interface may display a user interface component including a subcomponent indicating a request identified from parsing the input audio signal and a user interface component including output from an agent corresponding to the second application within the graphical user interface of the digital assistant application. The agent may not have natural language processing capabilities.

[0012] The natural language processor can receive a data packet that includes an input audio signal detected by a sensor of a client device, the client device displaying a graphical user interface of a first agent corresponding to a first application in a foreground process; and the agent interface is used to display a user interface component that includes output from a second application as an overlay on the graphical user interface of the first agent.

[0013] The plurality of action lists may have a first subset of action lists including action lists defined by an administrator of the data processing system and a second subset of action lists including action lists provided by an agent service processing one of the plurality of applications.

[0014] The action handler can provide the action data structure to the second application so that the second application parses the action data structure to identify the action to be performed, generates output by executing the action identified from the action data structure, and provides the output to a user interface component to be presented together with the graphical user interface of the first application that is authorized to be presented with the second application.

[0015] According to a second aspect of the present disclosure, a method for rendering a visual component on an application is provided. A data processing system having one or more processors may maintain multiple action lists for multiple applications. At least one action list in the multiple action lists may indicate that a user interface component of one of the multiple applications is rendered when an action is performed. The data processing system may receive a data packet including an input audio signal detected by a sensor of a client device. The client device may display a graphical user interface of a first application in a foreground process on the client device. The data processing system may parse the input audio signal of the data packet to identify a request. The data processing system may select an action list for performing an action corresponding to the request through a second application in multiple applications from multiple action lists. The second application may be installed on the client device without being in a foreground process. The data processing system may generate an action data structure based on the action list. The data processing system may provide the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output. The data processing system may determine that the output from the second application performing the action is authorized to be presented together with the graphical user interface of the first application based on the authorization policy of the second application. The data processing system may identify the user interface component of the second application for an action list selected from the multiple action lists in response to determining that the output of the second application is authorized to be presented together with the first application. The data processing system may display on the client device a user interface component including output from the second application along with a graphical user interface of the first application that is authorized to be presented with the second application.

[0016] Optional features of the first aspect described above or in the detailed description below may be combined with the second aspect.

[0017] These and other aspects and embodiments are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and embodiments, and provide an overview or framework for understanding the nature and characteristics of the claimed aspects and embodiments. The accompanying drawings provide illustration and further understanding of the various aspects and embodiments, and are incorporated into and constitute a part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are not intended to be drawn to scale. In the various figures, similar reference symbols and designations represent similar elements. For the sake of brevity, it may not be necessary to label every component in every figure. In the drawings:

[0019] Figure 1 A block diagram of an example system for rendering visual components on an application in response to voice input commands according to examples of the present disclosure is shown.

[0020] Figure 2 An example of the present disclosure is shown. Figure 1 A timing diagram of an example data flow for rendering a visual component on an application in response to a voice input command in the system shown.

[0021] Figure 3 An example according to the present disclosure is shown. Figure 1 A use case diagram of an example client device in the illustrated system showing input messages and generating addresses.

[0022] Figure 4 An example of using the present disclosure is shown. Figure 1 The example system shown is a flow chart of a method for rendering a visual component on an application in response to a voice input command.

[0023] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0024] Below is a more detailed description of various concepts and implementations thereof related to methods, apparatuses, and systems for rendering visual components on applications in response to voice input commands. The various concepts introduced above and discussed in more detail below may be implemented in any of a number of ways.

[0025] Multiple applications can be installed on a client device to perform preconfigured functions according to the request of the user operating the client device. At a given time, the client device can run a subset of these applications as a foreground process or a non-foreground process set by the operating system running on the client device. A foreground process (including a visible process) can correspond to an application with a graphical user interface that is presented on the display of the client device with which the user may be interacting. A non-foreground process (e.g., a service process, a background process, or a cached process) can correspond to an application that does not display a visual component to the user and executes without affecting the graphical user interface of the foreground process. The application running on the client device can switch back and forth between executing as a foreground process or executing as a non-foreground process. After switching, the graphical user interface of the application that was previously the foreground process can be replaced by the graphical user interface of the application that is now set as the foreground process. In addition, due to the switching between the foreground process and the non-foreground process, processing and memory allocation may change.

[0026] One of these applications running in the foreground or non-foreground may include a digital assistant application. Unlike at least some other applications installed on the client device, the digital assistant application may have natural language processing capabilities. Since the digital assistant application is configured with these capabilities, the digital assistant application can obtain and parse the input audio signal detected by the microphone of the client device to identify words from the input audio signal. According to the recognized words, the digital assistant application can determine the request to be executed and one or more parameters defining the request. The request may refer to one of the preconfigured functions with input variables compatible with the parameters that the digital assistant application can execute (for example, retrieving the results of a search query). For such requests, the digital assistant application itself can process the request and present audio or visual output by executing the request. However, some requests may not correspond to any preconfigured functions on the digital assistant application (for example, playing a video from an online video platform). For these types of requests, the digital assistant application can identify another application that can execute the request and call the function of the application to pass the parameter.

[0027] By calling another application to perform a request, the digital assistant application can cause a change in whether the application is a foreground process or a non-foreground process, resulting in a change in the graphical user interface displayed on the client device. In addition, the processing and memory allocation for each executing application may change and fluctuate, especially when calling a previously unexecuted application. Although the required request can ultimately be executed on the client device by the called application, the switching may interfere with the overall user experience of the application on the client device. For example, a sudden change in the graphical user interface displayed on the client device may force the user to context switch between the interfaces of multiple applications, thereby reducing the human-computer interaction (HCI) with the client device. When performing a switch between a foreground process and a non-foreground process, the changes and fluctuations in the new allocation of computing resources may also exacerbate the degradation of the client device.

[0028] The present disclosure provides an improved user interface. Specifically, in order to solve the technical challenges in docking, the present disclosure can provide an action inventory for identifying and presenting user interface components on the graphical user interface of the application to be covered in the foreground. The action inventory can specify whether to provide the user interface elements of the application while maintaining the application in the background. The action inventory can also include an address template to construct an address for accessing the functions of the digital assistant application and providing the user interface elements on the client device. The address template of the action inventory can be, for example, a uniform resource identifier (URI) template for constructing a network address using a host name, a path name, and a query to call the functions of the application and the user interface components. The host name (or scheme) can reference the location or domain name of the application. The path name can reference the function. The query can include one or more input variables for executing the function. The address template can also define a mapping from a request parsed from an input audio signal to the function to be executed by the application.

[0029] After identifying the request from the input audio signal, the digital assistant application can determine that the request does not correspond to any functionality of the digital assistant application or any application running as a foreground process. Instead of outputting that the digital assistant application cannot execute the indicated request, the digital assistant application can identify another application to perform the function. When processing the request, the digital assistant application can be retained as a background process, while the original application is retained as a foreground process. Through this identification, the digital assistant application can select an action list for the application for the function. The selected action list can indicate the user interface component of the application to be presented. Using the action list, the digital assistant application can construct, expand and generate the address of the function according to the address template. The address may include a host name (or scheme) referencing the application, a path referencing the function and the user interface component, and one or more queries including parameters.

[0030] As the address is generated according to the address template, the digital assistant application can generate an action data structure (e.g., an API function call) including the address and pass it to the application. The application to which the action data structure is passed can be opened as a background process and retained in the background process after processing and completing the action. The action data structure can be passed to other applications because the original application will continue to be retained as a foreground process. For example, the action data structure can be passed from the digital assistant application to the application by the operating system on the client device through an API function call. Since the address includes a host name that references the application, the action data structure can be directed to the referenced application. After receiving, the application can parse the action data structure to identify the address. The application can further parse the address to identify the path name used to identify the function to be executed, and parse the string query to identify one or more parameters for defining the execution of the function. Based on the identification, the application can execute the function to generate output. Using the output generated by the application, the digital assistant can render the user interface component of the application. The user interface component of the application can be overlaid on the graphical user interface of the application of the foreground process.

[0031] In this way, the digital assistant application can call the functions of other applications and present the rendering of user interface components from these applications, thereby enhancing the functionality of the digital assistant application. The call can be performed without the user manually searching, opening, and entering to achieve the desired function or without having to switch contexts between different applications. Therefore, eliminating context switching on the user side of operating the client device can improve HCI. In addition, even if a request is identified from the input audio signal, the application can remain in the foreground process and the non-foreground process. Since the switch between the foreground process and the background process is eliminated, the digital assistant application can reduce the consumption of computing resources and reduce the time on the user side.

[0032] Reference Figure 1 , depicts an example system 100 for rendering visual components on an application. System 100 may include at least one data processing system 102, at least one client device 104, and at least one proxy service 106. Data processing system 102, client device 104, and proxy service 106 may be communicatively coupled to each other via at least one network 114.

[0033] The components of the system 100 can communicate via a network 114. The network 114 can include, for example, a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an ATM (asynchronous transfer mode) network, a SONET (synchronous optical network) network, an SDH (synchronous digital hierarchy) network, an NFC (near field communication) network, a local area network (LAN), a wireless network or a wired network, and combinations thereof. The network 114 can include a wireless link, such as an infrared channel or a satellite band. The topology of the network 114 can include a bus, a star, or a ring network topology. The network 114 can include a mobile phone network using any one or more protocols for communicating between mobile devices, including: Advanced Mobile Phone Protocol (AMPS), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), or Universal Mobile Telecommunications System (UMTS). Different types of data can be transmitted via different protocols, or the same type of data can be transmitted via different protocols.

[0034] The data processing system 102 and the proxy service 106 may include a plurality of logically grouped servers, respectively, and contribute to distributed computing technology. The logical group of servers may be referred to as a data center, a server farm, or a machine farm. The servers may be geographically dispersed. A data center or a machine farm may be managed as a single entity, or a machine farm may include a plurality of machine farms. The servers in each computer farm may be heterogeneous: one or more servers or machines may operate according to one or more types of operating system platforms. The data processing system 102 and the proxy service 106 may include: servers stored in one or more high-density rack systems in a data center, and associated storage systems, such as those located in an enterprise data center. In this way, the data processing system 102 and the proxy service 106 with integrated servers may improve the manageability of the system, data security, physical security of the system, and system performance by locating servers and high-performance storage systems on a localized high-performance network. Centralizing all or some of the data processing system 102 and the proxy service 106 components (including servers and storage systems) and coupling them with advanced system management tools allows more efficient use of server resources, thereby saving power and processing requirements and reducing bandwidth usage. Each component of data processing system 102 may include at least one processing unit, server, virtual server, circuit, engine, agent, appliance, or other logic device, such as a programmable logic array configured to communicate with data repository 128 and other computing devices. Proxy service 106 may also include at least one processing unit, server, virtual server, circuit, engine, agent, appliance, or other logic device, such as a programmable logic array configured to communicate with data repository and other computing devices.

[0035] The data processing system 102 may include at least one instance of a digital assistant application 108. The digital assistant application 108 may include at least one natural language processor (NLP) component 116 (sometimes referred to herein as a natural language processor) to parse audio-based input. The digital assistant application 108 may include at least one audio signal generator component 118 (sometimes referred to herein as an audio signal generator) to generate audio-based signals. The digital assistant application 108 may include at least one action processor component 120 (sometimes referred to herein as an action processor) to generate an action data structure based on audio-based input. The digital assistant application 108 may include at least one response selector component 122 to select a response to an audio-based input signal. The digital assistant application 108 may include at least one proxy registry component 124 (sometimes referred to herein as a proxy registry) to maintain an action list for generating an action data structure. The digital assistant application 108 may include at least one proxy interface component 126 (sometimes referred to herein as a proxy interface) to communicate with one or more proxy applications 110.

[0036] The digital assistant application 108 may include at least one data repository 128. The data repository 128 of the data processing system 102 may include one or more local or distributed databases, and may include a database management system. The data repository 128 may include one or more regular expressions 130, one or more data parameters 132, one or more strategies 134, one or more response data 136, one or more templates 138, and one or more action lists 140. The NLP component 116, the audio signal generator component 118, the action processor component 120, the response selector component 122, the agent registry component 124, the agent interface component 126, and the data repository 128 may be separated from each other. The data repository 128 may include a computer data storage or memory, and may store one or more regular expressions 130, one or more data parameters 132, one or more strategies 134, response data 136, templates 138, and one or more action lists 140 and other data. The data parameters 132, the strategies 134, and the templates 138 may include information such as rules for a voice-based conversation between the client device 104 and the data processing system 102. Response data 136 may include content items or associated metadata for audio output and input audio messages that may be part of one or more communication sessions with client device 104. Action manifest 140 may include information for invoking or interfacing with agent application 110.

[0037] The data processing system 102 may include at least one instance of an agent application 110 (also referred to herein as an "agent" or "application") to perform various functions. The agent service 106 may include at least one instance of an agent application 110 to perform various functions. The agent application 110 may not have natural language processing capabilities (such as the capabilities provided by the NLP component 116). The agent application 110 may also have a set of preconfigured or predetermined input capabilities. The input capabilities of the agent application 110 may include, for example, input from a mouse, keyboard, touch screen (e.g., display 148), or camera of the client device 104. The input capabilities of the agent application 110 may not have audio (e.g., microphone 146) from the client device 104. The agent application 110 may have a graphical user interface (GUI) to receive input from the client device 104. Based on the input from the predetermined input capabilities, the agent application 110 may perform or execute one or more functions. For example, the agent application 110 may be a coffee purchasing application and may have a graphical user interface with one or more elements to specify an order for coffee. In this application, a user operating a client device 104 can manually enter coffee, size, and billing information, etc., on a graphical user interface rendered on a display 148 of the client device 104. Once set up, the agent application 110 on the client device 104 can send the input to the agent service 106 via the network 114. The agent service 106 can then execute the purchase order indicated by the input entered through the graphical user interface of the agent application 110.

[0038] The components of data processing system 102 may each include at least one processing unit or other logic device, such as a programmable logic array engine or module configured to communicate with data storage repository 128. The components of data processing system 102 may be separate components, a single component, or part of multiple data processing systems 102. System 100 and its components, such as data processing system 102, may include hardware elements, such as one or more processors, logic devices, or circuits.

[0039] The functionality of data processing system 102, such as digital assistant application 108 or proxy application 110, may be included or may be accessed from one or more client devices 104. The functionality of data processing system 102 may correspond to functionality executed on client device 104 or interface with digital assistant application 108 executed on client device 104. Client devices 104 may each include and execute separate instances of one or more components of digital assistant application 108. Client devices 104 may otherwise have access to the functionality of components of digital assistant application 108 on remote data processing system 102 via network 114. For example, client device 104 may include the functionality of NLP component 116 and may access the remaining components of digital assistant application 108 to data processing system 102 via network 114. The functionality of data processing system 102 may correspond to functionality of proxy application 110 executed on client device 104 or interface with proxy application 110 executed on client device 104. Client device 104 may otherwise have access to the functionality of components of agent application 110 on data processing system 102 or agent service 106 via network 114. For example, client device 104 may pre-process input for agent application 110 from a mouse, keyboard, touch screen, or camera and provide the input to agent service 106 to perform the functions of agent application 110. Digital assistant application 108 executing on client device 104 may include the functionality of digital assistant application 108 executing on data processing system 102.

[0040] Client devices 104 may each include at least one logical device, such as a computing device having a processor to communicate with each other via network 114 with data processing system 102. Client devices 104 may include instances of any components described with respect to data processing system 102 or proxy service 106. Client devices 104 may include instances of digital assistant application 108. Client devices 104 may include instances of proxy application 110. Client devices 104 may include desktop computers, laptop computers, tablet computers, personal digital assistants, smartphones, mobile devices, portable computers, thin client computers, virtual servers, speaker-based digital assistants, or other computing devices.

[0041] The client device 104 may include one or more of at least one instance of a digital assistant application 108, at least one instance of a proxy application 110, and at least one communication interface 112, etc., to execute, interface with, or otherwise communicate with. In addition, the client device 104 may include at least one speaker 144, at least one microphone 146, and at least one display 148, to interface with, or otherwise communicate with. The client device 104 may include an audio driver to provide a software interface to the speaker 144 and the microphone 146. The audio driver may execute instructions provided by the data processing system 102 to control the speaker 144 to generate corresponding acoustic waves or sound waves. The audio driver may execute an audio file or other instructions to convert the acoustic waves or sound waves obtained from the microphone 146 to generate audio data. For example, the audio driver may execute an analog-to-driver converter (ADC) to convert the acoustic waves or sound waves into audio data. The client device 104 may include a graphics driver to provide a software interface to the display 148. The graphics driver may execute instructions provided by the data processing system 102 or the proxy service 106 to control the display 148 to generate corresponding renderings thereon.

[0042] The instance of the digital assistant application 108 on the client device 104 may include one or more processors, logic arrays, or memories, or be executed by them. The instance of the digital assistant application 108 on the client device 104 may detect a keyword and perform an action based on the keyword. The digital assistant application 108 on the client device 104 may be an instance of a digital assistant application 108 executed at the data processing system 102, or may perform any function of the digital assistant application 108. The instance of the digital assistant application 108 on the client device 104 may filter out one or more items or modify the items before sending the items as data to the data processing system 102 (e.g., an instance of the digital assistant application 108 on the data processing system 102) for further processing. The instance of the digital assistant application 108 on the client device 104 may convert an analog audio signal detected by the speaker 144 into a digital audio signal, and send one or more data packets carrying the digital audio signal to the data processing system 102 via the network 114. The instance of the digital assistant application 108 on the client device 104 can transmit a data packet carrying some or the entire input audio signal in response to detecting an instruction to perform such a transmission. The instruction can include, for example, a trigger keyword or other keyword or approval to send a data packet including the input audio signal to the data processing system 102.

[0043] The instance of the digital assistant application 108 on the client device 104 can perform pre-filtering or pre-processing on the input audio signal to remove certain frequencies of the audio. Pre-filtering can include filters such as low-pass filters, high-pass filters, or band-pass filters. The filter can be applied in the frequency domain. The filter can be applied using digital signal processing techniques. The filter can maintain the frequency corresponding to human speech or human speech while eliminating the frequency outside the typical frequency of human speech. For example, the band-pass filter can remove frequencies below a first threshold (e.g., 70Hz, 75Hz, 80Hz, 85Hz, 90Hz, 95Hz, 100Hz, or 105Hz) and above a second threshold (e.g., 200Hz, 205Hz, 210Hz, 225Hz, 235Hz, 245Hz, or 255Hz). Applying a band-pass filter can reduce the computing resource utilization in downstream processing. The instance of the digital assistant application 108 on the client device 104 can apply a band-pass filter before sending the input audio signal to the data processing system 102, thereby reducing network bandwidth utilization. However, based on the computing resources available to the client device 104 and the available network bandwidth, it may be more efficient to provide the input audio signal to the data processing system 102 to allow the data processing system 102 to perform the filtering. The instance of the digital assistant application 108 on the client device 104 may apply additional pre-processing or pre-filtering techniques, such as noise reduction techniques, to reduce the level of ambient noise that may interfere with the natural language processor. Noise reduction techniques may improve the accuracy and speed of the natural language processor, thereby improving the performance of the data processing system 102 and managing the rendering of the graphical user interface provided via the display.

[0044] The client device 104 may be operated by an end user who inputs a voice query as an audio input (via a microphone 146 or a speaker 144) into the client device 104 and receives an audio (or other) output from the data processing system 102 or the proxy service 106 to present, display, or render to the end user of the client device 104. The digital component may include a computer-generated voice that may be provided to the client device 104 from the data processing system. The client device 104 may render the computer-generated voice to the end user via the speaker 144. The computer-generated voice may include a recording from a real person or computer-generated speech. The client device 104 may provide a visual output via a display 148 communicatively coupled to the client device 104.

[0045] The client device 104 may have or execute as at least one foreground process 150 (also referred to herein as a foreground service). The foreground process 150 may correspond to at least one of the applications (e.g., the digital assistant application 108 or the proxy application 110) rendered on the display 148 and executed on the client device 104. For example, the foreground process 150 may correspond to an application having a graphical user interface that occupies a majority of the display 148. An application running as a foreground process 150 on the client device 104 may have a graphical user interface rendered on the display 148 of the client device 104. The graphical user interface of the application in the foreground process 150 may occupy at least a portion of the display 148 of the client device 104 (e.g., at least one third of the display 148). The application running as a foreground process 150 on the client device 104 may wait (e.g., via an I / O device communicatively coupled to the client device 104) for user interaction with the graphical user interface to execute one or more routines. The application in the foreground process 150 may process the user interaction to execute one or more routines of the application. In the process of user interaction, the application in the foreground process 150 may generate output, and the resulting output may be rendered on a graphical user interface of the application in the foreground process 150 .

[0046] The client device 104 may have or execute at least one non-foreground process 152 (also referred to herein as a background process, background service, non-foreground process, or non-foreground service). The non-foreground process 152 may correspond to at least one application (e.g., a digital assistant application 108 or another proxy application 110) that is not rendered on the display 148 and executed on the client device 104. An application running as a non-foreground process 152 on the client device 104 may not render any graphical user interface of the application on the display 148 of the client device 104. For example, the non-foreground process 152 may correspond to an application that is minimized and does not render any graphical user interface components in the display 148. An application running as a non-foreground process 152 on the client device 104 may execute one or more routines without receiving any user interaction with the application. For example, an application running as a non-foreground process 152 may process audio detected on the microphone 146 of the client device 104 without rendering any graphical user interface on the display 148. In the process of user interaction, the application in the foreground process 150 may generate output without rendering the resulting output as part of the graphical user interface rendered on the display 148 .

[0047] The client device 104 can manage applications in the foreground process 150 and the non-foreground process 152 (e.g., using an operating system executing on the client device 104). The client device 104 can switch applications between the foreground process 150 and the non-foreground process 152. Switching applications between the foreground process 150 and the non-foreground process 152 can be in response to a call by at least one application. The called application can be running in the non-foreground process 152, or it can be running in neither the foreground process 150 nor the non-foreground process 152. The call can be made by the operating system executing on the client device 104, by another application (e.g., the digital assistant application 108 or the agent application 110), or through a user interaction event. In response to the call, the client device 104 can set the application running in the foreground process 150 to the non-foreground process 152. In addition, the client device 104 can set the application running in the non-foreground process 152 to the foreground process 150. When the called application is not running as a non-foreground process 152, the client device 104 can also instantiate the application to the foreground process 150. For example, a web browser may be executed as a foreground process 150 on the client device 104, while rendering a graphical user interface on the display 148. A clock application may be run as a non-foreground process 152 on the client device 104 to track time without rendering any graphical user interface on the display 148. When the alarm function of the clock application is invoked, the client device 104 may set the web browser as a non-foreground process 152, thereby removing the rendering of the web browser's graphical user interface. In combination, the client device 104 may set the clock application as the foreground process 150.

[0048] The data processing system 102 and the proxy service 106 may include at least one server having at least one processor, respectively. For example, the data processing system 102 and the proxy service 106 may include a plurality of servers located in at least one data center or server farm, respectively. The data processing system 102 may include at least one computing resource or server. The data processing system 102 may include at least one communication interface 112, which is docked or otherwise communicated with. The data processing system 102 may include at least one instance of a digital assistant application 108 on the data processing system 102, which is docked or otherwise communicated with. The instance of the digital assistant application 108 on the data processing system 102 may include at least one NLP component 116, at least one audio signal generator component 118, at least one action processor component 120, at least one response selector component 122, at least one proxy registry component 124, and at least one proxy interface component 126, etc., which are docked or otherwise communicated with. The data processing system 102 may include at least one data repository 128, which is docked or otherwise communicated with. At least one data repository 128 may include or store regular expressions 130, data parameters 132, policies 134, response data 136, templates 138, and action lists 140 in one or more data structures or databases. Data repository 128 may include one or more local or distributed databases and may include database management.

[0049] An instance of the digital assistant application 108 of the data processing system 102 may execute or run an instance of the proxy registry component 124 to maintain a set of action lists 140 on the data repository 128. The proxy registry component 124 may maintain and store a set of action lists 140 for one or more agent applications 110 on the data repository 128. A subset of the action lists 140 may be defined by an administrator of the data processing system 102 or the digital assistant application 108 for an application type corresponding to the agent application 110. For example, an administrator of the digital assistant application 108 may configure or define a set of built-in action lists 140 for a purchase order application. Another subset of the action lists 140 may be provided by one of the agent services 106 that processes resources for the corresponding agent application 110. At least one of the action lists 140 may be a preset action list 140 for performing an action in response to determining that a request does not match any other action list 140 of the agent application 110. For example, the proxy service 106 may provide a set of customized action manifests 140 to the data processing system 102 for storage on the data repository 128. The proxy registry component 124 may receive a subset of the action manifests 140 provided by the proxy service 106. Upon receipt, the proxy registry component 124 may store the received action manifests 140 on the data repository 128. The presence of an action manifest 140 for an action of the proxy application 110 on the data repository 128 may indicate that the proxy application 110 is capable of performing the action corresponding to the action manifest 140.

[0050] The proxy registry component 124 can maintain and store a set of user interface components 142 for one or more proxy applications 110 on the data repository 128. Each user interface component 142 can be associated with at least one of the action lists 140. A subset of the user interface components 142 can be defined by an administrator of the data processing system 102 or the digital assistant application 108. For example, the administrator of the digital assistant application 108 can configure or define a set of built-in user interface components 142. Another subset of the user interface components 142 can be provided by a proxy service in the proxy service 106 that processes resources for the corresponding proxy application 110. The proxy service 106 can store and maintain the user interface components 142 corresponding to the subset. For example, the proxy service 106 can provide a set of customized user interface components 142 to the data processing system 102 for storage on the data repository 128. The proxy registry component 124 can receive a subset of the user interface components 142 provided by the proxy service 106. After receiving, the proxy registry component 124 can store the received user interface components 142 on the data repository 128. Maintenance of the user interface component 142 by the proxy service 106 may be separate from maintaining the user interface component 142 on the data repository 128 .

[0051] Each user interface component 142 may have one or more interface elements. Each interface element may correspond to a subcomponent of the user interface component 142. At least one interface element may correspond to a request to be identified from an input audio signal detected by a microphone 146 of the client device 104. At least one interface element may correspond to a response indicating an output generated by an agent application 110 associated with the user interface component 142. In addition, each user interface component may have one or more attributes. These attributes may include element type (e.g., command button, scroll bar, text box, and image), size, position within the user interface component 142, transparency, and shape, etc. For example, the user interface component 142 may have a button generally placed along the bottom, a text box above the button, and a slot for an image generally in the middle, etc. The user interface component 142 itself may have one or more attributes. The attributes of the user interface component 142 may include size, position within the display 148, transparency, and shape, etc.

[0052] Each action list 140 may have at least one address template for at least one action of one of the proxy applications 110. The address template may be used for a single action of the proxy application 110 or for multiple actions carried by the proxy application 110. The address template may also be used to access or retrieve a user interface component 142 to be generated when the action is performed. The address template for the action may include a first part and a second part. The address template may be, for example, a uniform resource identifier (URI) template from which a URI is generated. The first part of the address template may correspond to one of the proxy applications 110 or a proxy service 106 that provides resources for the proxy application 110. For example, the first part of the address template may include a scheme in a custom deep link URI that references the proxy application 110. The first part of the address template may also include a host name that references the proxy service 106.

[0053] The second part of the address template may correspond to an action to be performed by the proxy application 110. The second part of the address template may also include one or more input variables (also referred to herein as input variables of a field) for performing the action. For example, the second part of the address template may include a path name corresponding to the action to be taken in the URI. The second part may also include one or more query strings after the path name in the URI. Each query string may correspond to one of the input variables for performing the action. A subset of the input variables may include optional input variables or auxiliary input variables. The action may be performed by the proxy application 110 without auxiliary input variables in the address.

[0054] The action list 140 may include a mapping (sometimes referred to herein as a parameter mapping) for the second part of the address template. The mapping may specify or define a correspondence between at least one input variable of the second part and one or more words to be identified from the parsed input audio signal. One or more words from the parsed input audio signal may correspond to the input variables defining the action to be performed by the agent application 110. The mapping may define the insertion of words into the input variables of the address template. For example, a mapping for a coffee purchase order action may specify that the coffee name parsed from the audio signal is inserted into the coffee input variable of the address template. The action list 140 may specify or include a set of allowed values ​​(sometimes referred to as an entity list) for each input variable. This set of allowed values ​​may be part of the mapping of the action list 140. For example, for the input variable of the coffee name, the allowed values ​​of the coffee purchase order may include "latte", "cafe latte", "latte cafe", etc. The action list 140 may also include a tag indicating that the corresponding input variable of the address template is optional or auxiliary. The tag may be part of the mapping for the second part of the address template.

[0055] In addition to the address template, each action list 140 maintained by the proxy registry component 124 may include information related to the action for the action list 140. The action list 140 may include an interface mode indicator. The indicator may specify whether a user interface component 142 of the proxy application 110 will be displayed when the action is performed. The user interface component 142 may be a visual element to be rendered in a display 148 of the client device 104. The size of the user interface component 142 may be smaller than the size of the display 148. For example, the user interface component 142 may include a box-sized visual element to occupy 12.5% ​​to 25% of the display 148. The user interface component 142 may be associated with at least one of the action lists 140 maintained on the data repository 128.

[0056] The user interface component 142 may be associated with the action list 140. The action list 140 may include an address that references the user interface component 142. The address that references the user interface component 142 may be different or separate from the address template used to perform the action. The address that references the user interface component 142 may have the same scheme or host name as the address template for the same action list 140. For example, the address may include the host name of the proxy service 106 that manages resources for the proxy application 110. The address that references the user interface component 142 may have a different host name from the address template of the same action list 140. For example, the host name may reference the data repository 128 of the data processing system 102. The action list 140 may also include a user interface identifier corresponding to the user interface component 142. The identifier may be used to index and reference the user interface component 142 maintained on the data repository 128. In addition, the address template of the action list 140 may be used to reference the user interface component 142 and the action to be performed by the proxy application 110.

[0057] The action list 140 may also include at least one agent identifier corresponding to one of the agent applications 110 that perform the action. The agent identifier may include a set of alphanumeric characters that reference the agent application 110. The agent identifier may be used to identify the agent application 110 associated with the action list 140 based on the words parsed from the input audio signal. The action list 140 may include at least one request identifier corresponding to the action to be performed by the agent application 110. The request identifier (sometimes referred to as an "intention identifier" or "intention name" in this article) may include a set of alphanumeric characters that reference the action to be performed. The address template may be uniquely identified by one or more request identifiers. The request identifier may be used to identify the action list 140 based on the words parsed from the input audio signal.

[0058] Each action list 140 may be implemented or specified using a markup language such as an extensible markup language (XML) file. For example, the action list 140 may be an XML file in the following form:

[0059]

[0060] In the example above, action list 140 can be used to order rides performed by a personal fitness application. The entry "actions.intent.CHECK_STEPS" can correspond to the request identifier of action list 140. The entry "actions.fulfillment.SLICE" can specify the display of user interface component 142 associated with action list 140. The entry "https: / / ex_personalmetrics.example.com / checksteps{?username}" can correspond to an address template, where "https: / / ex_personalmetrics.example.com / corresponds to the first part and "checksteps{?username}" corresponds to the second part. Entries starting with "parameter-mapping" can correspond to mappings. Finally, the entry "https: / / ex_personal metrics.example.com / checksteps" can correspond to a separate address for user interface component 142.

[0061] The proxy registry component 124 may construct, create or generate a set of action lists 140 for each proxy application 110. The proxy service 104 of the proxy application 110 may provide a configuration file associated with the proxy application 110 to the proxy registry component 124 via the network 114. The configuration file may be, for example, an application binary interface (ABI) of the proxy application 110 submitted to the proxy registry component 124 by an application developer associated with the proxy application 110. The proxy service 104 of the proxy application 110 may provide metadata associated with the proxy application 110. The metadata may include a proxy identifier and an application type to the proxy registry component 124, etc. The application type may indicate the use of the proxy application 110. The proxy service 104 of the proxy application 110 may provide an interface mode indicator for one or more actions that can be performed by the proxy application 110. The proxy registry component 124 may receive information from the proxy service 106 for generating a set of action lists 140 for the proxy application 110.

[0062] The proxy registry component 124 can read, ingest and parse the configuration file for the proxy application 110 to identify one or more actions. For each identified function, the proxy registry component 124 can generate an action list 140 for the action. The action can be one of the actions in the set of action lists 140 defined by the administrator of the data processing system 102 or the digital assistant application 108 for the application type of the proxy application 110. The proxy registry component 1240 generates an address template for the action performed by the proxy application 110. The first part of the address template (e.g., the host name or scheme) can correspond to the proxy application 110. The second part of the address template can be predefined for the action (e.g., using a preset path name or query string). The proxy registry component 124 can also generate a mapping of the action list 140 for the identified function. The proxy registry component 124 can include or add a request identifier corresponding to the identified action to the action list 140. The mapping can include a set of allowed values ​​for the input variables of the address template. The proxy registry component 124 can include or add a proxy identifier corresponding to the proxy application 110 to the action list 140. With the action manifest 140 constructed, the proxy registry component 124 can store the action manifest 140 on the data repository 128 .

[0063] The data processing system 102 may include at least one communication interface 112. The communication interface 112 may be configured, constructed or operated to receive and send information using, for example, data packets. The communication interface 112 may receive and send information using one or more protocols such as network protocols. The communication interface 112 may include a hardware interface, a software interface, a wired interface or a wireless interface. The communication interface 112 may be a data interface or a network interface that enables the components of the system 100 to communicate with each other. The communication interface 112 of the data processing system 102 may provide or send one or more data packets including action data structures, audio signals or other data to the client device 104 or the proxy service 106 via the network 114. For example, the data processing system 102 may provide an output signal from the data repository 128 or the audio signal generator component 118 to the client device 104.

[0064] The data processing system 102 may also instruct the client device 104 to perform the function indicated in the action data structure via the data packet transmission. The output signal may be acquired, generated, converted to one or more data packets (or other communication protocols), or sent from the data processing system 102 (or other computing device) to the client device 104 as one or more data packets (or other communication protocols). The communication interface 112 may facilitate the conversion or formatting of data from one format to another. For example, the communication interface 112 may include an application programming interface (“API”) that includes definitions for communicating between various components such as software components. An application, script, program, or other component associated with the data processing system 102 may be installed at the client device 104. The application may enable the client device 104 to transmit the input audio signal (and other data) to the communication interface 112 of the data processing system 102.

[0065] The data processing system 102 may include an application, script, or program installed on the client device 104, such as an instance of a digital assistant application 108 on the client device 104, to transmit the input audio signal to the communication interface 112 of the data processing system 102, and drive the components of the client computing device to render the output audio signal or visual output. The data processing system 102 may receive data packets, digital files, or other signals including or identifying the input audio signal (or multiple input audio signals). The client device 104 may detect the audio signal via a speaker 144, and convert the analog audio signal to a digital file via an analog-to-digital converter. For example, the audio driver may include an analog-to-digital converter component. The preprocessor component may convert the audio signal to a digital file that can be transmitted via a data packet through the network 114.

[0066] An instance of the digital assistant application 108 of the data processing system 102 or the client device 104 can execute or run the NLP component 116 to receive or obtain a data packet including an input audio signal detected by the microphone 146 of the client device 104. The data packet can provide a digital file. The NLP component 116 can receive or obtain a digital file or data packet including an audio signal and parse the audio signal. For example, the NLP component 116 can provide interaction between a person and a computer. The NLP component 116 can be configured with a technology for understanding natural language and causing the data processing system 102 to derive meaning from human or natural language input. The NLP component 116 can include or be configured with a technology based on machine learning (such as statistical machine learning). The NLP component 116 can parse the input audio signal using a decision tree, a statistical model, or a probability model.

[0067] For example, the NLP component 116 can perform functions such as named entity recognition (e.g., given a stream of text, determining which items in the text map to names such as people or places, and the type of each such name, such as a person, a location (e.g., "home"), or an organization), natural language generation (e.g., converting information or semantic intent in a computer database into understandable human language), natural language understanding (e.g., converting text into a more formal representation, such as a first-order logic structure that a computer module can manipulate), machine translation (e.g., automatically translating text from one human language to another), morpheme segmentation (e.g., separating words into individual morphemes and identifying categories of morphemes, which can be challenging given the complexity of the morphemes or structure of the words of the language), question answering (e.g., determining answers to human language questions, which can be specific or open-ended), and semantic processing (e.g., processing that can be performed after words are identified and their meanings are encoded to relate the identified words to other words with similar meanings).

[0068] The NLP component 116 (and the digital assistant application 108, as a whole) can be executed as one of the applications in the non-foreground process 152. When an input audio signal is detected on the microphone 146, the client device 104 can continue the digital assistant application 108 as one of the applications in the non-foreground process 152. The client device 104 can have one or more other applications (e.g., the proxy application 110) running as non-foreground processes 152. The client device 104 can have at least one application (e.g., another proxy application 110) running in the foreground process 150. An instance of the NLP component 116 of the digital assistant application 108 on the client device 104 can initiate processing of the input audio signal without switching the foreground process 150 or the non-foreground process 152. The applications running as the foreground process 150 and the non-foreground process 152 can be maintained. For example, the NLP component 116 on the client device 104 can receive the input audio signal and apply natural language processing functions to the input audio signal without minimizing the graphical user interface of the application running as the foreground process 150. When the input audio signal is received, the NLP component 116 (and the digital assistant application 108, as a whole) can alternatively execute as one of the applications in the foreground process 150. After receiving the input audio signal, the digital assistant application 108 can continue to execute as one of the applications in the foreground process 150 of the client device 104.

[0069] The NLP component 116 can parse the audio input signal and convert the audio input signal into a recognized string by comparing the input signal with a stored representative audio waveform set (e.g., in the data repository 128) and selecting the closest match. The NLP component 116 can also segment or divide the input audio signal into one or more audio segments with a duration (e.g., 15 seconds to 2 minutes) to process each segment. The audio waveform set can be stored in the data repository 128 or other databases accessible to the data processing system 102. The representative waveform is generated between a large number of users and can then be enhanced using speech samples from the user. After the audio signal is converted into recognized text, the NLP component 116 matches the text with words associated with actions that can be dispatched by the data processing system 102, such as through training between users or by manual designation. The NLP component 116 can determine that the input audio signal obtained from the microphone 146 does not contain any recognizable string. When determining that the input audio signal does not contain any recognizable string, the NLP component 116 can determine that the input audio signal contains silence (e.g., the maximum amplitude is less than 0dB). In addition, the NLP component 116 can determine a signal-to-noise ratio (SNR) of the input audio signal. The NLP component 116 can compare the SNR of the input audio signal to a threshold SNR (e.g., -20 dB). In response to determining that the SNR of the input audio signal is greater than the threshold SNR, the NLP component 116 can determine that the input audio signal does not contain any recognizable character strings.

[0070] In addition to or in lieu of an input audio signal, the data processing system 102 may receive an image or video input signal. The NLP component 116 may convert the image or video input into a text or digital file. The NLP component 116 may process, analyze, or interpret the image or video input to perform an action, generate a request, or select or identify a data structure. The data processing system 102 may process the image or video input signal using, for example, image interpretation technology, computer vision, a machine learning engine, or other technology to identify or interpret the image or video to convert the image or video into a digital file. One or more image interpretation technologies, computer vision technologies, or machine learning technologies may be collectively referred to as imaging technologies. In addition to or in lieu of audio processing technology, the data processing system 102 (e.g., the NLP component 116) may be configured using imaging technology.

[0071] By parsing the input audio signal, the NLP component 116 can determine or identify at least one request. The input audio signal may include, for example, a query, question, command, instruction or other statement in natural language. The request may correspond to at least one trigger keyword, which is identified from the recognized string converted from the input audio signal. The request may indicate the action to be taken. For example, the NLP component 116 may parse the input audio signal to identify at least one request to leave home for dinner and watch a movie at night. The trigger keyword may include at least one word, phrase, root or partial word, or derivative indicating the action to be taken. For example, the trigger keyword "go" or "to go to" from the input audio signal may indicate a request for transportation. In this example, the input audio signal (or the recognized request) does not directly express the transmission intention, but the trigger keyword indicates that the transmission is an auxiliary action of at least one other action indicated by the request.

[0072] To identify the request, the NLP component 116 may apply semantic processing techniques to the input audio signal to identify a trigger keyword corresponding to the request. The NLP component 116 may apply semantic processing techniques to the input audio signal to identify a trigger phrase including one or more trigger keywords, such as a first trigger keyword and a second trigger keyword. For example, the input audio signal may include the sentence "Find the nearest café". The NLP component 116 may determine that the input audio signal includes the trigger keyword "find". The NLP component 116 may determine that the request is to search for a location near the client device 104.

[0073] In addition to the request, the NLP component 116 can identify at least one parameter that defines the request. The parameter can define the request and serve as a supplement or restriction to the action to be taken corresponding to the request. The parameter can further specify the action to be taken. The parameter can include a subset of the recognized string converted from the input audio signal (excluding the trigger keyword). The NLP component 116 can apply semantic processing techniques to the input audio signal to identify one or more descriptor words related to the identified trigger keyword. From the example "Find the nearest café", the NLP component 116 may have identified the item "Find" as a trigger keyword. Combined, the NLP component 116 can identify "nearest" as a first parameter and "café" as a second parameter that defines the request "Find".

[0074] In addition, the NLP component 116 can identify an application identifier based on a string identified from the input audio signal. The application identifier may correspond to one of the proxy applications 110. In order to identify the application identifier, the NLP component 116 may perform a named entity recognition algorithm on the string identified from the input audio signal. When using the named entity recognition algorithm, the NLP component 116 may maintain a list of proxy identifiers of the proxy application 110. The list may include proxy identifiers of the proxy application 110 installed on the client device 104 that receives the input audio signal including the request. By applying the named entity recognition algorithm, the NLP component 116 may determine that the input audio signal includes an application identifier corresponding to the proxy application 110. For example, the NLP component 116 may determine that the input audio signal "Order car with ride sharing service XYZ (booking a car with ride sharing service XYZ)" includes an explicit proxy application 110 "ride sharing service XYZ". On the contrary, by applying the named entity recognition algorithm, the NLP component 116 may determine that the input audio does not have any application identifier corresponding to any proxy application 110.

[0075] Based on the request, the NLP component 116 can determine whether the request corresponds to a function of the digital assistant application 108 or a function of the proxy application 110. To determine this, the NLP component 116 can access a list of requests for functions of the digital assistant application 108. The list of requests for functions of the digital assistant application 108 can be maintained in the data repository 128. The request list can include a string set of trigger keywords and requests corresponding to requests that are predetermined to be associated with the functions of the digital assistant application 108. For example, the list can include "take", "show", "search for", and "find", etc. Utilizing the identification of the request, the NLP component 116 can compare the request with the list of requests for functions of the digital assistant application 108.

[0076] The NLP component 116 may also use a regular expression 130 maintained on the data repository 128 to determine whether the request corresponds to a function of the digital assistant application 108. The regular expression 130 may define a pattern for matching to determine whether a keyword identified from the input audio signal references at least one function of the digital assistant application 108. The regular expression 130 may also specify which keywords are used to execute the command indicated in the input audio signal. For example, the regular expression 130 may take the form of {[request], [auxiliary arguments]}. For keywords of the input audio signal that are determined to reference a function of the digital assistant application 108, the regular expression 130 may specify one or more keywords including a request for the digital assistant application 108 and auxiliary arguments. The regular expression 130 may specify the order of the request and the referenced keywords in the one or more keywords identified from the input audio signal.

[0077] Based on the comparison, the NLP component 116 can determine that the request identified from the input audio signal matches one of the requests in the request list for functions of the digital assistant application 108. In response to the determination, the NLP component 116 can determine that the request corresponds to one of the functions of the digital assistant application 108. For example, the NLP component 116 can parse the words "What is the weather?" from the input audio signal, and can identify that the input audio signal includes a request for the weather. The request list for functions of the digital assistant application 108 can specify that the request for the weather is one of the functions of the digital assistant application 108. In this example, the NLP component 116 can determine a match between the two requests, and can determine that the request references one of the functions of the digital assistant application 108.

[0078] In response to determining that the request corresponds to the function of the digital assistant application 108, the digital assistant application 108 can execute and satisfy the request identified from the parsed input audio signal. In addition, the digital assistant application 108 can execute the request according to the parameters defining the request. For example, the request can be a definition of a search term (e.g., "abnegation"). The digital assistant application 108 can perform an Internet search to retrieve the definition of the term. When satisfying the request, the digital assistant application 108 can call a response selector component 122. The response selector component 122 can select or identify a response phrase using a strategy 134 or response data 136 maintained on a data repository 128. The strategy 134 can be specific to the request and can specify the response data 136 for the request. The response selector component 122 can search for a strategy 134 using the request type of the response to generate an output when satisfying the request. Once the strategy 134 is identified, the response selector component 122 can select a response phrase. The response phrase can include a set of strings, such as words or phrases. The digital assistant application 108 may display the response phrase on the display 148 of the client device 104. For example, the digital assistant application 108 may display the content item including the response phrase on a graphical user interface of the digital assistant application 108 on the display 148 of the client device 104.

[0079] Utilizing the recognition of the response phrase, the digital assistant application 108 can call and audio signal generator component 118. The instance of the digital assistant application 108 running on the data processing system 102 or the client device 104 can execute the audio signal generator component 118. Using the selected response phrase, the audio signal generator component 118 can generate an output audio signal. Based on one or more words included in the response phrase, the audio signal generator component 118 can generate an output audio signal. The audio signal generator component 118 can play the output audio signal on the speaker 144 of the client device 104.

[0080] On the contrary, by comparison, the NLP component 116 can determine that the request parsed from the input audio signal does not match any request in the request list for the functions of the digital assistant application 108. In response to this determination, the NLP component 116 can determine whether the request corresponds to one of the functions of the agent application 110. In order to determine this, the NLP component 116 can access the request list for the functions of the agent application 110. The request list for the functions can be for the agent application 110 installed on the client device 104. The request list for the functions of the agent application 110 can be maintained on the data repository 128. The request list can include a string set of trigger keywords or requests corresponding to requests that are predetermined to be associated with the functions of the agent application 110. The NLP component 116 can compare the request with the request list for the functions of the agent application 110.

[0081] By comparison, the NLP component 116 can determine that the request identified from the input audio signal matches one of the requests in the request list for the functions of the agent application 110. In response to the determination, the NLP component 116 can determine that the request corresponds to one of the functions of the agent application 110. For example, the NLP component 116 can parse the words "Buy melaundry detergent" from the input audio signal, and can identify that the input audio signal includes buying detergent. The request list for the functions of the agent application 110 can specify that the request for weather is one of the functions of the agent application 110. In this example, the NLP component 116 can determine a match between the two requests, and can determine that the request references one of the functions of the agent application 110, namely, a function for such a purchase. In addition, in response to determining that the input audio signal includes an application identifier of the agent application 110, the NLP component 116 can determine that the request does not correspond to a function of the digital assistant application 108.

[0082] On the other hand, the NLP component 116 can determine by comparison that the request does not correspond to any function of any proxy application 110. By determining that the request does not correspond to the digital assistant application 108 or the proxy application 110, the response selector component 122 can generate or identify a response phrase indicating that the request does not correspond to any function available on the client device 014. For example, the response phrase can include "Sorry, I don't understand your request". The response phrase can be selected from the response data 136 based on the determination. In addition, the audio signal generator component 118 can use one or more words in the response phrase to generate a corresponding output audio signal. The audio signal generator component 118 can play the output audio signal on the speaker 144 of the client device 104.

[0083] An instance of the digital assistant application 108 running on the data processing system 102 or the client device 104 can execute an action handler component 120 to generate an action data structure based on the determination. The action handler component 120 can execute a script or program based on the input received from the NLP component 116. The proxy service 106 can provide the script or program. The proxy service 106 can make the script or program available to the data processing system 102 through an API. The action handler component 120 can determine the parameters or responses to the input fields and can package the data into an action data structure. The action data structure can be provided to the proxy application 110 through an API. The action handler component 120 can send the action data structure to the proxy service 106 to satisfy the request identified from the input audio signal parsed by the NLP component 116.

[0084] In response to determining that the request corresponds to a function of the proxy application 110, the action handler component 120 can identify or select an action manifest 140 from the data repository 128 for the action corresponding to the request. The action handler component 120 can select at least one action manifest 140 from the set of action manifests 140 maintained on the data repository 128. The action handler component 120 can use the request identified from the input audio signal to search the action repository 128 for the action manifest 140. The action handler component 120 can also use one or more parameters identified from the input audio signal to search the data repository 128 for the action manifest 140. The action handler component 120 can traverse the set of action manifests 140 maintained on the data repository 128. For each action manifest 140, the action handler component 120 can identify a request identifier for the action manifest 140.

[0085] The action handler component 120 can compare the request identified from the input audio signal with the request identifier of the action list 140. Comparing the request identifier from the input audio signal with the request identifier of the action list 140 can be based on the semantic knowledge graph. The action handler component 120 can call the NLP component 116 to use the semantic knowledge graph to determine whether the request is semantically related to the request identifier. The semantic knowledge graph can include a set of nodes and edges. Each node can correspond to one or more words or phrases. Each edge can define the semantic distance between a pair of words represented by the corresponding words on the node. Through the semantic knowledge graph, the NLP component 116 can identify the node corresponding to the request, the node corresponding to the request identifier, and the semantic distance indicated in the pair of nodes. The NLP component 116 can compare the semantic distance with the threshold distance. When it is determined that the semantic distance does not meet the threshold distance (e.g., greater than the threshold distance), the NLP component 116 can determine that the request does not match the request identifier of the action list 140. On the other hand, when it is determined that the semantic distance satisfies the threshold distance (e.g., is less than or equal to the threshold distance), the NLP component 116 can determine that the request does match the request identifier of the action manifest 140. Through the comparison, the action handler component 120 can determine that the request does not match the request identifier of the action manifest 140. In response to this determination, the action handler component 120 can identify the next action manifest 140 in the set maintained on the data repository 128, and can repeat the comparison.

[0086] On the contrary, through comparison, the action handler component 120 can determine that the request matches the request identifier of the action list 140. In response to the determination, the action handler component 120 can select an action list 140 for the action corresponding to the request. Because, as described above, multiple requests can correspond to the same action list 140, the action handler component 120 can select an action list 140 that was or will be selected for another request. Before selecting the action list 140, the action handler component 120 can also compare one or more parameters identified from the input audio signal with the input variables of the action list 140. The action handler component 120 can identify the input variables from the parameter mapping or address template of the action list 140. When comparing the identified parameters with the input variables of the action list 140, the action handler component 120 can exclude the auxiliary parameters indicated in the action list 140. The action handler component 120 can determine that the identified parameters match the input variables of the action list 140 based on the allowed value set of each input variable. In response to the determination, the action handler component 120 may select an action list 140 for an action corresponding to the request defined by one or more parameters. The action handler component 120 may also stop traversing the set of action lists 140 in the data repository 128. Otherwise, the action handler component 120 may determine that the identified parameters do not match the input variables of the action list 140. The action handler component 120 may identify another action list 140 for an action corresponding to the same request, and may repeat the comparison. After comparing with all action lists 140, the action handler component 120 may determine that the identified parameters do not match the input variables in any action list 140 of the proxy application 110. In response to the determination, the action handler component 120 may select a preset action list 140 for the proxy application 110 to respond to the request.

[0087] The action handler component 120 can also search the action list 140 from the data repository 128 using the agent identifier identified from the input audio signal. Before searching using the request, the action handler component 120 can identify a subset of the action list 140 with an application identifier that matches the agent identifier. The agent identifier can correspond to one of the agent applications 110. Each action list 140 of the identified subset can include an application identifier that is determined to match the agent identifier identified from the input audio signal. In order to identify the subset, the action handler component 120 can compare the agent identifier with the application identifier of each action list 140. When the agent identifier is determined to match the application identifier of the action list 140, the action handler component 120 can include the action list 140 in the subset. Otherwise, when the agent identifier is determined to not match the application identifier of the action list 140, the action handler component 120 can exclude the action list 140 from the subset. The action handler component 120 can traverse the identified subset of the action list 140 for the application identifier using the request to select the action list 140 for the action corresponding to the request.

[0088] When the input audio signal is determined to not have an agent identifier, the action handler component 120 can select an action list 140 from a set maintained on the data repository 128 based on the request, one or more parameters, and the agent usage history. The agent usage history can indicate the usage statistics of the agent application 110 on the client device 104 running the digital assistant application 108 from which the input audio signal is received. The agent usage history can also indicate the functions of the agent application 110 called on the client device 104. Before the request identified from the input audio signal detected on the client device 104, the agent usage history can be limited to a defined time window. The action handler component 120 can use the request identifier that matches the request and one or more parameters to identify a subset of the action list 140. The subset may include an action list 140 with different application identifiers corresponding to multiple agent applications 110. The action handler component 120 can use the agent usage history of the client device 104 to select the action list 140 from the subset. The action handler component 120 can identify at least one agent identifier from the agent usage history. The action handler component 120 may compare the agent identifier with the application identifier of each action list 140 in the identified subset. Through the comparison, the action handler component 120 may determine that the application identifier of the action list 140 matches the agent identifier from the agent usage history. In response to the determination, the action handler component 120 may select an action list 140 for an action to be performed by the agent application 110 based on the parameter. Conversely, through the comparison, the action handler component 120 may determine that the application identifier of the action list 140 does not match the agent identifier from the agent usage history. In response to the determination, the action handler component 120 may identify another action list 140 and may repeat the comparison using the agent identifier.

[0089] With the identification of the action list 140, the action handler component 120 can verify one or more parameters identified from the input audio signal against the address template of the action list 140. The verification of the action handler component 120 can be to check whether the necessary parameters specified by the action list 140 are present to perform the action corresponding to the request. For example, for a ride sharing request, the action handler component 120 can check whether there are parameters corresponding to the destination parsed from the input audio signal. The action handler component 120 can use the parameter mapping for the address template of the action list 140 when verifying the parameters. The action handler component 120 can identify the correspondence between the parameters and input variables defined in the mapping for the action list 140. The action handler component 120 can identify a subset of the correspondence of the input variables that are not specified as auxiliary through the mapping of the action list 140 of the address template.

[0090] With each input variable that is not indicated as auxiliary, the action handler component 120 can determine whether at least one parameter corresponds to the input variable specified by the mapping. The determination of the correspondence can be performed using a semantic analysis algorithm (e.g., a semantic knowledge graph) provided by the NLP component 116. For each input variable, the action handler component 120 can call the NLP component 116 to determine whether the input variable corresponds to any parameter, and vice versa. In response to determining that all input variables correspond to at least one parameter, the action handler component 120 can determine that the parameter is successfully verified. The action handler component 120 can also continue to use the address template of the action list 140 to generate an address to implement the action corresponding to the request.

[0091] On the other hand, in response to determining that at least one input variable (not indicated as auxiliary) does not correspond to any parameter, the action handler component 120 can determine that the parameter has not been successfully verified. The action handler component 120 can stop or terminate further processing of the request and parameters (e.g., generating an address using an address template of the action list 140). The action handler component 120 can detect or determine an error in verifying one or more parameters identified from the input audio signal. The digital assistant application 108 can present an indication that the request is invalid or unsuccessful (e.g., using a visual content item or audio). The action handler component 120 can provide an indication of the error to the agent service 106 or an administrator of the agent application 110 associated with the action list 140.

[0092] After identifying the action list 140, the action handler component 120 can determine whether the client device 104 is authenticated by the proxy application 110 associated with the action list 140. The action handler component 120 can identify the proxy application 110 associated with the action list 140. Once identified, the action handler component 120 can use the proxy application 110 to check the authentication status of the client device 104. The account identifier of the client device 104 or the digital assistant application 108 can be used to perform the authentication status check. The action handler component 120 can send a request for the authentication status to the proxy service 106 that manages the resources of the proxy application 110. The request may include the account identifier of the client device 104 or the digital assistant application 108. The proxy service 106 can provide the action handler component 120 with an indication of whether the client device 104 is authenticated by the proxy application 110. When the response indicates that the client device 104 is authenticated, the action handler component 120 can determine that the client device 104 is authenticated by the proxy application 110. The action handler component 120 can also continue to process the request. On the other hand, when the response indicates that the client device 104 is not authenticated, the action handler component 120 can determine that the client device 104 is not authenticated by the proxy application 110. The action handler component 120 can stop or terminate further processing of the request (e.g., generating an address using an address template of the action list 140). The digital assistant application 108 can also present an indication to authenticate with the proxy application 110 to perform an action corresponding to the request. For example, the digital assistant application 108 can render a prompt for logging into the proxy application 110 on the display 148 of the client device 104. The prompt can include fields for the user's account identifier and password.

[0093] In addition, the action handler component 120 can also determine whether the proxy application 110 associated with the action list 140 is installed on the client device 104. The action handler component 120 can identify the proxy application 110 associated with the action list 140. Once identified, the action handler component 120 can access the client device 104 to determine whether the proxy application 110 has been installed on the client device 104. For example, the action handler component 120 can access a list of applications installed on the client device 104 to determine whether the proxy application 110 is installed. The action handler component 120 can check the application identifier or the proxy identifier against the list. In response to determining that the proxy application 110 is installed, the action handler component 120 can continue to process the request. On the other hand, in response to determining that the proxy application 110 is not installed, the action handler component 120 can terminate the processing of the request (e.g., using the address template of the action list 140 to generate an address). The digital assistant application 108 can also present an instruction to install the proxy application 110 to perform an action corresponding to the request. For example, the digital assistant application 108 may render a prompt indicating that the corresponding agent application 110 is not installed on the display 148 of the client device 104. The prompt may also include a link for installing the agent application 110.

[0094] The action list 140 selected by the action handler component 120 may correspond to a proxy application 110 that is not running as one of the applications in the foreground process 150 or the non-foreground process 152. For example, the action list 140 may correspond to a proxy application 110 that is installed on the client device 104 but is not running on the client device 104. The action list 140 selected by the action handler component 120 may correspond to a proxy application 110 that is running in a non-foreground process 152. For example, another application (e.g., another proxy application 110 or a digital assistant application 108) may be running in an application in the foreground process 150. The application in the foreground process 150 may have a graphical user interface rendered on at least a portion of the display 148 of the client device 104. In contrast, the application running in the non-foreground process 152 may not have a graphical user interface rendered on the display 148 of the client device 104, and may execute a routine (e.g., an action). In addition, the action list 140 selected by the action handler component 120 may correspond to a proxy application 110 that is running in the foreground process 150. For example, the agent application 110 corresponding to the action manifest 140 may have a graphical user interface currently rendered on the display 148 of the client device 104 .

[0095] The action handler component 120 can generate an address according to the address template of the action list 140. The action handler component 120 can identify the address template of the action list 140 selected from the set maintained in the data repository 128. As described above, the address template of the action list 140 may include: a first part that references the proxy application 110; and a second part that references the action to be performed and one or more input variables for performing the action. For example, the address template may be a URI template. The first part may include a scheme that references the proxy application 110 or a host name that references the proxy service 106 or the proxy application 110. The second part may include a path name that references the action and one or more query strings corresponding to one or more input variables in the URI. For example, for a request to check the number of steps taken, the address template of the action list 140 may be "https: / / ex_personalmetrics.example.com / checksteps{?username}". The first part may correspond to "https: / / ex_personalmetrics.example.com / ", and the second part may correspond to "checksteps{?username}".

[0096] The address generated according to the address template may include a first substring and a second substring. The first substring of the generated address may correspond to the first part of the address template. The generated address may be part of a URI, and the first substring may include a scheme referencing the proxy application 110 or a host name referencing the proxy service 106 or the proxy application 110. The scheme and host name may be adopted by the action handler component 120 from the first part of the address template. The second substring of the generated address may correspond to the second part of the second template. The generated address may also be part of a URI, and the second substring may include a path name referencing the action to be performed by the proxy application 110 and a query string including parameters identified from the input audio signal. For example, the address generated according to the address template may be "https: / / ex_personalmetrics.example.com / checksteps?username=example_user_id". The first substring of the address may include "https: / / ex_personalmetrics.example.com", and the second substring of the address may include "checksteps?username=example_user_id".

[0097] When generating an address, the action handler component 120 can use or set the first portion of the address template as a first substring of the address to reference the proxy service 106 or the proxy application 110 associated with the proxy service 106. The action handler component 120 can use the second portion of the address template as a second substring of the address to correspond to the action and the parameters for performing the action. The action handler component 120 can populate the input variables of the second portion of the address template with one or more parameters identified from the input audio signal.

[0098] The action handler component 120 can be filled according to the parameter mapping of the action list 140 defined for the address template. As discussed above, the parameter mapping can define the correspondence between the input variables of the second part and the words to be recognized from the parsed input audio signal. The action handler component 120 can use the parameters as the words recognized from the input audio signal to be inserted into the second part of the address template when filling the input variables. The filling of the second part of the address template can be based on the expression expansion in the URI template. For each input variable specified in the address template, the action handler component 120 can identify the corresponding parameters as specified by the mapping of the action list 140. Through this identification, the action handler component 120 can insert, replace or set the query string with the name of the input variable and the identified parameters. For example, in the address template, the input variable can be "{?username}", and the identified parameter can be "example_user_id". In this example, the action handler component 120 can replace "?username=example_user_id" in the second substring of the address according to the expression expansion of the URI template.

[0099] When filling the input variables of the address template, the action handler component 120 can determine whether the input variable is an auxiliary input variable as specified in the mapping of the address template of the action list 140. In response to determining that the input variable is an auxiliary input variable, the action handler component 120 can also determine whether there are identified parameters corresponding to the auxiliary input variable. When it is determined that there are no parameters corresponding to the auxiliary input variable, the action handler component 120 can omit or remove the auxiliary input variable from the address. Therefore, the generated address may not have one or more parameters corresponding to the auxiliary input variables in the address template. Otherwise, when it is determined that there are parameters corresponding to the auxiliary input variables, the action handler component 120 can insert or fill the auxiliary input variable with the parameter. Conversely, in response to determining that the input variable is not an auxiliary input variable, the action handler component 120 can identify the parameters to be included in the input variable from the identified parameters.

[0100] By generating an address according to an address template, the action handler component 120 can verify the address. The verification of the address by the action handler component 120 can be to check whether the address is in the correct format. For verification, the action handler component 120 can check the address against a regular expression 130 (e.g., a URI or URL) for the address. The regular expression 130 for the address can include, for example, a scheme for a valid address, a host name, a path name, and a query string separated by delimiters. The action handler component 120 can compare the address with the regular expression 130 to determine whether the address matches the pattern of the address specified by the regular expression 130. In response to the determination that the address matches, the action handler component 120 can determine that the generated address has been successfully verified.

[0101] In contrast, in response to determining that the address does not match, the action handler component 120 can determine that the generated address has not been successfully verified. Based on this determination, the action handler component 120 can also detect an error in verifying the generated address. The error can indicate that the address format is incorrect. In addition, the action handler component 120 can stop or terminate further processing of the address (e.g., generating an action data structure). The action handler component 120 can provide an indication of the error to the agent service 106 or an administrator of the agent application 110 associated with the action list 140.

[0102] The action handler component 120 can use the address for performing the action to package or generate an action data structure. The generation of the action data structure can be based on an application programming interface (API). For example, the API can be specified by the operating system running on the client device 104, and can be used by both the digital assistant application 108 and the proxy application 110. The action data structure can be transmitted between the digital assistant application 108 and the proxy agent 110 via the API. The generation of the action data structure can be based on the hypertext transfer protocol (HTTP). For example, the action data structure generated by the action handler component 120 can be an HTTP request (e.g., GET or POST function). The action data structure can include at least one header and at least one body. The header can include the device modality (e.g., smart phone, smart speaker, tablet computer, notebook computer and desktop) of the client device 104 that executes the digital assistant application 108 and the proxy application 110. The header can include a source address set to the client device 104. The header can include a generated address set to the destination address. The body of the action data structure can include other information related to the request identified from parsing the input audio signal. For example, the subject may include an identifier that references the digital assistant application 108 or the client device 104 that runs the digital assistant application 108.

[0103] An instance of the digital assistant application 108 on the data processing system 102 or the client device 104 can execute the agent interface component 126 to provide the action data structure to the agent application 110. By generating the action data structure, the agent interface component 126 can provide the action data to the agent application 110 or direct the action data to the agent application 110. The agent interface component 126 can send the action data structure to the agent service 106 (e.g., as an HTTP request) that manages the resources of the agent application 110 through the network 114. For example, the HTTP request corresponding to the action data structure can be sent by the agent interface component 126 through the network 114. The agent interface component 126 can use the action data structure (e.g., via an API) to call the agent application 110. For example, the agent interface component 126 can make a function call to facilitate calling the agent application 110 according to the API and by the operating system of the client device 104. When calling, the agent interface component 126 can pass the action data structure to the agent application 110. When calling, the agent interface component 126 can pass the action data structure to the agent application 110.

[0104] The data processing system 102, the client device 104, or the proxy service 106 can execute an instance of the proxy application 110. Since the destination address of the data structure can reference the proxy application 110 or the proxy service 106, the proxy application 110 can receive the action data structure from the proxy interface component 126. For example, the action data structure may have been sent via the network 114, and the destination address may include a host name that references the proxy application 110 or the proxy service 106. Since the destination address of the action data structure references the proxy application 110 or the proxy service 106, the action data structure can be routed to the proxy application 110 or the proxy service 106 via the network 114. In another example, the destination address of the action data structure can be a deep link to a solution that references the proxy application 110. By calling the destination address, the action data structure can be directed by the client device 104 or the network 114 to the proxy application 110 referenced by the solution.

[0105] When the action data structure is passed, the proxy application 110 can parse the action data structure to identify the address. The identification of the address can come from the destination address header of the action data structure. The proxy application 110 can parse the address to identify a first substring of the address, which first substring references the proxy application 110 itself or the proxy service 106 that manages the resources of the proxy application 110. The proxy application 110 can parse a second substring of the address. The proxy application 110 can further parse the second substring of the address to identify the action and one or more parameters for performing the action. The proxy application 110 can parse the path name of the URI to identify the action, and parse one or more query strings to identify one or more parameters for performing the action. For example, the proxy application 110 can parse the address https: / / ex_personalmetrics.example.com / checksteps? username=example_user_id to identify the second substring "checksteps?username=example_user_id". The proxy application 110 can further identify "checksteps" as a request to obtain the measured number of steps and identify "Example_user_id" as the name of the user whose steps were measured. Using this identification, the proxy application 110 can perform, conduct, or execute an action based on one or more parameters. Using the previous example, the proxy application 110 can execute a series of processes and routines to retrieve the number of steps taken by the user of the proxy application 110 with the account identifier "example_user_id". The proxy application 110 can provide the results of the execution of the action to the proxy interface component 126.

[0106] The proxy application 110 may also generate an output to indicate whether the execution of the action was successful or failed to send to the digital assistant application 108. When executing the agent, the proxy application 110 may fail to complete the action or terminate before completion. For example, when processing a request for the number of steps, the proxy service 106 may determine that the information of the number of steps taken by the corresponding user cannot be retrieved. Based on this determination, the proxy application 110 can create an output indicating failure. In response to the failure, the proxy application 110 can generate an output indicating failure. On the contrary, the proxy application 110 can complete the execution of the action. In response to the completion, the proxy application 110 can generate an output indicating success. The proxy application 110 can provide the output of the action execution to the proxy interface component 126.

[0107] In conjunction with the passing of the action data structure, the proxy interface component 126 can control whether the proxy application 110 runs as a non-foreground process 152 or as a foreground process 150. The setting or control of the foreground process 150 or the non-foreground process 152 can be based on the interface mode indicator of the action manifest 140 used to generate the action data structure. The setting of the foreground process 150 and the non-foreground process 152 by the proxy interface component 126 can be facilitated by the operating system running on the client device 104. As discussed above, the proxy application 110 can run as one of the applications in the foreground process 150 or the non-foreground process 152, or as one of the applications that are neither in the foreground process 150 nor in the non-foreground process 152. Through the passing of the action data structure, the proxy interface component 126 can control whether the proxy application 110 is set from the non-foreground process 152 to the foreground process 150.

[0108] In addition, the proxy interface component 126 can identify which applications are running in the foreground process 150 and running as non-foreground processes 152, such as the digital assistant application 108 and the proxy application 110 installed on the client device 104. For example, the proxy interface component 126 can access the operating system of the client device 104 to identify the applications running as the foreground process 150 and the applications running in the non-foreground process 152. The proxy interface component 126 can identify whether the proxy application 110 (or other applications on the client device 104) is one of the foreground process 150 or the non-foreground process 152. The proxy interface component 126 can identify the proxy application 110 as running as one of the applications in the foreground process 150. The proxy interface component 126 can identify the proxy application 110 as running as one of the applications in the non-foreground process 152. The proxy interface component 126 can identify the proxy application 110 as running as one of the applications that are neither in the foreground process 150 nor in the non-foreground process 152. In response to determining that the proxy application 110 is not executing as either a foreground process 150 or a non-foreground process 152 , the proxy interface component 126 can cause the client device 104 to initiate execution of the proxy application 110 .

[0109] The agent interface component 126 may identify an interface mode indicator from the selected action list 140. The interface mode indicator may specify that a user interface component 142 of the agent application 110 is to be rendered on the display 148 of the client device 104 when the action corresponding to the action list 140 is performed. When the interface mode indicator specifies that no user interface component 142 is to be rendered, the agent interface component 126 may set the agent application 110 to the foreground process 150. When the agent application 110 is identified as not being executed on the client device 104, the agent interface component 126 may launch the agent application 110 in the foreground process 150. When the agent application 110 is identified as running in a non-foreground process 152, the agent interface component 126 may remove the application (e.g., another agent application 110 or a digital assistance application 108) that was originally in the foreground process 150 and set it to a non-foreground process 152. In addition, the agent processor component 126 may set the agent application 110 from the non-foreground process 152 to the foreground process 150. When the agent application 110 is identified as already in the foreground process 150 , the agent interface component 126 may maintain the agent application 110 in the foreground process 150 .

[0110] When the interface mode indicator specifies that the user interface component 142 is to be rendered, the proxy interface component 126 can apply at least one authorization policy of the proxy application 110. The authorization policy can specify which applications the output of the proxy application 110 is restricted or allowed to be presented with. For example, the authorization policy of a personal fitness application can specify that the output is restricted from being presented with another personal fitness application. The authorization policy can include a list of applications (also referred to as a blacklist herein) to which the output of the proxy application 110 is restricted from being presented simultaneously with it. The authorization policy can also include a list of applications (also referred to as a whitelist herein) to which the output of the proxy application 110 is allowed to be presented simultaneously with it. Each list can include a proxy identifier (e.g., an application file name) corresponding to the included application. The authorization policy can be specific to the output type of the proxy application 110. The output type can depend on the request identified from parsing the input audio signal. For example, when another personal fitness application is in the foreground, the authorization policy can limit the display of the number of steps, but can allow the display of account credential information.

[0111] The determination of whether to apply the authorization policy can be based on whether the proxy application 110 is running in the foreground process 150 or in the non-foreground process 152. When the proxy application 110 is identified as being in the foreground process 150, the proxy interface component 126 can determine that the authorization policy is not applied. The proxy interface component 126 can also allow the user interface component 142 of the proxy application 110 to render the output in the display 148 of the client device 104. When another application other than the proxy application 110 is identified as running in the foreground process 150, the proxy interface component 126 can determine to apply the authorization policy. In this way, the proxy application 110 can be retained as part of the non-foreground process 152, or not executed on the client device 104. When determining whether to apply, the proxy interface component 126 can also identify the authorization policy based on the request parsed from the input audio signal. The proxy interface component 126 can compare the identified request with the output type of the authorization policy. When determining a match, the proxy interface component 126 can determine that the authorization policy specified for the corresponding output type is applied.

[0112] Based on the authorization policy, the proxy interface component 126 can determine whether the output of the proxy application 110 is authorized to be presented together with the graphical user interface component of the application in the foreground process 150. The proxy interface component 126 can identify the application in the foreground process 150. The proxy interface component 126 can identify the proxy identifier of the application in the foreground process 150. The proxy interface component 126 can compare the application with the application list specified in the authorization policy of the proxy application 110. When comparing, the proxy interface component 126 can use the identified proxy identifier to search the whitelist and blacklist of the authorization policy of the proxy application 110. If the application in the foreground process 150 matches one of the applications in the whitelist, the proxy interface component 126 can determine that the output of the proxy application 110 is allowed to be presented together with the application. On the other hand, if the application in the foreground process 150 matches one of the applications in the blacklist, the proxy interface component 126 can determine that the output is restricted from being presented together with the application.

[0113] When it is determined that the output of the proxy application 110 is allowed, the proxy interface component 126 can identify the user interface component 142 of the proxy application 110. In order to identify the user interface component 142, the proxy interface component 126 can use an address generated according to the address template of the action list 140. For example, the address generated according to the address template can reference the action performed by the proxy application 110 and the user interface component 142 to be rendered when the action is performed. The address generated according to the address template can reference the proxy application 110 or the proxy service 106 that manages the resources of the proxy application 110. The proxy interface component 126 can identify a separate address or identifier for the user interface component 142 from the action list 140. The address can reference the user interface component 142 maintained by the proxy service 106. The address and identifier can reference the user interface component 142 maintained on the data repository 128. Using a separate address or identifier, the proxy interface component 126 can identify the user interface component 142 that will be rendered together with the proxy application 110 performing the action.

[0114] After identification, the proxy interface component 126 can access or retrieve the user interface component 142. Using the address generated according to the address template of the action list 140, the proxy interface component 126 can retrieve the user interface component 142. The proxy interface component 126 can send a request for the user interface component 142 to the proxy service 106 including the address. Since the address references the proxy service 106, the request can be directed to the proxy service 106 that handles the resources of the proxy application 110. The proxy service 106 can then provide a response with the referenced user interface component 142 to the proxy interface component 126. Using the address included in the action list 140, the proxy interface component 126 can retrieve the user interface component 142. The address included in the action list 140 can be separate from the address generated using the address template and can reference the proxy service 106 or the data repository 128. The proxy interface component 126 can send a request for the user interface component 142 including the address. When the address references the proxy service 106, the request can be directed to the proxy service 106 that handles the resources of the proxy application 110. When the address references a data repository 128, the request can be directed to the data repository 128. The proxy service 106 or the data repository 128 can then provide a response with the referenced user interface component 142 to the proxy interface component 126. Using the identifier included in the action manifest 140, the proxy interface component 126 can retrieve the user interface component 142 from the data repository 128. The identifier can correspond to an index of user interface components 142 maintained on the data repository 128. The proxy interface component 126 can search the data repository 128 using the identifier to find the user interface component 142. Once identified, the proxy interface component 126 can retrieve the user interface component 142 from the data repository 128.

[0115] Prior to rendering the user interface component 142, the proxy interface component 126 may also apply the authorization policy of the application in the foreground process 150. The authorization policy may specify which applications the graphical user interface component of the proxy in the foreground process 150 is restricted or allowed to be presented with. For example, the authorization policy of a banking application may specify that the output is restricted from being presented with another banking application. The authorization policy may include a list of applications (also referred to herein as a blacklist) with which the graphical user interface of the application in the foreground process 150 is restricted from being presented simultaneously. The authorization policy may include a list of applications (also referred to herein as a whitelist) with which the graphical user interface of the application in the foreground process 150 is allowed to be presented simultaneously. Each list may include a proxy identifier (e.g., an application file name) corresponding to the included application.

[0116] Based on the authorization policy, the proxy interface component 126 can determine whether the graphical user interface component of the application in the foreground process 150 is authorized to be presented together with the output of the proxy application 110. The proxy interface component 126 can identify the proxy application 110 that performs the action. The proxy interface component 126 can identify the proxy identifier of the proxy application 110. The proxy interface component 126 can compare the application with the application list specified in the authorization policy of the application in the foreground process 150. When comparing, the proxy interface component 126 can use the identified proxy identifier to search the whitelist and blacklist of the authorization policy of the application in the foreground process 150. If the proxy application 110 matches one of the applications in the whitelist, the proxy interface component 126 can determine to allow simultaneous presentation with the graphical user interface of the application in the foreground 150. On the other hand, if the proxy application 110 matches one of the applications in the blacklist, the proxy interface component 126 can determine to restrict simultaneous presentation.

[0117] By identifying the user interface component 142, the proxy interface component 126 can render the user interface component 142 of the proxy application 110 in the display 148 of the client device 104. By allowing the determination of simultaneous presentation, the user interface component 142 of the proxy application 110 can be presented on the display 148 simultaneously with the graphical user interface of the application running in the foreground process 150. The proxy interface component 126 can render the user interface component 142 as an overlay superimposed on the graphical user interface of the application in the foreground process 150. The proxy interface component 126 can also include the user interface component 142 as part of the graphical user interface of the application in the foreground process 150. For example, the application running as the foreground process 150 can be a digital assistant application 108 and can have a graphical user interface that includes a dialog with a user operating the digital assistant application 108 on the client device 104. The proxy interface component 126 can insert the user interface component 142 into the dialog of the graphical user interface of the digital assistant application 108.

[0118] The agent interface component 126 can add or insert information into the user interface component 142 rendered on the display 148 of the client device 104. The agent interface component 126 can add, insert or include words or requests from the input audio signal into the corresponding interface element of the user interface component 142. The NLP component 116 can identify the words or requests from parsing the input audio signal. For example, the agent interface component 126 can merge the words with the request into the text box of the user interface component 142. The agent interface component 126 can retrieve, receive or identify the output generated by the agent application 110. The agent interface component 126 can retain the rendering of the user interface component 142 and can wait for the output from the agent application 110. After receiving the output from the agent application 110, the agent interface component 126 can add, insert or include the output into the corresponding interface element of the user interface component 142. For example, the agent interface component 126 can receive an output indicating the number of steps taken by the user of the agent application 110 installed on the client device 104. The proxy interface component 126 can insert the step number into another text box of the user interface component 142. With the output received and inserted, the proxy interface component 126 can render the user interface component 142 on the display 148.

[0119] When the user interface component 142 is rendered on the display 148 together with the graphical user interface of the application in the foreground process 150, the proxy interface component 126 can monitor the user interaction events with the user interface component 142. The interaction events may include, for example, hovering, clicking, scrolling, flicking, pressing keys, and touching the screen. After detecting the user interaction event on the user interface component 142, the proxy interface component 126 can process the user interaction event to update the user interface component 142. The proxy interface component 126 can relay the detected user interaction event to the proxy application 110. The proxy application 110 can process the user interaction event (e.g., using an event handler) and can generate another output in response to the user interaction event. The proxy application 110 can provide the output to the proxy interface component 126. Using the output received from the proxy application 110, the proxy interface component 126 can update the user interface component 142.

[0120] In this way, the proxy interface component 126 can maintain the application originally in the foreground process 150 while adding the user interface component 142 from the proxy application 110 to be displayed as an overlay. The proxy interface component 126 can have both the original application and the proxy application 110, both as applications in the foreground process 150. The proxy interface component 126 can also maintain the visual components from both applications rendered on the display 148 of the client device 104, thereby reducing the user of the client device 104 from performing context switching. By maintaining both at the same time, the proxy interface component 459 can also reduce the computing resources generated when switching rendering and between the foreground process 150 and the non-foreground process 152.

[0121] In contrast, when it is determined that the output of the proxy application 110 is limited, the proxy interface component 126 may set the proxy application 110 as a foreground process 150. When the proxy application 110 is identified as not running on the client device 104, the proxy interface component 126 may call and launch the proxy application 110 (e.g., via the operating system of the client device 104). The proxy interface component 126 may move the application that was originally running as the foreground process 150, and may set the application to run as a non-foreground process 152. When switching the foreground process 150 and the non-foreground process 152, the proxy interface component 126 may remove the graphical user interface component of the application that was originally in the foreground process 150 rendered on the display 148. In addition, the proxy interface component 126 may call the proxy application 110 using the action data structure generated by the action handler component 120. Instead of rendering the user interface component 142, the proxy interface component 126 may render the graphical user interface of the proxy application 110 on the display 148 of the client device 104.

[0122] Running as a foreground process 150, the proxy application 110 can display the output of the action indicated in the address of the action data structure on a graphical user interface rendered in the display 148. The proxy application 110 can monitor user interaction events with the graphical user interface. When a user interaction event is detected on the user interface component 142, the proxy application 110 can process the user interaction event (e.g., using an event handler), and can generate another output in response to the user interaction event. The proxy application 110 can use the generated output to update the graphical user interface rendered in the display 148 of the client device 104.

[0123] By controlling the switching between the foreground process 150 and the non-foreground process 152, the user's involvement in the search agent application 110 can be eliminated, thereby improving the HCI with the application on the client device 104. In addition, the digital assistant application 108 can adjust and reduce the consumption of computing resources on the client device 104 due to the switching.

[0124] Figure 2 Describes the use of Figure 1 The sequence diagram of an example data flow 200 for rendering visual components on an application in the system 100 shown in FIG. The data flow 200 may be composed of the above combination Figure 1 The system 100 described below may be combined with Figure 5 The detailed description is implemented or performed by system 500. Data flow 200 may include communications in the form of packets (eg, HTTP messages) between microphone 150, client device 104, data processing system 102, proxy service 106 (via network 114), and display 148.

[0125] The local instance of the digital assistant application 108 running on the client device 104 can receive audio data 205 (e.g., an input audio signal) from a microphone 146 that is communicatively coupled to the client device 104. The digital assistant application 108 can be executed on the client device 104 in a non-foreground process 152 without any visual components rendered on the display 148. The client device 104 can have another application in a foreground process 150 with a graphical user interface rendered on the display 148. The digital assistant application 108 on the client device 104 can perform initial processing (e.g., automatic speech recognition) on the audio data 205 to recognize one or more words from the audio data 205. The digital assistant application 108 can provide the one or more recognized words from the audio data 205 as input 210 to the remote instance of the digital assistant application 108 running on the data processing system 102. The NLP component 116 on the data processing system 102 can then further process the one or more words of the input 210 to recognize a request and one or more parameters. The request can indicate an action to be performed by the agent application 110 according to the one or more parameters.

[0126] The action handler component 120 on the data processing system 102 can use the request and one or more parameters to identify an action list 140 from the data repository 128. The proxy application 110 associated with the identified action list 140 may not be executed on the client device 104, or may be running as one of the applications of the non-foreground process 152. The action handler component 120 can generate an address based on the address template of the identified action list 140. The address may include a first substring (e.g., a scheme or host name) that references the proxy application to perform the action. The address may also include a second string corresponding to the action to be performed and one or more parameters based on which the action is to be performed. The action handler component 120 can package the address into an action data structure to provide it as input 215 to the proxy service 106 of the proxy application 110.

[0127] After receiving input 215, the proxy service 106 of the proxy application 110 can parse the action data structure to identify the first substring and the second substring. When processing the input, the client device 104 can maintain the application as a foreground process 150 and a non-foreground process 152 at runtime. The proxy application 110 executed on the proxy service 106 can further parse the second substring to identify the action and one or more parameters. Through this identification, the proxy application 110 can perform actions according to the parameters. When performing the action, the proxy application 110 can generate an output 220 to provide to the client device 104 via the data processing system 102. The proxy service 106 can provide the output 220 to the data processing system 102. The digital assistant application 108 on the data processing system 102 can send the output 220 as an output 225 to the client device 104.

[0128] In conjunction, the proxy interface component 126 can determine that the action manifest 140 specifies that the user interface component 142 is to be rendered when the action is performed. In response to the determination, the proxy interface component 126 can identify the user interface component 235 from the set of user interface components 142 (e.g., using an address generated using an address template of the action manifest 140). The proxy interface component 126 can provide the user interface component 235 to the client device 104. Using the user interface component 235, the client device 104 can display the rendering data 240 for the display 148 of the client device 104. The rendering data 240 can include the user interface component 235 as an overlay on the graphical user interface of the application in the foreground process 150. The client device 104 can maintain the application in the foreground process 150 and the non-foreground process 152 before receiving the audio data 205 from the microphone 146.

[0129] Figure 3 Shown in Figure 1A use case diagram of an example client device 104 in which an input message is displayed and a user interface component is provided in the illustrated system 100. The client device 104 may have a local instance of an application running as a foreground process 150 and a digital assistant application 108 running as a non-foreground process 152. The application running as a foreground process 150 may have a graphical user interface 305 rendered on a display 148 of the client device 104. As shown, the graphical user interface 305 may be used for a web browser application running as a foreground process 150. In this case, the NLP component 116 of the digital assistant application 108 running on the client device 104 may receive an input audio signal via the microphone 146 of the client device 104. The NLP component 116 may apply automatic speech recognition to identify one or more words 310 from the input audio signal. For example, the NLP component 116 may identify the word 310 "How many steps have I taken today? (I walked a few steps today?)" from the input audio signal.

[0130] The NLP component 116 on the client device 104 or the data processing system 102 can perform further processing on the words 310. Through further processing, the NLP component 116 can identify a request from the words 310 recognized from the input audio signal. In the depicted example, the NLP component 116 can identify the request as a request to obtain the number of steps taken by a user operating the client device 104 or associated with the client device 104. Using the identified request, the action handler component 120 can identify the proxy application 110 that can execute the request. The action handler component 120 can also select an action list 140 from the data repository 128. The action list 140 can be for an action corresponding to the request to be executed by the proxy application 110. Using the action list 140, the action handler component 120 can call the proxy application 110 to execute the action corresponding to the request.

[0131] In conjunction, the proxy interface component 126 can identify that a user interface component 142 is to be displayed when an action in the action list 140 is performed. The action list 140 can specify whether the user interface component 142 is to be displayed, and can identify which user interface component 142 is to be presented. In response to the identification, the proxy interface component 126 can determine whether to allow simultaneous presentation with the graphical user interface component 305 of the foreground application. The determination can be based on the authorization policy of the proxy application 110. The authorization policy of the proxy application 110 can specify which applications are allowed to be presented simultaneously with the user interface component 142 of the proxy application 110. In the depicted example, the proxy interface component 126 can determine that simultaneous presentation is allowed.

[0132] In response to the determination, the agent interface component 126 may identify a user interface component 142 from the action list 140 to perform the action indicated by the term 310. Upon identification, the agent interface component 126 may render the user interface component 142 as an overlay component 315 superimposed on top of the graphical user interface 305 of the foreground process 150. In the depicted example, the user interface component 142 rendered as the overlay component 315 may include the term 310 of the request corresponding to the action, "how many steps?". Additionally, the user interface component 142 may also have the result output from performing the action, "You have taken 3147 steps today." The user interface component 142 rendered as the overlay component 315 may also include a command button 320. Upon interaction with the command button 320, the agent interface component 125 may close the overlay component 315, thereby causing the user interface component 142 to disappear from rendering on the display 148 of the client device 104.

[0133] Figure 4 A block diagram of an example method 400 for rendering a visual component on an application is shown. The method 400 may be combined with the above Figures 1 to 3 The system 100 described below is combined with Figure 5 The system 500 described in detail is implemented or executed. In short, the NLP component 116 can receive an input audio signal (405). The NLP component 116 can parse the input audio signal (410). The action handler component 120 can select an action list 140 (415). The action handler component 120 can use the action list 140 to generate an address (420). The action handler component 120 can generate an action data structure (425). The proxy interface component 126 can guide the action data structure (430). The proxy interface component 126 can determine whether the user interface component 142 is authorized to be presented with the foreground application (435). If it is determined to be authorized, the proxy interface component 126 can present the user interface component 142 (440). If it is determined to be unauthorized, the proxy interface component 126 can call the proxy application 110 associated with the user interface component (445).

[0134] In more detail, the NLP component 116 can receive an input audio signal (405). The NLP component 116 can receive one or more data packets, including an input audio signal acquired at a sensor (e.g., microphone 146) on the client device 104. For example, the NLP component 116 of the digital assistant application 108 executed at least in part by the data processing system 102 can receive the input audio signal. The input audio signal may include a conversation facilitated by the digital assistant application 108. The conversation may include one or more inputs and outputs. The conversation may be audio-based, text-based, or a combination of audio and text. The input audio signal may include text input, or other types of input that may provide conversation information. When the input audio signal is received, the client device 104 may cause the application to run as a foreground process 150 and a non-foreground process 152.

[0135] The NLP component 116 can parse the input audio signal (410). By parsing the input audio signal, the NLP component 116 can use various natural language processing techniques to identify a request and one or more parameters. The request can be an intent or request that the digital assistant application 108 or the agent application 110 can satisfy. The parameters can define the request. Based on the request, the NLP component 116 can determine that the request corresponds to one of the functions of the agent application 110.

[0136] The action handler component 120 may select an action list 140 (415). Based on the request identified from the input audio signal, the action handler component 120 may select an action list 140 from a collection maintained on the data repository 128. For each action list 140, the action handler component 120 may compare the request with the request identifier of the action list 140. The comparison may be performed using a semantic knowledge graph. Through the comparison, the action handler component 120 may determine that the request matches the request identifier of the action list 140. In response to the determination, the action handler component 120 may select an action list 140 from a collection maintained on the data repository 128. The selected action list may indicate that a user interface component 142 of the proxy application 110 that is not running as a foreground process 150 is to be presented.

[0137] The action handler component 120 may use the action list 140 to generate an address (420). The address may be generated according to the address template of the action list 140. The address template may have a first part and a second part. The first part may reference the proxy application 110 or the proxy service 106 for the proxy application 110 to perform the action. The second part may reference the action and may include one or more input variables of the action. The action handler component 120 may obtain the first part and the second part from the address template when generating the address. The action handler component 120 may insert or fill parameters into the input variables of the second part of the address template.

[0138] The action handler component 120 may generate an action data structure (425). The action data structure may include a header and a body. The action handler component 120 may insert an address generated according to an address template of the action manifest 140 into the header of the action data structure. The action handler component 120 may set the address as the destination address of the header to route the action data structure to the proxy application 110 or the proxy service 106 for the proxy application 110.

[0139] The proxy interface component 126 may direct the action data structure (430). The proxy interface component 126 may direct the action data structure to the proxy application 110. The proxy interface component 126 may use the action data structure to call the proxy application 110, and may pass the action data structure to the proxy application 110. The proxy application 110 may then parse the action data structure to identify the action and one or more parameters from the address included in the action data structure. The proxy application 110 may perform the action according to the parameters.

[0140] The proxy interface component 126 may determine whether the user interface component 142 is authorized to be presented with the foreground application (435). The determination of whether the user interface component 142 is authorized may be based on an authorization policy. The authorization policy may specify whether the output of the proxy application 110 is allowed or prohibited to be presented simultaneously with the graphical user interface of the application running as the foreground process 150. The proxy interface component 126 may identify the application running in the foreground process 150. The proxy interface component 126 may compare the application with the specifications of the authorization policy. When the authorization policy allows simultaneous presentation, the proxy interface component 126 may determine that the user interface component 142 is authorized to be presented. Otherwise, when the authorization policy restricts simultaneous presentation, the proxy interface component 126 may determine that the user interface component 142 is not authorized to be presented with the application running as the foreground process 150.

[0141] If determined to be authorized, the proxy interface component 126 may present the user interface component 142 (440). The proxy interface component 126 may identify the user interface component 142 from the selected action list 140 (e.g., using the address included therein). The proxy interface component 126 may render the identified user interface component 142 as an overlay on the graphical user interface of the application running in the foreground process 150. The proxy interface component 126 may include the request for the action and the output generated by the proxy application 110 in the user interface component 142.

[0142] On the other hand, if it is determined that it is not authorized, the proxy interface component 126 can call the proxy application 110 associated with the user interface component 142 (445). The proxy interface component 126 can remove the original application running from the foreground process 150 and set the application as a non-foreground process 152, thereby removing the rendering of the graphical user interface of the application. The proxy interface component 126 can set the proxy application 110 as the foreground process 150. The proxy interface component 126 can also render the graphical user interface of the proxy application 110 on the display 148 of the client device.

[0143] Figure 5 5 is a block diagram of an example computer system 500. The computer system or computing device 500 may include or be used to implement the system 100 or its components (such as the data processing system 102). The computing system 500 includes a bus 505 or other communication component for transmitting information and a processor 510 or processing circuit coupled to the bus 505 to process information. The computing system 500 may also include one or more processors 510 or processing circuits coupled to the bus to process information. The computing system 500 also includes a main memory 515, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 505 for storing information and instructions to be executed by the processor 510. The main memory 515 may be or include the data repository 128. The main memory 515 may also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 510. The computing system 500 may further include a read-only memory (ROM) 520 or other static storage device coupled to the bus 505 for storing static information and instructions for the processor 510. A storage device 525 , such as a solid-state device, magnetic disk, or optical disk, may be coupled to bus 505 for persistently storing information and instructions. Storage device 525 may include or be a part of data repository 128 .

[0144] Computing system 500 may be coupled to a display 535 (such as a liquid crystal display or an active matrix display) via bus 505 for displaying information to a user. An input device 530, such as a keyboard including alphanumeric and other keys, may be coupled to bus 505 for communicating information and command selections to processor 510. Input device 530 may include a touch screen display 535. Input device 530 may also include a cursor control, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to processor 510 and controlling cursor movement on display 535. Display 535 may be, for example, a computer program product of data processing system 102, client device 104, or the like. Figure 1 part of other components in .

[0145] The processes, systems, and methods described herein can be implemented by the computing system 500 in response to the processor 510 executing an arrangement of instructions contained in the main memory 515. Such instructions can be read into the main memory 515 from another computer-readable medium, such as the storage device 525. Executing the arrangement of instructions contained in the main memory 515 causes the computing system 500 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement can also be employed to execute the instructions contained in the main memory 515. With the systems and methods described herein, hard-wired circuitry can be used in place of or in combination with software instructions. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0146] Although already Figure 5 An example computing system is described in the specification, but the present subject matter (including the operations described in the specification) may be implemented by using other types of digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed in the specification and their structural equivalents), or in a combination of one or more of them.

[0147] In the case where the systems discussed herein collect personal information about a user or can utilize personal information, the user may be provided with the following opportunities: control whether a program or feature collects user information (e.g., information about the user's social network, social actions, or activities; the user's preferences; or the user's location) or control whether or how to receive content that may be more relevant to the user from a content server or other data processing system. In addition, before storing or using specific data, the specific data may be anonymized in one or more ways so that personal identification information can be removed when generating parameters. For example, the user's identity may be anonymized so that the user's personal identification information cannot be determined, or the user's geographic location from which location information (such as a city, zip code, or state and county level) may be generalized so that the user's specific location cannot be determined. Thus, the user may control the manner in which the content server collects and / or uses information about the user.

[0148] The subject matter and operations described in this specification may be implemented in a digital electronic circuit system, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more circuits of computer program instructions encoded on one or more computer storage media for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or included therein. When the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer may also be one or more separate components or media (e.g., multiple CDs, disks, or other storage devices), or included therein. The operations described in this specification can be implemented as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.

[0149] The terms "data processing system," "computing device," "component," or "data processing device" encompass all kinds of devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or multiple or combinations of the foregoing. The device may include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The device and execution environment may implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures. The components of system 100 may include or share one or more data processing devices, systems, computing devices, or processors.

[0150] Computer programs (also referred to as programs, software, software applications, applications, scripts, or codes) can be written in any form of programming language, including compiled or interpreted languages, declarative languages, or procedural languages, and can be deployed in any form, including deployment as independent programs or deployment as modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs can correspond to files in a file system. Computer programs can be stored in a part of a file that keeps other programs or data (e.g., one or more scripts stored in a markup language document), or in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or partial codes). Computer programs can be deployed to execute on one computer or on multiple computers located at one site or distributed in multiple sites and interconnected by a communication network.

[0151] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs (e.g., components of the data processing system 102) to perform actions by operating input data and generating output. The processes and logic flows may also be performed by special-purpose logic circuits (e.g., FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits)), and the devices may also be implemented as special-purpose logic circuits (e.g., FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits)). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROMs, EEPROMs, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROM disks, and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a special-purpose logic circuit system.

[0152] The subject matter described herein can be implemented in a computing system that includes a back-end component (e.g., as a data processor), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or a combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: local area networks ("LANs") and wide area networks ("WANs"), internets (e.g., the Internet), and point-to-point networks (e.g., ad hoc point-to-point networks).

[0153] A computing system such as system 100 or system 500 may include a client and a server. The client and server are generally remote from each other and usually interact through a communication network (e.g., network 114). The relationship between the client and the server is generated by computer programs that run on corresponding computers and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., data packets representing content items) to the client device (e.g., in order to display data to a user interacting with the client device and receive user input from the user). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server (e.g., received by the data processing system 102 from the client device 104 or the proxy service 106).

[0154] Although operations are depicted in the drawings in a particular order, such operations need not be performed in the particular order shown or in sequential order, and not all illustrated operations need be performed. The actions described herein may be performed in a different order.

[0155] The separation of the various system components is not required in all embodiments, and the described program components may be included in a single hardware or software product. For example, the NLP component 116, the audio signal generator component 118, the action processor component 120, the response selector component 122, the agent registry component 124, and the agent interface component 126 may be a single component, application or program, or a logical device having one or more processing circuits, or a part of one or more servers of the data processing system 102.

[0156] Now, having described some illustrative embodiments, it is apparent that the foregoing description is illustrative rather than restrictive, and is presented by way of example. Specifically, although many of the examples presented herein relate to specific combinations of method actions or system elements, these actions and these elements may be combined in other ways to achieve the same purpose. The actions, elements, and features discussed in conjunction with one embodiment are not intended to be excluded from similar effects in other embodiments.

[0157] The words and phrases used herein are for descriptive purposes and should not be construed as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein is intended to encompass the items listed thereafter, their equivalents, and other items, as well as alternative embodiments consisting of the items listed thereafter exclusively. In one embodiment, the systems and methods described herein consist of one, or each combination of more, or all of the described elements, actions, or components.

[0158] Any reference to the embodiments, elements or actions of the systems and methods mentioned herein in the singular may also encompass embodiments that include a plurality of these elements, and any reference to any embodiments, elements or actions in the plural may also encompass embodiments that include only one element. Reference in the singular or plural is not intended to limit the currently disclosed systems or methods, their components, actions, or elements to single or multiple configurations. Reference to any action or element based on any information, action or element may include embodiments that the action or element is based at least in part on any information, action or element.

[0159] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment," "certain embodiments," "an embodiment," etc. are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment or example. As used herein, such terms do not necessarily all refer to the same embodiment. Any embodiment may be combined, inclusively or exclusively, with any other embodiment in any manner consistent with the aspects and embodiments disclosed herein.

[0160] References to "or" may be interpreted as inclusive, such that any term described using "or" may indicate any of a single, more than one, and all of the described terms. References to "at least one of 'A' and 'B'" may include only 'A', only 'B', and both 'A' and 'B'. Such references used in conjunction with "including" or other open-ended terms may include additional items.

[0161] Where technical features in the drawings, detailed description or any claims are followed by reference numerals, these reference numerals have been included to increase the intelligibility of the drawings, detailed description and claims. Therefore, reference numerals or their absence have no limiting effect on the scope of any claim element.

[0162] The systems and methods described herein may be embodied in other specific forms without departing from their characteristics. The foregoing embodiments are illustrative and do not limit the systems and methods described. Therefore, the scope of the systems and methods described herein is indicated by the appended claims rather than the foregoing description, and changes within the equivalent meaning and scope of the claims are included therein.

Claims

1. A system for rendering a visual component on an application, comprising: A digital assistant application running as a background process that interfaces with or includes: a proxy registry executed on a data processing system having one or more processors, the proxy registry being configured to maintain a plurality of action lists for a plurality of applications, at least one of the plurality of action lists indicating a user interface component of one of the plurality of applications to be rendered when an action is performed, at least one of the plurality of action lists having an address template for accessing the user interface component when the action is performed, the address template defining a first portion and a second portion, the first portion corresponding to one of the plurality of applications, the second portion comprising input variables when the action is performed; A natural language processor executed on the data processing system, configured to: receiving a data packet including an input audio signal detected by a sensor of a client device that displays a graphical user interface of a first application in a foreground process on the client device; as well as parsing the input audio signal of the data packets to identify a request and parameters defining the request to be executed by a second application; An action processor executed on the data processing system, for: selecting, from the plurality of action lists, an action list for executing an action corresponding to the request by the second application among the plurality of applications, the second application being installed on the client device and not in the foreground process; generating an address for executing the action in the action list according to the address template of the action list, the address comprising a first substring and a second substring, the first substring corresponding to the second application, the second substring having a parameter for controlling execution of the action; and generating an action data structure according to the action list; and An agent interface executed on the data processing system, configured to: providing the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output; determining, based on an authorization policy of the second application, that output from the second application performing the action is authorized to be presented with a graphical user interface of the first application; in response to determining that the output of the second application is authorized to be presented with the first application, identifying a user interface component of the second application for the action list selected from the plurality of action lists; as well as The user interface component including the output from the second application is displayed on the client device together with a graphical user interface of the first application that is authorized to be presented with the second application, wherein the user interface component including the output from the second application is displayed as an overlay on the first application or as a user interface component of the first application.

2. The system of claim 1, wherein: The action handler is configured to generate the action data structure including the address generated by the action handler according to the address template.

3. The system of claim 1, wherein: The proxy interface is configured to identify the user interface component of the second application for the action manifest selected from the plurality of action manifests using the address generated according to the address template of the action manifest.

4. The system of claim 1, wherein: The proxy registry is configured to maintain the plurality of action manifests for the plurality of applications, each action manifest having a proxy identifier corresponding to one of the plurality of applications and a request identifier corresponding to the action; the natural language processor being configured to parse the input audio signal of the data packet to identify an agent identifier corresponding to an agent, the agent corresponding to the second application installed on the client device; as well as The action handler is configured to select the action list from the plurality of action lists for performing the action, the action list having an agent identifier corresponding to the agent identifier identified from parsing the input audio signal.

5. The system of claim 1, wherein the action handler is configured to: determining that an interface mode indicator of the action manifest specifies rendering of the user interface component when performing the action; and In response to determining that the interface mode indicator specifies rendering of the user interface component, the user interface component of the second application is identified for the action manifest.

6. The system of claim 1, wherein the agent interface is configured to: determining, for the action list for performing the action corresponding to the second request, that the client device is not authenticated by the second application; and In response to determining that the client device is not authenticated by the second application, presenting a prompt interface on the client device to authenticate the client device to the second application to perform the action corresponding to the second request.

7. The system of claim 1, wherein the proxy interface is configured to provide the action data structure to the second application so that the second application: monitoring interaction events on the user interface component from the second application presented with the graphical user interface of the first application; and In response to detecting the interaction event, the interaction event is processed to update the user interface component.

8. The system of claim 1, wherein: The natural language processor is configured to receive the data packet via a digital assistant application, the data packet including the input audio signal detected by the sensor of the client device, the client device displaying the graphical user interface of the digital assistant application corresponding to the first application in the foreground process; as well as The agent interface is configured to display within the graphical user interface of the digital assistant application a user interface component including a subcomponent indicating the request identified from parsing the input audio signal and a user interface component including the output from the agent corresponding to the second application.

9. The system of claim 1, wherein: The natural language processor is configured to receive the data packet, the data packet including the input audio signal detected by the sensor of the client device, the client device displaying a graphical user interface of a first agent corresponding to the first application in the foreground process; as well as The agent interface is configured to display a user interface component including the output from the second application as an overlay on a graphical user interface of the first agent.

10. The system of claim 1 , wherein the proxy registry is configured to maintain the plurality of action lists for the plurality of applications, the plurality of action lists having a first subset of action lists and a second subset of action lists, the first subset comprising action lists defined by an administrator of the data processing system, the second subset comprising an action list provided by a proxy service that processes one of the plurality of applications.

11. The system according to any one of claims 1 to 10, wherein the action handler is configured to provide the action data structure to the second application so that the second application: parsing the action data structure to identify an action to be performed; generating an output by performing the action identified from the action data structure; as well as The output is provided to the user interface component to be presented with a graphical user interface of the first application that is authorized to be presented with the second application.

12. A system for rendering a visual component on an application, comprising: A digital assistant application running as a background process that interfaces with or includes: a proxy registry executed on a data processing system having one or more processors, the proxy registry being configured to maintain a plurality of action manifests for a plurality of applications, at least one action manifest of the plurality of action manifests indicating rendering of a user interface component of one of the plurality of applications when an action is performed; A natural language processor executed on the data processing system, configured to: receiving a data packet including an input audio signal detected by a sensor of a client device that displays a graphical user interface of a first application in a foreground process on the client device; as well as parsing the input audio signal of the data packets to identify a request; An action processor executed on the data processing system, for: selecting, from the plurality of action lists, an action list for executing an action corresponding to the request by a second application in the plurality of applications, the second application being installed on the client device and not in the foreground process; as well as generating an action data structure according to the action list; as well as An agent interface executed on the data processing system, configured to: providing the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output; determining, based on an authorization policy of the second application, that output from the second application performing the action is authorized to be presented with a graphical user interface of the first application; in response to determining that the output of the second application is authorized to be presented with the first application, identifying a user interface component of the second application for the action list selected from the plurality of action lists; displaying the user interface component including the output from the second application on the client device together with a graphical user interface of the first application that is authorized to be presented with the second application, wherein the user interface component including the output from the second application is displayed as an overlay to the first application or as a user interface component of the first application; determining, based on an authorization policy of a third application, that output of the third application is not authorized to be presented with the first application, the third application being installed on the client device but not in the foreground process; Setting the third application as the foreground process of the client device, and changing the first application from the foreground process to a non-foreground process running on the client device; as well as A graphical user interface of the third application is displayed on the client device, the graphical user interface including output generated by the third application.

13. A system for rendering a visual component on an application, comprising: A digital assistant application running as a background process that interfaces with or includes: a proxy registry executed on a data processing system having one or more processors, the proxy registry being configured to maintain a plurality of action manifests for a plurality of applications, at least one action manifest of the plurality of action manifests indicating rendering of a user interface component of one of the plurality of applications when an action is performed; A natural language processor executed on the data processing system, configured to: receiving a data packet including an input audio signal detected by a sensor of a client device that displays a graphical user interface of a first application in a foreground process on the client device; as well as parsing the input audio signal of the data packets to identify a request; An action processor executed on the data processing system, for: selecting, from the plurality of action lists, an action list for executing an action corresponding to the request by a second application in the plurality of applications, the second application being installed on the client device and not in the foreground process; as well as generating an action data structure according to the action list; as well as An agent interface executed on the data processing system, configured to: providing the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output; determining, based on an authorization policy of the second application, that output from the second application performing the action is authorized to be presented with a graphical user interface of the first application; in response to determining that the output of the second application is authorized to be presented with the first application, identifying a user interface component of the second application for the action list selected from the plurality of action lists; determining, based on an authorization policy of the first application, that the first application authorizes presentation of the user interface component of the second application with the graphical user interface of the first application, the authorization policy allowing user interface components from a first subset of the plurality of applications to be presented with the graphical user interface and restricting user interface components from a second subset of the plurality of applications from being presented with the graphical user interface; as well as In response to determining that the first application is authorized to present the user interface component of the second application together with the graphical user interface of the first application, the user interface component including the output from the second application is displayed on the client device together with the graphical user interface of the first application that is authorized to be presented with the second application, wherein the user interface component including the output from the second application is displayed as an overlay on the first application or as a user interface component of the first application.

14. A method for rendering a visual component on an application, the method being performed by a data processing system having one or more processors and including a digital assistant application running as a background process, the digital assistant application interfacing with or including a proxy registry, a natural language processor, an action handler, and a proxy interface, the method comprising: maintaining a plurality of action lists of a plurality of applications on the proxy registry executed on the data processing system, at least one action list of the plurality of action lists indicating rendering of a user interface component of one of the plurality of applications when an action is executed, at least one action list of the plurality of action lists having an address template for accessing the user interface component when the action is executed, the address template defining a first part and a second part, the first part corresponding to one of the plurality of applications, the second part including input variables when the action is executed; The natural language processor is executed on the data processing system to: receiving a data packet including an input audio signal detected by a sensor of a client device that displays a graphical user interface of a first application in a foreground process on the client device; parsing the input audio signal of the data packets to identify a request and parameters defining the request to be executed by a second application; The action processor is executed on the data processing system for: selecting, from the plurality of action lists, an action list for executing the action corresponding to the request by the second application among the plurality of applications, the second application being installed on the client device and not in the foreground process; generating an address for executing the action in the action list according to the address template of the action list, the address comprising a first substring and a second substring, the first substring corresponding to the second application, and the second substring having a parameter for controlling execution of the action; generating an action data structure according to the action list; The agent interface is executed on the data processing system for: providing the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output; determining, based on an authorization policy of the second application, that output from the second application performing the action is authorized to be presented with a graphical user interface of the first application; in response to determining that the output of the second application is authorized to be presented with the first application, identifying a user interface component of the second application for the action list selected from the plurality of action lists; as well as The user interface component including the output from the second application is displayed on the client device together with a graphical user interface of the first application that is authorized to be presented with the second application, wherein the user interface component including the output from the second application is displayed as an overlay on the first application or as a user interface component of the first application.

15. The method according to claim 14, comprising: The action data structure including the address generated by the data processing system according to the address template is generated by the data processing system.

16. The method according to claim 14, comprising: The user interface component of the second application is identified, by the data processing system, for the action manifest selected from the plurality of action manifests using the address generated according to the address template of the action manifest.

17. The method according to claim 14, comprising: determining, by the data processing system, for the action list for performing the action corresponding to the second request, that the client device is not authenticated by the second application; as well as In response to determining that the client device is not authenticated by the second application, the data processing system presents a prompt interface on the client device to authenticate the client device to the second application to perform the action corresponding to the second request.

18. The method according to claim 14, comprising: receiving, by the data processing system, the data packet via a digital assistant application, the data packet including the input audio signal detected by the sensor of the client device, the client device displaying the graphical user interface of the digital assistant application corresponding to the first application in the foreground process; as well as A user interface component including a subcomponent indicating the request identified from parsing the input audio signal and a user interface component including the output from the agent corresponding to the second application is displayed within the graphical user interface of the digital assistant application via the data processing system.

19. The method according to any one of claims 14 to 18, comprising: providing, by the data processing system, the action data structure to the second application so that the second application: monitoring interaction events on the user interface component from the second application presented with the graphical user interface of the first application; as well as In response to detecting the interaction event, the interaction event is processed to update the user interface component.

20. A method for rendering a visual component on an application, the method being performed by a data processing system having one or more processors and including a digital assistant application running as a background process, the digital assistant application being interfaced with or including a proxy registry, a natural language processor, an action handler, and a proxy interface, the method comprising: maintaining a plurality of action manifests for a plurality of applications on the proxy registry executed on the data processing system, at least one action manifest of the plurality of action manifests indicating rendering of a user interface component of one of the plurality of applications when an action is performed; The natural language processor is executed on the data processing system for: receiving a data packet including an input audio signal detected by a sensor of a client device that displays a graphical user interface of a first application in a foreground process on the client device; parsing the input audio signal of the data packets to identify a request; The action processor is executed on the data processing system for: selecting, from the plurality of action lists, an action list for executing the action corresponding to the request by a second application in the plurality of applications, the second application being installed on the client device and not in the foreground process; generating an action data structure according to the action list; The agent interface is executed on the data processing system for: providing the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output; determining, based on an authorization policy of the second application, that output from the second application performing the action is authorized to be presented with a graphical user interface of the first application; in response to determining that the output of the second application is authorized to be presented with the first application, identifying a user interface component of the second application for the action list selected from the plurality of action lists; as well as displaying the user interface component including the output from the second application on the client device together with a graphical user interface of the first application that is authorized to be presented with the second application, wherein the user interface component including the output from the second application is displayed as an overlay to the first application or as a user interface component of the first application; determining, based on an authorization policy of a third application, that output of the third application is not authorized to be presented with the first application, the third application being installed on the client device but not in the foreground process; as well as In response to determining that the output of the third application is not authorized to be presented with the first application, providing a second action data structure to the third application to cause the third application to: As the foreground process of the client device, the first application is started, and the first application is changed from the foreground process to a non-foreground process running on the client device; parsing the second action data structure to perform a second action according to the second action data structure; as well as A graphical user interface of the third application is displayed on the client device, the graphical user interface including output generated from performing the second action.

Images