Rendering visual components on applications in response to voice commands

Abstract

The invention discloses rendering a visual component on an application in response to a voice command. The present disclosure generally relates to systems and methods for rendering visual components on an application. The at least one action manifest may indicate rendering of the user interface component. A data processing system may identify a request from an input audio signal from a client device. The client device may display a first application in a foreground process. The data processing system may select an action manifest to perform an action corresponding to the request by a second application. The data processing system may generate an action data structure using the action manifest to provide to the second application. The data processing system may determine that output of the second application is authorized to be presented with the first application. The data processing system may display a user interface component from the second application with the first application on the client device.

Description

Case Analysis

[0001] This application is a divisional application of Chinese invention patent application 201980002177.6, filed on May 6, 2019. Technical Field

[0002] This disclosure relates to rendering visual components on an application in response to voice commands. Background Technology

[0003] Applications can be installed on computing devices. Computing devices can run applications. Applications can present digital content. Summary of the Invention

[0004] According to a first aspect of this disclosure, a system for rendering visual components on an application is disclosed. The system may include a proxy registry executing on a data processing system having one or more processors. The proxy registry may maintain multiple action lists for multiple applications. At least one of the multiple action lists may instruct the rendering of a user interface component of one of the multiple applications when an action is performed. The system may include a natural language processor executing on the data processing system. The natural language processor may receive data packets including input audio signals detected by sensors of a client device. The client device may display a graphical user interface of a first application in a foreground process on the client device. The natural language processor may parse the input audio signals of the data packets to identify a request. The system may include an action handler executing on the data processing system. The action handler may select from multiple action lists an action list to execute an action corresponding to a request through a second application among the multiple applications. The second application may be installed on the client device but not in a foreground process. The action handler may generate an action data structure based on the action list. The system may include an action handler executing on the data processing system. A proxy interface may provide the action data structure to the second application so that the second application parses the action data structure and performs the action to generate output. The proxy interface can determine, based on the authorization policy of the second application, that the output from the second application performing the action is authorized to be presented along with the graphical user interface of the first application. In response to determining that the output of the second application is authorized to be presented along with the first application, the proxy interface can identify the user interface components of the second application for an action list selected from multiple action lists. The proxy interface can then display the user interface components, including the output from the second application, along with the graphical user interface of the first application authorized to be presented along with the second application on the client device.

[0005] At least one of a plurality of action lists may have an address template for accessing user interface components when an action is performed. This address template defines a first part and a second part, the first part corresponding to one of a plurality of applications, and the second part including input variables when the action is performed. A natural language processor may parse the input audio signal of the data packets to identify parameters defining the request to be performed by a second application. An action processor may generate an address for performing an action of the action list, based on the address template of the action list. This address includes a first substring and a second substring, the first substring corresponding to the second application, and the second substring having parameters for controlling the execution of the action; and generate an action data structure including the address generated by the action processor based on the address template.

[0006] At least one of a plurality of action lists may have an address template for accessing user interface components when an action is performed. This address template defines a first part and a second part, the first part corresponding to one of a plurality of applications, and the second part including input variables when the action is performed. A natural language processor can parse the input audio signal of the data packets to identify parameters defining the request to be performed by a second application. An action processor is configured to: generate an address for performing an action of the action list based on the address template of the action list. This address includes a first substring and a second substring, the first substring corresponding to the second application, and the second substring having parameters for controlling the execution of the action. A proxy interface can use the address generated based on the address template of the action list to identify the user interface components of the second application for an action list selected from the plurality of action lists.

[0007] Each action list can have a proxy identifier corresponding to one of multiple applications and a request identifier corresponding to the action. The natural language processor can parse the input audio signal of the data packets to identify the proxy identifier corresponding to the proxy, which corresponds to a second application installed on the client device. The action processor can select an action list from multiple action lists to execute an action, each action having a proxy identifier corresponding to the proxy identifier identified from the parsed input audio signal.

[0008] The action handler can determine the interface mode indicator of the action list to specify the rendering of user interface components when the action is performed, and in response to determining that the interface mode indicator specifies the rendering of user interface components, it identifies the user interface components of the second application for the action list.

[0009] The proxy interface can determine that the client device has not been authenticated by the second application based on the action list used to perform the action corresponding to the second request, and can, in response to determining that the client device has not been authenticated by the second application, present a prompt interface on the client device to authenticate the client device for the second application to perform the action corresponding to the second request.

[0010] The proxy interface can determine, based on the user's third application's authorization policy, that the output of the third application is not authorized to be presented together with the first application, that the third application is installed on the client device but is not in the foreground process, set the third application as the foreground process of the client device, change the first application from the foreground process to a non-foreground process running on the client device, and display the graphical user interface of the third application on the client device, which includes the output generated by the third application.

[0011] The proxy interface can determine, based on the authorization policy of the first application, that the first application authorizes the presentation of the user interface components of the second application together with the graphical user interface of the first application. The authorization policy allows user interface components from a first subset of multiple applications to be presented together with the graphical user interface and restricts user interface components from a second subset of multiple applications to be presented together with the graphical user interface. In response to determining that the first application authorizes the presentation of the user interface components of the second application together with the graphical user interface of the first application, the proxy interface displays, on the client device, user interface components including the output from the second application together with the graphical user interface of the first application.

[0012] The proxy interface can provide the action data structure to the second application so that the second application can monitor interaction events on the user interface components of the second application that are presented together with the graphical user interface of the first application, and process the interaction events in response to the detection of the interaction events to update the user interface components.

[0013] A natural language processor can be used to receive data packets via a digital assistant application, including input audio signals detected by sensors on a client device, which displays a graphical user interface (GUI) of the digital assistant application corresponding to a first application in a foreground process. An agent interface can display, within the GUI of the digital assistant application, user interface components including sub-components indicating requests identified from parsing the input audio signals, and user interface components including output from an agent corresponding to a second application. This agent may not have natural language processing capabilities.

[0014] The natural language processor can receive data packets including input audio signals detected by sensors of a client device, the client device displaying a graphical user interface of a first agent corresponding to a first application in a foreground process; and the agent interface is used to display user interface components including output from a second application as an overlay on the graphical user interface of the first agent.

[0015] Multiple action lists can have a first subset of action lists and a second subset of action lists, the first subset including action lists defined by the administrator of the data processing system, and the second subset including action lists provided by an agent service that processes one of the multiple applications.

[0016] The motion processor can provide a motion data structure to a second application so that the second application can parse the motion data structure to identify the motion to be executed, generate output by executing the motion identified from the motion data structure, and provide output to a user interface component to be presented together with the graphical user interface of the first application authorized to be presented together with the second application.

[0017] According to a second aspect of this disclosure, a method for rendering visual components on an application is disclosed. A data processing system having one or more processors can maintain multiple action lists for multiple applications. At least one of the multiple action lists can instruct the rendering of a user interface component of one of the multiple applications when an action is performed. The data processing system can receive data packets including input audio signals detected by sensors of a client device. The client device can display a graphical user interface of a first application in a foreground process on the client device. The data processing system can parse the input audio signals of the data packets to identify a request. The data processing system can select from the multiple action lists an action list in which a second application performs an action corresponding to the request through the multiple applications. The second application can be installed on the client device but is not in a foreground process. The data processing system can generate an action data structure based on the action list. The data processing system can provide the action data structure to the second application so that the second application parses the action data structure and performs the action to generate output. The data processing system can determine, based on the authorization policy of the second application, that the output from the second application performing the action is authorized to be rendered together with the graphical user interface of the first application. In response to determining that the output of the second application is authorized to be rendered together with the first application, the data processing system can identify a user interface component of the second application for an action list selected from the multiple action lists. The data processing system can display user interface components, including output from a second application, together with the graphical user interface of a first application that is authorized to be presented along with the second application on a client device.

[0018] The optional features of the first aspect described above or below may be combined with the second aspect.

[0019] These and other aspects, as well as the implementation methods, will be discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementation methods, and provide an overview or framework for understanding the nature and characteristics of the claimed aspects and implementation methods. The accompanying drawings provide illustration and further understanding of the various aspects and implementation methods, and are incorporated in and constitute a part of this specification. Attached Figure Description

[0020] The accompanying drawings are not intended to be drawn to scale. Similar reference numerals and designations are used throughout the drawings to denote similar elements. For the sake of brevity, it is not necessary to label every component in each drawing. In the drawings:

[0021] Figure 1 A block diagram of an example system for rendering visual components on an application in response to a voice input command, according to an example of this disclosure, is shown.

[0022] Figure 2 Examples of methods for use in accordance with this disclosure are shown. Figure 1 The diagram shows a timing diagram of an example data flow in the system that renders visual components in response to voice input commands.

[0023] Figure 3 Examples according to this disclosure are shown. Figure 1 The diagram shows a use case diagram of an example client device in the system that displays input messages and generates addresses.

[0024] Figure 4 Examples of use according to this disclosure are shown. Figure 1 The flowchart shown illustrates how the example system renders visual components on the application in response to voice input commands.

[0025] Figure 5 This is a block diagram of an example computer system. Detailed Implementation

[0026] The following is a more detailed description of various concepts and implementations relating to methods, apparatuses, and systems for rendering visual components in response to voice input commands in an application. The various concepts described above and discussed in more detail below can be implemented in any of many ways.

[0027] Multiple applications can be installed on a client device to provide pre-configured functionality based on requests from users operating the client device. At any given time, a subset of these applications can run on the client device as either foreground or non-foreground processes set by the operating system running on the client device. Foreground processes (including visible processes) can correspond to applications with a graphical user interface (GUI) displayed on the client device's screen, with which the user may be interacting. Non-foreground processes (e.g., service processes, background processes, or cached processes) can correspond to applications that do not display visual components to the user and execute without affecting the foreground process's GUI. Applications running on the client device can switch back and forth between running as foreground processes or non-foreground processes. After a switch, the GUI of an application that was previously a foreground process can be replaced by the GUI of the application now set as a foreground process. Additionally, processing and memory allocation may change due to the switching between foreground and non-foreground processes.

[0028] One of these applications, running in the foreground or off-foreground, may include a digital assistant application. Unlike at least some other applications installed on the client device, a digital assistant application may have natural language processing capabilities. Because the digital assistant application is configured with these capabilities, it can acquire and parse input audio signals detected through the client device's microphone to identify words from the input audio signals. Based on the identified words, the digital assistant application can determine the request to be performed and one or more parameters defining that request. The request may refer to one of the pre-configured functions with input arguments compatible with the parameters that the digital assistant application can perform (e.g., retrieving the results of a search query). For such requests, the digital assistant application itself can process the request and present audio or visual output by executing the request. However, some requests may not correspond to any pre-configured functions on the digital assistant application (e.g., playing a video from an online video platform). For these types of requests, the digital assistant application can identify another application capable of performing the request and invoke that application's functions to pass the parameters.

[0029] By invoking another application to execute a request, a digital assistant application can cause the application to switch between foreground and non-foreground processes, resulting in changes to the graphical user interface displayed on the client device. Furthermore, processing and memory allocation for each running application can change and fluctuate, especially when invoking an application that wasn't previously running. While the required request can ultimately be executed on the client device via the invoking application, the switching can disrupt the overall user experience of the applications on the client device. For example, a sudden change in the graphical user interface displayed on the client device forces the user to switch contexts between multiple application interfaces, thus degrading human-computer interaction (HCI) with the client device. The changes and fluctuations in the re-allocation of computing resources during the switching between foreground and non-foreground processes can further exacerbate the degradation of the client device.

[0030] This disclosure provides an improved user interface. Specifically, to address technical challenges in interfacing, this disclosure provides an action inventory for identifying and presenting user interface components on a graphical user interface of an application that will be overlaid in the foreground. The action inventory can specify whether to provide user interface elements of the application while maintaining the application in the background. The action inventory can also include address templates to construct addresses for accessing functions of the digital assistant application and providing user interface elements on client devices. The address templates of the action inventory can be, for example, Uniform Resource Identifier (URI) templates, used to construct network addresses using hostnames, pathnames, and queries to invoke application functions and user interface components. Hostnames (or schemes) can refer to the location or domain name of the application. Pathnames can refer to functions. Queries can include one or more input arguments for performing functions. The address templates can also define a mapping from a request for parsing an input audio signal to the function the application is to perform.

[0031] After recognizing the request from the input audio signal, the digital assistant application can determine that the request does not correspond to any functionality of the digital assistant application or any application running as a foreground process. Instead of indicating that the digital assistant application cannot perform the indicated request, it can identify another application to perform the function. While processing the request, the digital assistant application can remain as a background process, while the original application remains as a foreground process. Through this recognition, the digital assistant application can select an action list for the application for the function. The selected action list can indicate the user interface components of the application to be rendered. Using the action list, the digital assistant application can construct, expand, and generate the address of the function based on an address template. This address can include the hostname (or scheme) referencing the application, the path referencing the function and user interface components, and one or more queries including parameters.

[0032] As the address is generated from the address template, the digital assistant application can generate an action data structure (e.g., an API function call) that includes that address and pass it to the application. The application to which the action data structure is passed can be opened as a background process and remain in the background process after processing and completing the action. The action data structure can be passed to other applications because the original application will continue to remain as a foreground process. For example, the action data structure can be passed from the digital assistant application to the application via an API function call from the operating system on the client device. Because the address includes the hostname referencing the application, the action data structure can be directed to the referenced application. Upon receipt, the application can parse the action data structure to identify the address. The application can further parse the address to identify the pathname used to identify the function to be executed, and parse the string query to identify one or more parameters used to define the execution of that function. Based on this identification, the application can execute the function to generate output. Using the output generated by the application, the digital assistant can render the application's user interface components. The application's user interface components can overlay the graphical user interface of the application in the foreground process.

[0033] In this way, digital assistant applications can invoke functions from other applications and render user interface components from these applications, thereby enhancing the functionality of the digital assistant application. Invocations can be performed without requiring the user to manually search, open, and type to achieve the desired function or without context switching between different applications. Therefore, eliminating user-side context switching in operating the client device improves HCI. Furthermore, even if a request is recognized from the input audio signal, the application can remain in both foreground and background processes. By eliminating switching between foreground and background processes, digital assistant applications can reduce computational resource consumption and user-side time.

[0034] Reference Figure 1 The illustration depicts an example system 100 for rendering visual components on an application. System 100 may include at least one data processing system 102, at least one client device 104, and at least one proxy service 106. The data processing system 102, client device 104, and proxy service 106 may be communicatively coupled to each other via at least one network 114.

[0035] Components of system 100 can communicate via network 114. Network 114 may include, for example, a point-to-point network, a broadcast network, a wide area network (WAN), a local area network (LAN), a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, an SDH (Synchronous Digital Hierarchy) network, an NFC (Near Field Communication) network, a LAN, a wireless network, or a wired network, and combinations thereof. Network 114 may include wireless links, such as infrared channels or satellite bands. The topology of network 114 may include bus, star, or ring network topologies. Network 114 may include a mobile phone network using any one or more protocols for communication between mobile devices, including: Advanced Mobile Phone Protocol (AMPS), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), or Universal Mobile Telecommunications System (UMTS). Different types of data may be transmitted via different protocols, or the same type of data may be transmitted via different protocols.

[0036] Data processing system 102 and agent service 106 may each include multiple logically grouped servers, facilitating distributed computing technologies. The logical group of servers may be referred to as a data center, server farm, or machine farm. Servers may be geographically distributed. A data center or machine farm may be managed as a single entity, or a machine farm may include multiple machine farms. Servers in each computer farm may be heterogeneous: one or more servers or machines may operate according to one or more types of operating system platforms. Data processing system 102 and agent service 106 may each include: servers stored in one or more high-density rack systems within a data center, and associated storage systems, such as those located in an enterprise data center. In this way, data processing system 102 and agent service 106 with consolidated servers can improve system manageability, data security, physical security, and system performance by positioning servers and high-performance storage systems on a localized, high-performance network. Centralizing all or some of the components of data processing system 102 and agent service 106 (including servers and storage systems) and coupling them with advanced system management tools allows for more efficient use of server resources, saving power and processing requirements and reducing bandwidth usage. Each component of the data processing system 102 may include at least one processing unit, server, virtual server, circuitry, engine, agent, appliance, or other logical device, such as a programmable logic array configured to communicate with the data repository 128 and other computing devices. The agent service 106 may also include at least one processing unit, server, virtual server, circuitry, engine, agent, appliance, or other logical device, such as a programmable logic array configured to communicate with the data repository and other computing devices.

[0037] Data processing system 102 may include at least one instance of digital assistant application 108. Digital assistant application 108 may include at least one natural language processor (NLP) component 116 (sometimes referred to herein as a natural language processor) to parse audio-based input. Digital assistant application 108 may include at least one audio signal generator component 118 (sometimes referred to herein as an audio signal generator) to generate audio-based signals. Digital assistant application 108 may include at least one action processor component 120 (sometimes referred to herein as an action processor) to generate action data structures based on audio-based input. Digital assistant application 108 may include at least one response selector component 122 to select a response to the audio-based input signal. Digital assistant application 108 may include at least one proxy registry component 124 (sometimes referred to herein as a proxy registry) to maintain a list of actions used to generate action data structures. Digital assistant application 108 may include at least one proxy interface component 126 (sometimes referred to herein as a proxy interface) to communicate with one or more proxy applications 110.

[0038] Digital assistant application 108 may include at least one data repository 128. Data repository 128 of data processing system 102 may include one or more local or distributed databases and may include a database management system. Data repository 128 may include one or more regular expressions 130, one or more data parameters 132, one or more policies 134, one or more response data 136, one or more templates 138, and one or more action lists 140. NLP component 116, audio signal generator component 118, action processor component 120, response selector component 122, proxy registry component 124, proxy interface component 126, and data repository 128 may be separate from each other. Data repository 128 may include computer data storage or memory and may store other data such as one or more regular expressions 130, one or more data parameters 132, one or more policies 134, response data 136, templates 138, and one or more action lists 140. Data parameters 132, policies 134, and templates 138 may include information such as rules regarding voice-based conversations between client device 104 and data processing system 102. Response data 136 may include content items or associated metadata for audio output, and input audio messages that may be part of one or more communication sessions with client device 104. Action list 140 may include information for invoking agent application 110 or interfacing with agent application 110.

[0039] Data processing system 102 may include an instance of at least one agent application 110 (also referred to herein as an "agent" or "application") to perform various functions. Agent service 106 may include an instance of at least one agent application 110 to perform various functions. Agent application 110 may not have natural language processing capabilities (such as those provided by NLP component 116). Agent application 110 may also have a set of pre-configured or predetermined input capabilities. Input capabilities of agent application 110 may include, for example, input from a mouse, keyboard, touchscreen (e.g., display 148), or camera from client device 104. Input capabilities of agent application 110 may not include audio (e.g., microphone 146) from client device 104. Agent application 110 may have a graphical user interface (GUI) to receive input from client device 104. Based on input from predetermined input capabilities, agent application 110 may perform one or more functions. For example, agent application 110 may be a coffee purchasing application and may have a graphical user interface with one or more elements to specify a coffee order. In this application, the user operating client device 104 can manually enter coffee, size, and billing information on a graphical user interface rendered on display 148 of client device 104. Once set up, agent application 110 on client device 104 can send the input to agent service 106 via network 114. Agent service 106 can then execute the purchase order indicated by the input entered through the graphical user interface of agent application 110.

[0040] Components of data processing system 102 may each include at least one processing unit or other logic device, such as a programmable logic array engine or module configured to communicate with data storage 128. Components of data processing system 102 may be separate components, a single component, or part of multiple data processing systems 102. System 100 and its components (such as data processing system 102) may include hardware elements such as one or more processors, logic devices, or circuitry.

[0041] The functionality of the data processing system 102 may include, for example, digital assistant application 108 or agent application 110, or may be accessible from one or more client devices 104. The functionality of the data processing system 102 may correspond to or interface with the functionality executed on the client device 104 or with the digital assistant application 108 executed on the client device 104. The client device 104 may include and execute separate instances of one or more components of the digital assistant application 108. The client device 104 may otherwise have access to the functionality of components of the digital assistant application 108 on the remote data processing system 102 via network 114. For example, the client device 104 may include the functionality of NLP component 116 and may access the remaining components of the digital assistant application 108 to the data processing system 102 via network 114. The functionality of the data processing system 102 may correspond to or interface with the functionality of the agent application 110 executed on the client device 104. Client device 104 may otherwise have functional access to components of agent application 110 on data processing system 102 or agent service 106 via network 114. For example, client device 104 may preprocess input from a mouse, keyboard, touchscreen, or camera for agent application 110 and provide that input to agent service 106 to perform the functions of agent application 110. Digital assistant application 108 running on client device 104 may include the functionality of digital assistant application 108 running on data processing system 102.

[0042] Client device 104 may include at least one logical device, such as a computing device having a processor to communicate with data processing system 102 via network 114. Client device 104 may include instances of any components described with respect to data processing system 102 or agent service 106. Client device 104 may include instances of digital assistant application 108. Client device 104 may include instances of agent application 110. Client device 104 may include desktop computers, laptop computers, tablet computers, personal digital assistants, smartphones, mobile devices, portable computers, thin client computers, virtual servers, speaker-based digital assistants, or other computing devices.

[0043] Client device 104 may include one or more of the following: at least one instance of digital assistant application 108, at least one instance of agent application 110, and at least one communication interface 112, executing, interfacing with, or otherwise communicating with them. Additionally, client device 104 may include at least one speaker 144, at least one microphone 146, and at least one display 148, interfacing with or otherwise communicating with them. Client device 104 may include an audio driver to provide a software interface to speaker 144 and microphone 146. The audio driver may execute instructions provided by data processing system 102 to control speaker 144 to generate corresponding acoustic waves or sound waves. The audio driver may execute audio files or other instructions to convert acoustic waves or sound waves acquired from microphone 146 to generate audio data. For example, the audio driver may perform an analog-to-digital converter (ADC) to convert acoustic waves or sound waves into audio data. Client device 104 may include a graphics driver to provide a software interface to display 148. The graphics driver may execute instructions provided by data processing system 102 or agent service 106 to control display 148 to generate corresponding renderings thereon.

[0044] An instance of digital assistant application 108 on client device 104 may include or be executed by one or more processors, logic arrays, or memories. An instance of digital assistant application 108 on client device 104 may detect keywords and perform actions based on those keywords. Digital assistant application 108 on client device 104 may be an instance of digital assistant application 108 executed at data processing system 102, or may perform any function of digital assistant application 108. An instance of digital assistant application 108 on client device 104 may filter out or modify items before sending items as data to data processing system 102 (e.g., an instance of digital assistant application 108 on data processing system 102) for further processing. An instance of digital assistant application 108 on client device 104 may convert analog audio signals detected by speaker 144 into digital audio signals and send one or more data packets carrying the digital audio signals to data processing system 102 via network 114. An instance of the digital assistant application 108 on client device 104 can transmit a data packet carrying some or all of the input audio signal in response to detecting an instruction to perform such transmission. This instruction may include, for example, triggering a keyword or other key or approving the transmission of a data packet including the input audio signal to data processing system 102.

[0045] An instance of a digital assistant application 108 on client device 104 can perform pre-filtering or preprocessing on the input audio signal to remove certain frequencies of the audio. Pre-filtering can include filters such as low-pass filters, high-pass filters, or band-pass filters. Filters can be applied in the frequency domain. Digital signal processing techniques can be used to apply the filters. Filters can preserve frequencies corresponding to human speech or human utterances while eliminating frequencies outside the typical frequencies of human utterances. For example, a band-pass filter can remove frequencies below a first threshold (e.g., 70 Hz, 75 Hz, 80 Hz, 85 Hz, 90 Hz, 95 Hz, 100 Hz, or 105 Hz) and above a second threshold (e.g., 200 Hz, 205 Hz, 210 Hz, 225 Hz, 235 Hz, 245 Hz, or 255 Hz). Applying band-pass filters can reduce computational resource utilization in downstream processing. An instance of the digital assistant application 108 on client device 104 can apply a bandpass filter before sending the input audio signal to data processing system 102, thereby reducing network bandwidth utilization. However, given the available computing resources and network bandwidth of client device 104, providing the input audio signal to data processing system 102 to allow data processing system 102 to perform filtering may be more efficient. An instance of the digital assistant application 108 on client device 104 can apply additional preprocessing or pre-filtering techniques, such as noise reduction techniques, to reduce the level of ambient noise that may interfere with the natural language processor. Noise reduction techniques can improve the accuracy and speed of the natural language processor, thereby improving the performance of data processing system 102 and managing the rendering of the graphical user interface provided via the display.

[0046] Client device 104 can be operated by an end user who inputs a voice query as audio input (via microphone 146 or speaker 144) to client device 104 and receives audio (or other) output from data processing system 102 or proxy service 106 to present, display, or render to the end user of client device 104. Digital components may include computer-generated speech, which may be provided to client device 104 from the data processing system. Client device 104 may render the computer-generated speech to the end user via speaker 144. Computer-generated speech may include recordings from real people or computer-generated language. Client device 104 may provide visual output via a display 148 communicatively coupled to client device 104.

[0047] Client device 104 may have or execute as at least one foreground process 150 (also referred to herein as a foreground service). Foreground process 150 may correspond to at least one application (e.g., digital assistant application 108 or agent application 110) rendered on display 148 and executed on client device 104. For example, foreground process 150 may correspond to an application having a graphical user interface occupying a large portion of display 148. The application running as foreground process 150 on client device 104 may have a graphical user interface rendered on display 148 of client device 104. The graphical user interface of the application in foreground process 150 may occupy at least a portion (e.g., at least one-third) of display 148 of client device 104. The application running as foreground process 150 on client device 104 may wait (e.g., via I / O devices communicatively coupled to client device 104) for user interaction with the graphical user interface to execute one or more routines. The application in foreground process 150 may process user interaction to execute one or more routines of the application. In the processing of user interaction, the application in the foreground process 150 can generate output and render the result output on the graphical user interface of the application in the foreground process 150.

[0048] Client device 104 may have or execute at least one non-foreground process 152 (also referred to herein as a background process, background service, non-foreground process, or non-foreground service). Non-foreground process 152 may correspond to at least one application (e.g., digital assistant application 108 or another agent application 110) that is not rendered on display 148 and is executing on client device 104. An application running as a non-foreground process 152 on client device 104 may not render any graphical user interface of the application on display 148 of client device 104. For example, non-foreground process 152 may correspond to an application that is minimized and does not render any graphical user interface components on display 148. An application running as a non-foreground process 152 on client device 104 may execute one or more routines without receiving any user interaction with the application. For example, an application running as a non-foreground process 152 may process audio detected on microphone 146 of client device 104 without rendering any graphical user interface on display 148. In the processing of user interaction, the application in the foreground process 150 can generate output without rendering the output as part of the graphical user rendering on the display 148.

[0049] Client device 104 can (e.g., using an operating system running on client device 104) manage applications in foreground process 150 and non-foreground process 152. Client device 104 can switch applications between foreground process 150 and non-foreground process 152. Switching applications between foreground process 150 and non-foreground process 152 can be in response to a call from at least one application. The called application can run in non-foreground process 152, or it can run in neither foreground process 150 nor non-foreground process 152. The call can be made by the operating system running on client device 104, by another application (e.g., digital assistant application 108 or agent application 110), or via a user interaction event. In response to the call, client device 104 can set an application running in foreground process 150 as a non-foreground process 152. Furthermore, client device 104 can set an application running in non-foreground process 152 as a foreground process 150. When the called application is not running as a non-foreground process 152, client device 104 can also instantiate the application into foreground process 150. For example, a web browser can execute as a foreground process 150 on client device 104, while rendering a graphical user interface on display 148. A clock application can run on client device 104 as a non-foreground process 152 to track time without rendering any graphical user interface on display 148. When invoking the clock application's alarm function, client device 104 can set the web browser as a non-foreground process 152, thereby removing the rendering of the web browser's graphical user interface. In summary, client device 104 can set the clock application as a foreground process 150.

[0050] Data processing system 102 and agent service 106 may each include at least one server having at least one processor. For example, data processing system 102 and agent service 106 may each include multiple servers located in at least one data center or server farm. Data processing system 102 may include at least one computing resource or server. Data processing system 102 may include at least one communication interface 112, which may be interfaced with or otherwise communicate with. Data processing system 102 may include at least one instance of digital assistant application 108 on data processing system 102, which may be interfaced with or otherwise communicate with. Instances of digital assistant application 108 on data processing system 102 may include at least one NLP component 116, at least one audio signal generator component 118, at least one motion processor component 120, at least one response selector component 122, at least one agent registry component 124, and at least one agent interface component 126, etc., which may be interfaced with or otherwise communicate with. Data processing system 102 may include at least one data repository 128, which may be interfaced with or otherwise communicate with. At least one data repository 128 may include or store regular expressions 130, data parameters 132, strategies 134, response data 136, templates 138, and action lists 140 in one or more data structures or databases. The data repository 128 may include one or more local or distributed databases and may include database management.

[0051] An instance of the digital assistant application 108 of the data processing system 102 can execute or run an instance of the proxy registry component 124 to maintain a set of action lists 140 on the data repository 128. The proxy registry component 124 can maintain and store a set of action lists 140 for one or more proxy applications 110 on the data repository 128. A subset of the action lists 140 can be defined by the administrator of the data processing system 102 or the digital assistant application 108 for an application type corresponding to the proxy application 110. For example, the administrator of the digital assistant application 108 can configure or define a built-in set of action lists 140 for a purchase order application. Another subset of the action lists 140 can be provided by a proxy service in the proxy service 106 that processes resources for the corresponding proxy application 110. At least one of the action lists 140 can be a preset action list 140 used to perform an action in response to a determination that a request does not match any other action list 140 of the proxy application 110. For example, agent service 106 can provide data processing system 102 with a customized list of actions 140 to be stored in data repository 128. Agent registry component 124 can receive a subset of the action list 140 provided by agent service 106. Upon receiving it, agent registry component 124 can store the received action list 140 in data repository 128. The existence of the action list 140 for actions against agent application 110 in data repository 128 can instruct agent application 110 to perform the actions corresponding to action list 140.

[0052] The proxy registry component 124 can maintain and store a set of user interface components 142 for one or more proxy applications 110 on the data repository 128. Each user interface component 142 can be associated with at least one of the action lists 140. A subset of the user interface components 142 can be defined by the administrator of the data processing system 102 or the digital assistant application 108. For example, the administrator of the digital assistant application 108 can configure or define a set of built-in user interface components 142. Another subset of the user interface components 142 can be provided by a proxy service in the proxy service 106 that processes resources for the corresponding proxy application 110. The proxy service 106 can store and maintain the user interface components 142 corresponding to this subset. For example, the proxy service 106 can provide a set of customized user interface components 142 to the data processing system 102 for storage on the data repository 128. The proxy registry component 124 can receive a subset of the user interface components 142 provided by the proxy service 106. Upon receiving it, the proxy registry component 124 can store the received user interface components 142 on the data repository 128. The maintenance of the user interface component 142 by the proxy service 106 can be separated from the maintenance of the user interface component 142 on the data repository 128.

[0053] Each user interface component 142 may have one or more interface elements. Each interface element may correspond to a sub-component of the user interface component 142. At least one interface element may correspond to a request to be identified from an input audio signal detected by the microphone 146 of the client device 104. At least one interface element may correspond to a response indicative of output generated by the agent application 110 associated with the user interface component 142. Furthermore, each user interface component may have one or more attributes. These attributes may include element type (e.g., command button, scroll bar, text box, and image), size, position within the user interface component 142, transparency, and shape, etc. For example, the user interface component 142 may have buttons typically positioned along the bottom, text boxes above the buttons, and slots for images typically in the center, etc. The user interface component 142 itself may have one or more attributes. The attributes of the user interface component 142 may include size, position within the display 148, transparency, and shape, etc.

[0054] Each action list 140 may have at least one address template for at least one action of one of the proxy applications 110. The address template may be used for a single action of the proxy application 110 or for multiple actions carried by the proxy application 110. The address template may also be used to access or retrieve a user interface component 142 to be generated when the action is performed. The address template for the action may include a first part and a second part. The address template may be, for example, a Uniform Resource Identifier (URI) template from which a URI will be generated. The first part of the address template may correspond to one of the proxy applications 110 or a proxy service 106 that provides resources to the proxy application 110. For example, the first part of the address template may include a scheme in a custom deep link URI referencing the proxy application 110. The first part of the address template may also include a hostname referencing the proxy service 106.

[0055] The second part of the address template may correspond to an action to be performed by the proxy application 110. The second part of the address template may also include one or more input variables (also referred to herein as input arguments of a field) for performing the action. For example, the second part of the address template may include a pathname in the URI corresponding to the action to be taken. The second part may also include one or more query strings after the pathname in the URI. Each query string may correspond to one of the input variables used to perform the action. A subset of the input variables may include optional input variables or auxiliary input variables. An action may be performed by the proxy application 110 without auxiliary input variables in the address.

[0056] Action List 140 may include a mapping (sometimes referred to herein as a parameter mapping) for the second part of the address template. The mapping may specify or define a correspondence between at least one input variable of the second part and one or more words to be identified from the parsed input audio signal. One or more words from the parsed input audio signal may correspond to input variables defining actions to be performed by the agent application 110. The mapping may define the insertion of words into the input variables of the address template. For example, a mapping for a coffee purchase order action may specify that the coffee name parsed from the audio signal be inserted into the coffee input variable of the address template. Action List 140 may specify or include a set of allowed values ​​(sometimes referred to as an entity list) for each input variable. This set of allowed values ​​may be part of the mapping of Action List 140. For example, for the input variable of coffee name, allowed values ​​for a coffee purchase order may include “latte,” “cafe latte,” and “latte cafe,” etc. Action List 140 may also include markers indicating the corresponding input variables of the address template as optional or auxiliary. The markers may be part of the mapping for the second part of the address template.

[0057] In addition to the address template, each action list 140 maintained by the agent registry component 124 may include information related to an action for the action list 140. The action list 140 may include an interface mode indicator. The indicator may specify whether a user interface component 142 of the agent application 110 is displayed when the action is performed. The user interface component 142 may be a visual element to be rendered on the display 148 of the client device 104. The size of the user interface component 142 may be smaller than the size of the display 148. For example, the user interface component 142 may include a box-sized visual element occupying 12.5% ​​to 25% of the display 148. The user interface component 142 may be associated with at least one of the action lists 140 maintained on the data repository 128.

[0058] User interface component 142 may be associated with action list 140. Action list 140 may include an address referencing user interface component 142. The address referencing user interface component 142 may be different from or separate from the address template used to perform the action. The address referencing user interface component 142 may have the same scheme or hostname as the address template for the same action list 140. For example, the address may include the hostname of proxy service 106 that manages resources for proxy application 110. The address referencing user interface component 142 may have a different hostname than the address template for the same action list 140. For example, the hostname may refer to data repository 128 of data processing system 102. Action list 140 may also include a user interface identifier corresponding to user interface component 142. This identifier can be used to index and reference user interface component 142 maintained on data repository 128. Furthermore, the address template of action list 140 can be used to reference user interface component 142 and the action to be performed by proxy application 110.

[0059] Action list 140 may also include at least one agent identifier corresponding to one of the agent applications 110 performing the action. The agent identifier may include a set of alphanumeric characters referencing agent application 110. The agent identifier can be used to identify the agent application 110 associated with action list 140 based on words parsed from the input audio signal. Action list 140 may include at least one request identifier corresponding to an action to be performed by agent application 110. This request identifier (sometimes referred to herein as an "intent identifier" or "intent name") may include a set of alphanumeric characters referencing the action to be performed. The address template may be uniquely identified by one or more request identifiers. The request identifier can be used to identify action list 140 based on words parsed from the input audio signal.

[0060] Each action list 140 can be implemented or specified using a markup language such as an Extensible Markup Language (XML) file. For example, action list 140 can be an XML file in the following form: <actions> <action intentname="actions.intent.CHECK_STEPS"> <fulfillment fulfillmentMode="actions.fulfillment.SLICE" urlTemplate="https: / / ex_personalmetrics.example.com / checksteps{?username}"> <parameter-mapping intentParameter="user.name" urlParameter="username" required="true" / > <fulfillment urlTemplate="https: / / ex_personalmetrics.example.com / checksteps" / > < / action> < / actions>

[0061] In the example above, action list 140 can be used for ordered rides performed by a personal fitness app. The entry “actions.intent.CHECK_STEPS” can correspond to the request identifier of action list 140. The entry “actions.fulfillment.SLICE” can specify the display of the user interface component 142 associated with action list 140. The entry “https: / / ex_personalmetrics.example.com / checksteps{?username}” can correspond to the address template, where “https: / / ex_personalmetrics.example.com / ” corresponds to the first part and “checksteps{?username}” corresponds to the second part. Entries starting with “parameter-mapping” can correspond to mappings. Finally, the entry “https: / / ex_personal metrics.example.com / checksteps” can correspond to the individual address of user interface component 142.

[0062] Proxy registry component 124 can construct, create, or generate a set of action lists 140 for each proxy application 110. Proxy service 104 of proxy application 110 can provide a configuration file associated with proxy application 110 to proxy registry component 124 via network 114. The configuration file can be, for example, an application binary interface (ABI) of proxy application 110 submitted to proxy registry component 124 by the application developer associated with proxy application 110. Proxy service 104 of proxy application 110 can provide metadata associated with proxy application 110. Metadata may include a proxy identifier and application type to proxy registry component 124, etc. The application type can indicate the use of proxy application 110. Proxy service 104 of proxy application 110 can provide interface mode indicators for one or more actions that can be performed by proxy application 110. Proxy registry component 124 can receive information from proxy service 106 for generating the set of action lists 140 for proxy application 110.

[0063] The proxy registry component 124 can read, ingest, and parse the configuration file used by the proxy application 110 to identify one or more actions. For each identified function, the proxy registry component 124 can generate an action list 140 for that action. The action can be one of the actions in the set of action lists 140 defined by the administrator of the data processing system 102 or the digital assistant application 108 for the application type of the proxy application 110. The proxy registry component 1240 generates an address template for the action performed by the proxy application 110. The first part of the address template (e.g., hostname or scheme) can correspond to the proxy application 110. The second part of the address template can be predefined for the action (e.g., using a preset pathname or query string). The proxy registry component 124 can also generate a mapping for the identified functions in the action list 140. The proxy registry component 124 can include or add a request identifier corresponding to the identified action to the action list 140. The mapping can include a set of allowed values ​​for the input variables of the address template. The proxy registry component 124 can include or add a proxy identifier corresponding to the proxy application 110 to the action list 140. Using the construction of action list 140, proxy registry component 124 can store action list 140 on data store 128.

[0064] Data processing system 102 may include at least one communication interface 112. Communication interface 112 may be configured, constructed, or operated to receive and send information using, for example, data packets. Communication interface 112 may use one or more protocols, such as network protocols, to receive and send information. Communication interface 112 may include a hardware interface, a software interface, a wired interface, or a wireless interface. Communication interface 112 may be a data interface or a network interface that enables components of system 100 to communicate with each other. The communication interface 112 of data processing system 102 may provide or send one or more data packets, including action data structures, audio signals, or other data, to client device 104 or proxy service 106 via network 114. For example, data processing system 102 may provide output signals from data repository 128 or audio signal generator component 118 to client device 104.

[0065] The data processing system 102 can also instruct the client device 104 to perform functions indicated in the action data structure via data packet transmission. It can acquire output signals, generate output signals, convert output signals into one or more data packets (or other communication protocols), or send output signals as one or more data packets (or other communication protocols) from the data processing system 102 (or other computing device) to the client device 104. The communication interface 112 can facilitate the conversion or formatting of data from one format to another. For example, the communication interface 112 may include an application programming interface (“API”) that includes definitions for communication between various components, such as software components. Applications, scripts, programs, or other components associated with the data processing system 102 can be installed on the client device 104. Applications can enable the client device 104 to transmit input audio signals (and other data) to the communication interface 112 of the data processing system 102.

[0066] Data processing system 102 may include applications, scripts, or programs installed on client device 104, such as an instance of digital assistant application 108 on client device 104, to transmit input audio signals to communication interface 112 of data processing system 102 and drive components of the client computing device to render output audio signals or visual output. Data processing system 102 may receive data packets, digital files, or other signals that include or identify input audio signals (or multiple input audio signals). Client device 104 may detect audio signals via speaker 144 and convert analog audio signals into digital files via an analog-to-digital converter. For example, an audio driver may include an analog-to-digital converter component. A preprocessor component may convert audio signals into digital files that can be transmitted via data packets over network 114.

[0067] An instance of the digital assistant application 108 of the data processing system 102 or client device 104 can execute or run the NLP component 116 to receive or acquire data packets including input audio signals detected by the microphone 146 of the client device 104. The data packets may provide digital files. The NLP component 116 can receive or acquire digital files or data packets including audio signals and parse the audio signals. For example, the NLP component 116 can provide human-computer interaction. The NLP component 116 may be configured with techniques for understanding natural language and enabling the data processing system 102 to derive meaning from human or natural language input. The NLP component 116 may include or be configured with machine learning-based techniques, such as statistical machine learning. The NLP component 116 may utilize decision trees, statistical models, or probabilistic models to parse the input audio signals.

[0068] For example, NLP component 116 can perform functions such as named entity recognition (e.g., given a stream of text, determining which items in the text map to names such as people or places, and the type of each such name, such as person, location (e.g., "home"), or organization), natural language generation (e.g., translating information or semantic intent from a computer database into understandable human language), natural language understanding (e.g., translating text into more formal representations, such as first-order logical structures that a computer module can manipulate), machine translation (e.g., automatically translating text from one human language to another), morpheme segmentation (e.g., separating words into individual morphemes and identifying the category of the morphemes, which can be challenging given the complexity of the morphemes or structures of words in that language), question answering (e.g., determining answers to human language questions, which can be specific or open-ended), and semantic processing (e.g., processing that can be performed after words have been identified and their meanings encoded to relate the identified words to other words with similar meanings).

[0069] NLP component 116 (and digital assistant application 108, as a whole) can execute as one of the applications in non-foreground process 152. When an input audio signal is detected on microphone 146, client device 104 can continue digital assistant application 108 as one of the applications in non-foreground process 152. Client device 104 may have one or more other applications running as non-foreground process 152 (e.g., agent application 110). Client device 104 may have at least one application running in foreground process 150 (e.g., another agent application 110). An instance of NLP component 116 of digital assistant application 108 on client device 104 can initiate processing of the input audio signal without switching between foreground process 150 and non-foreground process 152. Applications running as foreground process 150 and non-foreground process 152 can be maintained. For example, NLP component 116 on client device 104 can receive the input audio signal and apply natural language processing functions to the input audio signal without minimizing the graphical user interface of the application running as foreground process 150. When an input audio signal is received, alternatively, NLP component 116 (and digital assistant application 108, as a whole) can be executed as one of the applications in foreground process 150. After receiving the input audio signal, digital assistant application 108 can continue to be executed as one of the applications in foreground process 150 of client device 104.

[0070] NLP component 116 can parse the audio input signal and convert it into a recognizable string by comparing the input signal with a stored set of representative audio waveforms (e.g., in data repository 128) and selecting the closest match. NLP component 116 can also segment or divide the input audio signal into one or more audio segments with durations (e.g., 15 seconds to 2 minutes) to process each segment. The audio waveform set can be stored in data repository 128 or another database accessible to data processing system 102. Representative waveforms are generated among a large number of users and can then be augmented using utterance samples from users. After the audio signal is converted into recognizable text, NLP component 116 matches the text with words associated with actions that data processing system 102 can dispatch, for example, through training among users or by manual specification. NLP component 116 can determine that the input audio signal acquired from microphone 146 does not contain any recognizable string. When determining that the input audio signal does not contain any recognizable string, NLP component 116 can determine that the input audio signal contains silence (e.g., maximum amplitude less than 0 dB). Additionally, NLP component 116 can determine the signal-to-noise ratio (SNR) of the input audio signal. NLP component 116 can compare the SNR of the input audio signal with a threshold SNR (e.g., -20 dB). In response to determining that the SNR of the input audio signal is greater than the threshold SNR, NLP component 116 can determine that the input audio signal does not contain any recognizable strings.

[0071] In addition to or in lieu of input audio signals, data processing system 102 can receive image or video input signals. NLP component 116 can convert the image or video input into text or digital files. NLP component 116 can process, analyze, or interpret the image or video input to perform actions, generate requests, or select or recognize data structures. Data processing system 102 can use, for example, image interpretation techniques, computer vision, machine learning engines, or other techniques to process the image or video input signals to recognize or interpret the images or videos to convert them into digital files. One or more image interpretation techniques, computer vision techniques, or machine learning techniques can be collectively referred to as imaging techniques. In addition to or in lieu of audio processing techniques, data processing system 102 (e.g., NLP component 116) can utilize imaging techniques for configuration.

[0072] By parsing the input audio signal, NLP component 116 can determine or identify at least one request. The input audio signal may include, for example, a query, question, command, instruction, or other statement in natural language. The request may correspond to at least one trigger keyword, which is identified from the recognized string derived from the input audio signal. The request may indicate an action to be taken. For example, NLP component 116 can parse the input audio signal to identify at least one request to leave home for dinner and to watch a movie in the evening. The trigger keyword may include at least one word, phrase, root word, or part of a word, or derivative, indicating the action to be taken. For example, the trigger keyword "go" or "to go to" from the input audio signal may indicate a request for transportation. In this example, the input audio signal (or the identified request) does not directly express a transportation intent, but the trigger keyword indicates that transportation is an auxiliary action to at least one other action indicated by the request.

[0073] To identify the request, NLP component 116 can apply semantic processing techniques to the input audio signal to identify trigger keywords corresponding to the request. NLP component 116 can apply semantic processing techniques to the input audio signal to identify trigger phrases that include one or more trigger keywords (such as a first trigger keyword and a second trigger keyword). For example, the input audio signal may include the sentence "Find the nearest café." NLP component 116 can determine that the input audio signal includes the trigger keyword "find." NLP component 116 can determine that the request is to search for locations near client device 104.

[0074] In addition to the request, NLP component 116 can identify at least one parameter that defines the request. This parameter can define the request, serving as a supplement or limitation to the action to be taken corresponding to the request. The parameter can further specify the action to be taken. The parameter may include a subset of recognized strings (excluding trigger keywords) converted from the input audio signal. NLP component 116 can apply semantic processing techniques to the input audio signal to identify one or more descriptor words associated with the identified trigger keywords. From the example “Find the nearest café”, NLP component 116 may have already identified the item “Find” as the trigger keyword. Combined, NLP component 116 can identify “nearest” as the first parameter and “café” as the second parameter qualifying the request “Find”.

[0075] Furthermore, NLP component 116 can identify an application identifier based on a string recognized from the input audio signal. This application identifier may correspond to one of the proxy applications 110. To identify the application identifier, NLP component 116 can perform a named entity recognition algorithm on the string recognized from the input audio signal. When using the named entity recognition algorithm, NLP component 116 can maintain a list of proxy identifiers for the proxy applications 110. This list may include proxy identifiers for the proxy applications 110 installed on the client device 104 that receives the input audio signal including the request. By applying the named entity recognition algorithm, NLP component 116 can determine that the input audio signal includes an application identifier corresponding to a proxy application 110. For example, NLP component 116 can determine that the input audio signal “Order car with ride sharing service XYZ” includes the explicit proxy application 110 “ride sharing service XYZ”. Conversely, by applying the named entity recognition algorithm, NLP component 116 can determine that the input audio does not have any application identifier corresponding to any proxy application 110.

[0076] Based on the request, NLP component 116 can determine whether the request corresponds to a function of digital assistant application 108 or agent application 110. To determine this, NLP component 116 can access a list of requests for functions of digital assistant application 108. This list of requests for functions of digital assistant application 108 can be maintained in a data repository 128. The request list may include a set of strings containing trigger keywords and requests corresponding to those pre-defined as being associated with functions of digital assistant application 108. For example, the list may include "take", "show", "search for", and "find", etc. Using the identification of the request, NLP component 116 can compare the request with the list of requests for functions of digital assistant application 108.

[0077] NLP component 116 can also use regular expression 130 maintained on data repository 128 to determine whether the request corresponds to a function of digital assistant application 108. Regular expression 130 can define a matching pattern to determine whether keywords identified from the input audio signal reference at least one function of digital assistant application 108. Regular expression 130 can also specify which keywords are used to execute the command indicated in the input audio signal. For example, regular expression 130 can take the form {[request], [auxiliary arguments]}. For keywords in the input audio signal determined to reference a function of digital assistant application 108, regular expression 130 can specify one or more keywords including a request to digital assistant application 108 and auxiliary arguments. Regular expression 130 can specify the order of requests and referential keywords among one or more keywords identified from the input audio signal.

[0078] Based on the comparison, NLP component 116 can determine that the request identified from the input audio signal matches a request in a request list for functions of digital assistant application 108. In response to this determination, NLP component 116 can determine that the request corresponds to one of the functions of digital assistant application 108. For example, NLP component 116 can parse the words "What is the weather?" from the input audio signal and can recognize that the input audio signal includes a request for the weather. The request list for functions of digital assistant application 108 can specify that the request for the weather is one of the functions of digital assistant application 108. In this example, NLP component 116 can determine a match between two requests and can determine that the request references one of the functions of digital assistant application 108.

[0079] In response to determining that the request corresponds to a function of the digital assistant application 108, the digital assistant application 108 can execute and satisfy the request identified from parsing the input audio signal. Alternatively, the digital assistant application 108 can execute the request based on parameters defining the request. For example, the request could be a search for the definition of a word (e.g., "abnegation"). The digital assistant application 108 can perform an internet search to retrieve the definition of the word. Upon satisfying the request, the digital assistant application 108 can invoke the response selector component 122. The response selector component 122 can select or identify a response phrase using a strategy 134 or response data 136 maintained on the data repository 128. The strategy 134 can be request-specific, and the response data 136 can be specified for that request. The response selector component 122 can search the strategy 134 to generate output when the request is satisfied using the request type of the response. Once the strategy 134 is identified, the response selector component 122 can select a response phrase. The response phrase can include a set of strings, such as words or phrases. Digital assistant application 108 can display a response phrase on the display 148 of client device 104. For example, digital assistant application 108 can display content items including the response phrase on the graphical user interface of digital assistant application 108 on the display 148 of client device 104.

[0080] By recognizing the response phrase, the digital assistant application 108 can invoke the audio signal generator component 118. An instance of the digital assistant application 108 running on the data processing system 102 or the client device 104 can execute the audio signal generator component 118. Using the selected response phrase, the audio signal generator component 118 can generate an output audio signal. Based on one or more words included in the response phrase, the audio signal generator component 118 can generate the output audio signal. The audio signal generator component 118 can then play the output audio signal on the speaker 144 of the client device 104.

[0081] Conversely, by comparison, NLP component 116 can determine that the request parsed from the input audio signal does not match any of the requests in the request list for functions of digital assistant application 108. In response to this determination, NLP component 116 can determine whether the request corresponds to one of the functions of agent application 110. To determine this, NLP component 116 can access the request list for functions of agent application 110. The request list for functions may be for agent application 110 installed on client device 104. The request list for functions of agent application 110 may be maintained on data repository 128. The request list may include a set of strings of trigger keywords or requests corresponding to requests pre-defined as associated with functions of agent application 110. NLP component 116 can compare the request with the request list for functions of agent application 110.

[0082] By comparison, NLP component 116 can determine that the request identified from the input audio signal matches a request in a list of requests for functions of agent application 110. In response to this determination, NLP component 116 can determine that the request corresponds to one of the functions of agent application 110. For example, NLP component 116 can parse the phrase "Buy melaundry detergent" from the input audio signal and can recognize that the input audio signal includes the purchase of detergent. The list of requests for functions of agent application 110 can specify that a request for weather is one of the functions of agent application 110. In this example, NLP component 116 can determine a match between the two requests and can determine that the request references one of the functions of agent application 110, namely, the function used for this purchase. Additionally, in response to determining that the input audio signal includes the application identifier of agent application 110, NLP component 116 can determine that the request does not correspond to a function of digital assistant application 108.

[0083] On the other hand, NLP component 116 can determine by comparison that the request does not correspond to any function of agent application 110. Based on this determination that the request does not correspond to digital assistant application 108 or agent application 110, response selector component 122 can generate or identify a response phrase indicating that the request does not correspond to any function available on client device 014. For example, the response phrase could include “Sorry, I don’t understand your request”. A response phrase can be selected from response data 136 based on this determination. Additionally, audio signal generator component 118 can use one or more words from the response phrase to generate a corresponding output audio signal. Audio signal generator component 118 can play the output audio signal on speaker 144 of client device 104.

[0084] An instance of the digital assistant application 108 running on the data processing system 102 or client device 104 can execute the motion processor component 120 to generate a motion data structure based on the determination. The motion processor component 120 can execute a script or program based on input received from the NLP component 116. The agent service 106 can provide the script or program. The agent service 106 can make the script or program available to the data processing system 102 via an API. The motion processor component 120 can determine parameters or responses to input fields and can package data into a motion data structure. The motion data structure can be provided to the agent application 110 via an API. The motion processor component 120 can send the motion data structure to the agent service 106 to fulfill requests identified from the input audio signal parsed by the NLP component 116.

[0085] In response to determining that a request corresponds to the functionality of the proxy application 110, the action processor component 120 can identify or select an action list 140 from the data repository 128 for the action corresponding to the request. The action processor component 120 can select at least one action list 140 from the set of action lists 140 maintained on the data repository 128. The action processor component 120 can use the request identified from the input audio signal to search the action repository 128 for the action list 140. The action processor component 120 can also use one or more parameters identified from the input audio signal to search the data repository 128 for the action list 140. The action processor component 120 can iterate through the set of action lists 140 maintained on the data repository 128. For each action list 140, the action processor component 120 can identify the request identifier of that action list 140.

[0086] Action processor component 120 can compare a request identified from an input audio signal with a request identifier in action list 140. This comparison can be made based on a semantic knowledge graph. Action processor component 120 can invoke NLP component 116 to use the semantic knowledge graph to determine whether the request is semantically relevant to the request identifier. The semantic knowledge graph can include a set of nodes and edges. Each node can correspond to one or more words or phrases. Each edge can define the semantic distance between a pair of words represented by corresponding words on the node. Using the semantic knowledge graph, NLP component 116 can identify the node corresponding to the request, the node corresponding to the request identifier, and the semantic distance indicated in that pair of nodes. NLP component 116 can compare the semantic distance to a threshold distance. When it is determined that the semantic distance does not meet the threshold distance (e.g., is greater than the threshold distance), NLP component 116 can determine that the request does not match the request identifier in action list 140. On the other hand, when the semantic distance is determined to meet a threshold distance (e.g., less than or equal to a threshold distance), NLP component 116 can determine that the request does indeed match the request identifier of action list 140. By comparison, action processor component 120 can determine that the request does not match the request identifier of action list 140. In response to this determination, action processor component 120 can identify the next action list 140 in the set maintained on data repository 128 and can repeat the comparison.

[0087] Conversely, by comparison, motion processor component 120 can determine that a request matches the request identifier of action list 140. In response to this determination, motion processor component 120 can select action list 140 for the action corresponding to the request. Because, as described above, multiple requests can correspond to the same action list 140, motion processor component 120 can select action list 140 that was previously or will be selected for another request. Before selecting action list 140, motion processor component 120 can also compare one or more parameters identified from the input audio signal with the input variables of action list 140. Motion processor component 120 can identify input variables from the parameter mapping or address template of action list 140. When comparing the identified parameters with the input variables of action list 140, motion processor component 120 can exclude auxiliary parameters indicated in action list 140. Motion processor component 120 can determine that the identified parameters match the input variables of action list 140 based on the set of allowed values ​​for each input variable. In response to this determination, the action processor component 120 may select an action list 140 for an action corresponding to a request defined by one or more parameters. The action processor component 120 may also stop iterating through the set of action lists 140 in the data repository 128. Otherwise, the action processor component 120 may determine that the identified parameters do not match the input variables of the action list 140. The action processor component 120 may identify another action list 140 for an action corresponding to the same request, and may repeat the comparison. After comparing with all action lists 140, the action processor component 120 may determine that the identified parameters do not match the input variables in any action list 140 of the agent application 110. In response to this determination, the action processor component 120 may select a preset action list 140 for the agent application 110 to respond to the request.

[0088] The motion processor component 120 can also search for action lists 140 from the data repository 128 using a proxy identifier identified from the input audio signal. Before performing a search using a request, the motion processor component 120 can identify a subset of action lists 140 that have application identifiers that match the proxy identifier. The proxy identifier may correspond to one of the proxy applications 110. Each action list 140 in the identified subset may include an application identifier determined to match the proxy identifier identified from the input audio signal. To identify a subset, the motion processor component 120 can compare the proxy identifier with the application identifier of each action list 140. When the proxy identifier is determined to match the application identifier of the action list 140, the motion processor component 120 can include the action list 140 in the subset. Otherwise, when the proxy identifier is determined not to match the application identifier of the action list 140, the motion processor component 120 can exclude the action list 140 from the subset. The motion processor component 120 can iterate through the identified subset of action lists 140 for the application identifier used with the request to select the action list 140 corresponding to the request.

[0089] When an input audio signal is determined to lack a proxy identifier, the action processor component 120 can select an action list 140 from a set maintained on the data repository 128 based on a request, one or more parameters, and proxy usage history. The proxy usage history can indicate usage statistics of a proxy application 110 running on a client device 104 that receives input audio signals from it. The proxy usage history can also indicate the functionality of the proxy application 110 invoked on the client device 104. The proxy usage history can be limited to a defined time window before a request is identified from the input audio signal detected on the client device 104. The action processor component 120 can identify a subset of the action list 140 using a request identifier that matches the request and one or more parameters. This subset can include action lists 140 with different application identifiers corresponding to multiple proxy applications 110. The action processor component 120 can use the proxy usage history of the client device 104 to select action lists 140 from the subset. The action processor component 120 can identify at least one proxy identifier from the proxy usage history. Action processor component 120 can compare the agent identifier with the application identifier of each action list 140 in the identified subset. Through comparison, action processor component 120 can determine that the application identifier of action list 140 matches an agent identifier from the agent usage history. In response to this determination, action processor component 120 can select an action list 140 for the action to be performed by agent application 110 based on parameters. Conversely, through comparison, action processor component 120 can determine that the application identifier of action list 140 does not match an agent identifier from the agent usage history. In response to this determination, action processor component 120 can identify another action list 140 and repeat the comparison using the agent identifier.

[0090] By recognizing action list 140, action processor component 120 can verify one or more parameters identified from the input audio signal by referring to the address template of action list 140. Verification by action processor component 120 may involve checking whether the necessary parameters specified in action list 140 exist to perform the action corresponding to the request. For example, for a ride-sharing request, action processor component 120 may check whether a parameter corresponding to the destination parsed from the input audio signal exists. Action processor component 120 may map parameters to the address template of action list 140 during parameter verification. Action processor component 120 can identify the correspondence between parameters and input variables defined in the mapping for action list 140. Action processor component 120 can identify a subset of correspondences for input variables not specified as auxiliary variables through the mapping of action list 140 to the address template.

[0091] For each input variable not indicated as auxiliary, action processor component 120 can determine whether at least one parameter corresponds to an input variable specified by the mapping. The determination of correspondence can be performed using a semantic analysis algorithm (e.g., a semantic knowledge graph) provided by NLP component 116. For each input variable, action processor component 120 can invoke NLP component 116 to determine whether the input variable corresponds to any parameter, and vice versa. In response to determining that all input variables correspond to at least one parameter, action processor component 120 can determine that the parameter has been successfully validated. Action processor component 120 can also continue to use the address template of action list 140 to generate an address to implement the action corresponding to the request.

[0092] On the other hand, in response to determining that at least one input variable (not indicated as auxiliary) does not correspond to any parameter, the action processor component 120 can determine that the parameter has not been successfully validated. The action processor component 120 can stop or terminate further processing of the request and parameter (e.g., generating an address using the address template of action list 140). The action processor component 120 can detect or determine an error in validating one or more parameters identified from the input audio signal. The digital assistant application 108 can (e.g., using visual content items or audio) present an indication that the request is invalid or unsuccessful. The action processor component 120 can provide this error indication to the agent service 106 or the administrator of the agent application 110 associated with action list 140.

[0093] After identifying Action List 140, Action Processor Component 120 can determine whether Client Device 104 has been authenticated by the proxy application 110 associated with Action List 140. Action Processor Component 120 can identify the proxy application 110 associated with Action List 140. Once identified, Action Processor Component 120 can use the proxy application 110 to check the authentication status of Client Device 104. The authentication status check can be performed using the account identifier of Client Device 104 or Digital Assistant Application 108. Action Processor Component 120 can send a request for authentication status to a proxy service 106 that manages the resources of proxy application 110. This request can include the account identifier of Client Device 104 or Digital Assistant Application 108. Proxy service 106 can provide Action Processor Component 120 with an indication of whether Client Device 104 has been authenticated by proxy application 110. When the response indicates that Client Device 104 has been authenticated, Action Processor Component 120 can determine that Client Device 104 has been authenticated by proxy application 110. Action Processor Component 120 can also continue processing the request. On the other hand, when the response indicates that client device 104 is not authenticated, action processor component 120 can determine that client device 104 has not been authenticated by proxy application 110. Action processor component 120 can stop or terminate further processing of the request (e.g., generate an address using the address template of action list 140). Digital assistant application 108 can also present instructions to authenticate using proxy application 110 to perform an action corresponding to the request. For example, digital assistant application 108 can render a prompt for logging into proxy application 110 on display 148 of client device 104. This prompt may include fields for the user's account identifier and password.

[0094] Additionally, the action processor component 120 can also determine whether the proxy application 110 associated with the action list 140 is installed on the client device 104. The action processor component 120 can identify the proxy application 110 associated with the action list 140. Once identified, the action processor component 120 can access the client device 104 to determine whether the proxy application 110 is installed on the client device 104. For example, the action processor component 120 can access a list of applications installed on the client device 104 to determine if the proxy application 110 is installed. The action processor component 120 can check the list for application identifiers or proxy identifiers. In response to determining that the proxy application 110 is installed, the action processor component 120 can continue processing the request. On the other hand, in response to determining that the proxy application 110 is not installed, the action processor component 120 can terminate processing of the request (e.g., generate an address using the address template of the action list 140). The digital assistant application 108 can also present instructions to install the proxy application 110 to perform the action corresponding to the request. For example, digital assistant application 108 may render a prompt indicating that the corresponding agent application 110 is not installed on the display 148 of client device 104. The prompt may also include a link for installing agent application 110.

[0095] The action list 140 selected by the action processor component 120 may correspond to a proxy application 110 that is not running as one of the applications in the foreground process 150 or the non-foreground process 152. For example, action list 140 may correspond to a proxy application 110 installed on the client device 104 but not running on the client device 104. The action list 140 selected by the action processor component 120 may correspond to a proxy application 110 running in the non-foreground process 152. For example, another application (e.g., another proxy application 110 or digital assistant application 108) may run in the application of the foreground process 150. The application in the foreground process 150 may have a graphical user interface rendered on at least a portion of the display 148 of the client device 104. Conversely, the application running in the non-foreground process 152 may not have a graphical user interface rendered on the display 148 of the client device 104 and may execute routines (e.g., actions). Additionally, the action list 140 selected by the action processor component 120 may correspond to a proxy application 110 running in the foreground process 150. For example, the agent application 110 corresponding to action list 140 may have a graphical user interface currently rendered on the display 148 of the client device 104.

[0096] Action processor component 120 can generate an address based on the address template of action list 140. Action processor component 120 can recognize the address template of action list 140 selected from a set maintained in data repository 128. As described above, the address template of action list 140 may include: a first part referencing proxy application 110; and a second part referencing the action to be performed and one or more input variables for performing the action. For example, the address template may be a URI template. The first part may include the scheme referencing proxy application 110 or the hostname referencing proxy service 106 or proxy application 110. The second part may include the pathname referencing the action and one or more query strings corresponding to one or more input variables in the URI. For example, for a request to check the number of steps taken, the address template of action list 140 may be "https: / / ex_personalmetrics.example.com / checksteps{?username}". The first part may correspond to "https: / / ex_personalmetrics.example.com / ", and the second part may correspond to "checksteps{?username}".

[0097] The address generated from the address template may include a first substring and a second substring. The first substring of the generated address may correspond to the first part of the address template. The generated address may be part of a URI, and the first substring may include a scheme referencing proxy application 110 or a hostname referencing proxy service 106 or proxy application 110. This scheme and hostname may be adopted by action processor component 120 from the first part of the address template. The second substring of the generated address may correspond to the second part of a second template. The generated address may also be part of a URI, and the second substring may include a pathname referencing an action to be performed by proxy application 110 and a query string including parameters identified from the input audio signal. For example, the address generated from the address template may be “https: / / ex_personalmetrics.example.com / checksteps?username=example_user_id”. The first substring of the address may include “https: / / ex_personalmetrics.example.com”, and the second substring of the address may include “checksteps?username=example_user_id”.

[0098] When generating an address, the action processor component 120 can use or set a first portion of the address template as a first substring of the address to reference the proxy service 106 or the proxy application 110 associated with the proxy service 106. The action processor component 120 can use a second portion of the address template as a second substring of the address to correspond to an action and the parameters used to perform that action. The action processor component 120 can populate the input variables of the second portion of the address template with one or more parameters identified from the input audio signal.

[0099] Action processor component 120 can populate the address template according to the parameter mapping of action list 140 defined for the address template. As discussed above, the parameter mapping can define the correspondence between the input variables of the second part and the words to be identified from the parsed input audio signal. Action processor component 120 can use the parameters as words identified from the input audio signal to insert them into the second part of the address template when populating the input variables. The population of the second part of the address template can be expanded according to the expression in the URI template. For each input variable specified in the address template, action processor component 120 can identify the corresponding parameter specified as in the mapping of action list 140. Through this identification, action processor component 120 can insert, replace, or set the query string with the name of the input variable and the identified parameter. For example, in the address template, the input variable could be "{?username}", and the identified parameter could be "example_user_id". In this example, action processor component 120 can replace "?username=example_user_id" in the second substring of the address according to the expression expansion of the URI template.

[0100] When populating the input variables of an address template, motion processor component 120 can determine whether the input variable is an auxiliary input variable as specified in the mapping of the address template in action listing 140. In response to determining that the input variable is an auxiliary input variable, motion processor component 120 can also determine whether there is an identified parameter corresponding to the auxiliary input variable. When it is determined that there is no parameter corresponding to the auxiliary input variable, motion processor component 120 can omit or remove the auxiliary input variable from the address. Therefore, the generated address may not have one or more parameters corresponding to the auxiliary input variable in the address template. Otherwise, when it is determined that there is a parameter corresponding to the auxiliary input variable, motion processor component 120 can insert or populate the auxiliary input variable with that parameter. Conversely, in response to determining that the input variable is not an auxiliary input variable, motion processor component 120 can identify the parameter to be included in the input variable from the identified parameters.

[0101] By generating an address based on an address template, the action processor component 120 can verify the address. Address verification by the action processor component 120 can include checking whether the address is formatted correctly. For verification, the action processor component 120 can check the address against a regular expression 130 (e.g., a URI or URL). The regular expression 130 can include, for example, a scheme for a valid address, a hostname, a pathname, and a query string separated by delimiters. The action processor component 120 can compare the address with the regular expression 130 to determine whether the address matches the pattern specified by the regular expression 130. In response to the determination that the address matches, the action processor component 120 can determine that the generated address has been successfully verified.

[0102] Conversely, in response to determining an address mismatch, action processor component 120 can determine that the generated address has not been successfully validated. Based on this determination, action processor component 120 can also detect errors during the validation of the generated address. This error may indicate that the address format is incorrect. Furthermore, action processor component 120 can stop or terminate further processing of the address (e.g., generating action data structures). Action processor component 120 can provide this error indication to the administrator of agent service 106 or agent application 110 associated with action list 140.

[0103] The motion processor component 120 can use an address for performing an action to package or generate an action data structure. The generation of the action data structure can be based on an application programming interface (API). For example, the API can be specified by the operating system running on the client device 104 and can be used by both the digital assistant application 108 and the agent application 110. The action data structure can be passed between the digital assistant application 108 and the agent application 110 via the API. The generation of the action data structure can be based on the Hypertext Transfer Protocol (HTTP). For example, the action data structure generated by the motion processor component 120 can be an HTTP request (e.g., a GET or POST function). The action data structure can include at least one header and at least one body. The header can include the device modality (e.g., smartphone, smart speaker, tablet, laptop, and desktop) of the client device 104 executing the digital assistant application 108 and the agent application 110. The header can include a source address set to the client device 104. The header can include a generated address set to the destination address. The body of the action data structure can include other information related to the request identified from parsing the input audio signal. For example, the subject may include an identifier that refers to the digital assistant application 108 or the client device 104 running the digital assistant application 108.

[0104] An instance of the digital assistant application 108 on the data processing system 102 or client device 104 can execute the proxy interface component 126 to provide an action data structure to the proxy application 110. By generating the action data structure, the proxy interface component 126 can provide action data to or direct action data to the proxy application 110. The proxy interface component 126 can send the action data structure to the proxy service 106 that manages the resources of the proxy application 110 via network 114 (e.g., as an HTTP request). For example, an HTTP request corresponding to the action data structure can be sent by the proxy interface component 126 via network 114. The proxy interface component 126 can use the action data structure (e.g., via an API) to invoke the proxy application 110. For example, the proxy interface component 126 can make a function call to facilitate the invocation of the proxy application 110 according to the API and by the operating system of the client device 104. During the invocation, the proxy interface component 126 can pass the action data structure to the proxy application 110.

[0105] Data processing system 102, client device 104, or proxy service 106 can execute an instance of proxy application 110. Since the destination address of the data structure can reference proxy application 110 or proxy service 106, proxy application 110 can receive action data structures from proxy interface component 126. For example, the action data structure may have been sent via network 114, and the destination address may include the hostname referencing proxy application 110 or proxy service 106. Because the destination address of the action data structure references proxy application 110 or proxy service 106, the action data structure can be routed to proxy application 110 or proxy service 106 via network 114. In another example, the destination address of the action data structure could be a deep link to a scheme referencing proxy application 110. By invoking the destination address, the action data structure can be directed by client device 104 or network 114 to the proxy application 110 referenced by that scheme.

[0106] When transmitting an action data structure, proxy application 110 can parse the action data structure to identify an address. Address identification can come from the destination address header of the action data structure. Proxy application 110 can parse the address to identify a first substring of the address, which references proxy application 110 itself or a proxy service 106 that manages the resources of proxy application 110. Proxy application 110 can parse a second substring of the address. Proxy application 110 can further parse the second substring of the address to identify the action and one or more parameters used to perform the action. Proxy application 110 can parse the pathname of the URI to identify the action, and parse one or more query strings to identify one or more parameters used to perform the action. For example, proxy application 110 can parse the address... The agent application 110 identifies the second substring "checksteps?username=example_user_id". It can further identify "checksteps" as a request to retrieve the measured steps and "Example_user_id" as the name of the user whose steps were measured. Using this identification, the agent application 110 can perform, conduct, or execute actions based on one or more parameters. Using the previous example, the agent application 110 can execute a series of processes and routines to retrieve the number of steps taken by the user of the agent application 110 with the account identifier "example_user_id". The agent application 110 can then provide the result of the action to the agent interface component 126.

[0107] Agent application 110 can also generate output indicating whether the action was executed successfully or failed, to be sent to digital assistant application 108. While executing the agent, agent application 110 may fail to complete the action or terminate before completion. For example, when processing a request for steps, agent service 106 may determine that information on the number of steps taken by the corresponding user cannot be retrieved. Based on this determination, agent application 110 may create output indicating failure. In response to this failure, agent application 110 may generate output indicating failure. Conversely, agent application 110 may complete the execution of the action. In response to this completion, agent application 110 may generate output indicating success. Agent application 110 may provide the output of the action execution to agent interface component 126.

[0108] By passing the action data structure, the agent interface component 126 can control whether the agent application 110 runs as a non-foreground process 152 or as a foreground process 150. The setting or control of the foreground process 150 or non-foreground process 152 can be based on the interface mode indicator used to generate the action list 140 of the action data structure. The settings of the foreground process 150 and non-foreground process 152 by the agent interface component 126 can be facilitated by the operating system running on the client device 104. As discussed above, the agent application 110 can run as one of the applications in the foreground process 150 or non-foreground process 152, or as one of the applications in neither the foreground process 150 nor the non-foreground process 152. Through the passing of the action data structure, the agent interface component 126 can control whether the agent application 110 is set from a non-foreground process 152 to a foreground process 150.

[0109] Furthermore, the proxy interface component 126 can identify which applications are running in the foreground process 150 and as non-foreground processes 152, such as the digital assistant application 108 and the proxy application 110 installed on the client device 104. For example, the proxy interface component 126 can access the operating system of the client device 104 to identify applications running as foreground processes 150 and applications running as non-foreground processes 152. The proxy interface component 126 can identify whether the proxy application 110 (or other applications on the client device 104) is one of the foreground processes 150 or non-foreground processes 152. The proxy interface component 126 can identify the proxy application 110 as running as one of the applications in the foreground process 150. The proxy interface component 126 can identify the proxy application 110 as running as one of the applications in non-foreground processes 152. The proxy interface component 126 can identify the proxy application 110 as running as one of the applications that is neither in the foreground process 150 nor non-foreground processes 152. In response to determining that the proxy application 110 is neither running as a foreground process 150 nor as a non-foreground process 152, the proxy interface component 126 can enable the client device 104 to initiate the execution of the proxy application 110.

[0110] The proxy interface component 126 can identify an interface mode indicator from the selected action list 140. The interface mode indicator can specify that the user interface component 142 of the proxy application 110 should be rendered on the display 148 of the client device 104 when the action corresponding to action list 140 is executed. When the interface mode indicator specifies that no user interface component 142 should be rendered, the proxy interface component 126 can set the proxy application 110 as a foreground process 150. When the proxy application 110 is identified as not executing on the client device 104, the proxy interface component 126 can start the proxy application 110 in the foreground process 150. When the proxy application 110 is identified as running in a non-foreground process 152, the proxy interface component 126 can remove the application (e.g., another proxy application 110 or digital assist application 108) that was initially in the foreground process 150 and set it as a non-foreground process 152. Furthermore, the proxy processor component 126 can set the proxy application 110 from a non-foreground process 152 to a foreground process 150. When the proxy application 110 is identified as already in the foreground process 150, the proxy interface component 126 can keep the proxy application 110 in the foreground process 150.

[0111] When the interface mode indicator specifies that user interface component 142 should be rendered, proxy interface component 126 may apply at least one authorization policy of proxy application 110. This authorization policy may specify which applications the output of proxy application 110 is restricted from being rendered with. For example, an authorization policy for a personal fitness application may specify that its output is restricted from being rendered with another personal fitness application. The authorization policy may include a list (also referred to herein as a blacklist) of applications with which the output of proxy application 110 is restricted from being rendered simultaneously. The authorization policy may also include a list (also referred to herein as a whitelist) of applications with which the output of proxy application 110 is allowed to be rendered simultaneously. Each list may include a proxy identifier (e.g., application filename) corresponding to the included application. The authorization policy may be specific to the output type of proxy application 110. The output type may depend on the request identified from parsing the input audio signal. For example, when another personal fitness application is in the foreground, the authorization policy may restrict the display of steps but may allow the display of account credentials information.

[0112] The determination of whether to apply an authorization policy can be based on whether the proxy application 110 is running in the foreground process 150 or in the non-foreground process 152. When the proxy application 110 is identified as being in the foreground process 150, the proxy interface component 126 can determine that an authorization policy should not be applied. The proxy interface component 126 can also allow the user interface component 142 of the proxy application 110 to render the output on the display 148 of the client device 104. When another application besides the proxy application 110 is identified as being running in the foreground process 150, the proxy interface component 126 can determine that an authorization policy should be applied. In this way, the proxy application 110 can remain part of the non-foreground process 152 or not execute on the client device 104. In determining whether to apply an authorization policy, the proxy interface component 126 can also identify the authorization policy based on the request parsed from the input audio signal. The proxy interface component 126 can compare the identified request with the output type of the authorization policy. Upon determining a match, the proxy interface component 126 can determine that the authorization policy specified for the corresponding output type should be applied.

[0113] Based on the authorization policy, the proxy interface component 126 can determine whether the output of the proxy application 110 is authorized to be rendered along with the application's graphical user interface components in the foreground process 150. The proxy interface component 126 can identify the application in the foreground process 150. The proxy interface component 126 can identify the proxy identifier of the application in the foreground process 150. The proxy interface component 126 can compare the application with the list of applications specified in the authorization policy of the proxy application 110. During the comparison, the proxy interface component 126 can use the identified proxy identifier to search the whitelist and blacklist of the authorization policy of the proxy application 110. If the application in the foreground process 150 matches one of the applications in the whitelist, the proxy interface component 126 can determine that the output of the proxy application 110 is allowed to be rendered along with the application. On the other hand, if the application in the foreground process 150 matches one of the applications in the blacklist, the proxy interface component 126 can determine that the output is restricted from being rendered along with the application.

[0114] When it is determined that the output of proxy application 110 is permitted, proxy interface component 126 can identify user interface component 142 of proxy application 110. To identify user interface component 142, proxy interface component 126 can use an address generated according to an address template of action list 140. For example, an address generated according to an address template can reference an action performed by proxy application 110 and the user interface component 142 to be rendered when that action is performed. An address generated according to an address template can reference proxy application 110 or proxy service 106 that manages the resources of proxy application 110. Proxy interface component 126 can identify a separate address or identifier for user interface component 142 from action list 140. The address can reference user interface component 142 maintained by proxy service 106. The address and identifier can reference user interface component 142 maintained on data repository 128. Using a separate address or identifier, proxy interface component 126 can identify the user interface component 142 that will be rendered along with the action performed by proxy application 110.

[0115] After identification, proxy interface component 126 can access or retrieve user interface component 142. Using an address generated according to the address template in action list 140, proxy interface component 126 can retrieve user interface component 142. Proxy interface component 126 can send a request for user interface component 142 to proxy service 106, which includes the address. Since the address references proxy service 106, the request can be directed to proxy service 106, which handles resources of proxy application 110. Proxy service 106 can then provide a response with the referenced user interface component 142 to proxy interface component 126. Using the address included in action list 140, proxy interface component 126 can retrieve user interface component 142. The address included in action list 140 can be separate from the address generated using the address template and can reference proxy service 106 or data repository 128. Proxy interface component 126 can send a request for user interface component 142, which includes the address. When the address references proxy service 106, the request can be directed to proxy service 106, which handles resources of proxy application 110. When an address references data repository 128, a request can be directed to data repository 128. Proxy service 106 or data repository 128 can then provide a response with the referenced user interface component 142 to proxy interface component 126. Using the identifier included in action list 140, proxy interface component 126 can retrieve user interface component 142 from data repository 128. The identifier may correspond to an index of user interface component 142 maintained on data repository 128. Proxy interface component 126 can use the identifier to search data repository 128 for user interface component 142. Once identified, proxy interface component 126 can retrieve user interface component 142 from data repository 128.

[0116] Before rendering user interface component 142, proxy interface component 126 can also apply the authorization policy of the application in foreground process 150. The authorization policy can specify which applications a proxy's graphical user interface component in foreground process 150 is restricted from or allowed to be rendered with. For example, the authorization policy of a banking application can specify that its output is restricted from being rendered with another banking application. The authorization policy can include a list (also referred to herein as a blacklist) of applications whose graphical user interfaces are restricted from being rendered simultaneously with the application in foreground process 150. The authorization policy can also include a list (also referred to herein as a whitelist) of applications whose graphical user interfaces are allowed to be rendered simultaneously with the application in foreground process 150. Each list can include a proxy identifier (e.g., application filename) corresponding to the included application.

[0117] Based on the authorization policy, the proxy interface component 126 can determine whether the graphical user interface components of the application in the foreground process 150 are authorized to be rendered simultaneously with the output of the proxy application 110. The proxy interface component 126 can identify the proxy application 110 performing this action. The proxy interface component 126 can identify the proxy identifier of the proxy application 110. The proxy interface component 126 can compare the application with the list of applications specified in the authorization policy of the application in the foreground process 150. During the comparison, the proxy interface component 126 can use the identified proxy identifier to search the whitelist and blacklist of the authorization policy of the application in the foreground process 150. If the proxy application 110 matches one of the applications in the whitelist, the proxy interface component 126 can determine that simultaneous rendering with the graphical user interface of the application in the foreground 150 is allowed. On the other hand, if the proxy application 110 matches one of the applications in the blacklist, the proxy interface component 126 can determine that simultaneous rendering is restricted.

[0118] By recognizing the user interface component 142, the proxy interface component 126 can render the user interface component 142 of the proxy application 110 on the display 148 of the client device 104. By allowing simultaneous rendering, the user interface component 142 of the proxy application 110 can be simultaneously displayed on the display 148 along with the graphical user interface of the application running in the foreground process 150. The proxy interface component 126 can render the user interface component 142 as an overlay on top of the graphical user interface of the application in the foreground process 150. The proxy interface component 126 may also include the user interface component 142 as part of the graphical user interface of the application in the foreground process 150. For example, the application running as the foreground process 150 could be a digital assistant application 108, and could have a graphical user interface including a dialogue with a user operating the digital assistant application 108 on the client device 104. The proxy interface component 126 can insert the user interface component 142 into the dialogue of the graphical user interface of the digital assistant application 108.

[0119] The proxy interface component 126 can add or insert information into the user interface component 142 rendered on the display 148 of the client device 104. The proxy interface component 126 can add, insert, or include words or requests from the input audio signal into the corresponding interface elements of the user interface component 142. The NLP component 116 can identify the words or requests by parsing the input audio signal. For example, the proxy interface component 126 can merge words with requests into the text boxes of the user interface component 142. The proxy interface component 126 can retrieve, receive, or identify output generated by the proxy application 110. The proxy interface component 126 can retain the rendering of the user interface component 142 and can wait for output from the proxy application 110. After receiving output from the proxy application 110, the proxy interface component 126 can add, insert, or include the output into the corresponding interface elements of the user interface component 142. For example, the proxy interface component 126 can receive output indicating the number of steps taken by the user of the proxy application 110 installed on the client device 104. The proxy interface component 126 can insert the step number into another text box in the user interface component 142. As the output is received and inserted, the proxy interface component 126 can render the user interface component 142 on the display 148.

[0120] When user interface component 142 is rendered on display 148 along with the graphical user interface of the application in foreground process 150, proxy interface component 126 can monitor user interaction events with user interface component 142. Interaction events can include, for example, hovering, clicking, scrolling, swiping, key presses, and screen touches. Upon detecting a user interaction event on user interface component 142, proxy interface component 126 can process the user interaction event to update user interface component 142. Proxy interface component 126 can relay the detected user interaction event to proxy application 110. Proxy application 110 can process the user interaction event (e.g., using an event handler) and can generate additional output in response to the user interaction event. Proxy application 110 can provide the output to proxy interface component 126. Using the output received from proxy application 110, proxy interface component 126 can update user interface component 142.

[0121] In this way, proxy interface component 126 can maintain the application initially in foreground process 150 while adding user interface component 142 from proxy application 110 to be displayed as an overlay. Proxy interface component 126 can have either the original application or proxy application 110, both of which can serve as applications in foreground process 150. Proxy interface component 126 can also maintain visual components from both applications rendered on display 148 of client device 104, thereby reducing user execution context switching on client device 104. By retaining both simultaneously, proxy interface component 459 can also reduce computational resources generated during rendering switching and between foreground process 150 and non-foreground process 152.

[0122] Conversely, when it is determined that the output of proxy application 110 is restricted, proxy interface component 126 can set proxy application 110 as foreground process 150. When proxy application 110 is identified as not running on client device 104, proxy interface component 126 can invoke and launch proxy application 110 (e.g., via the operating system of client device 104). Proxy interface component 126 can move the application that was initially running as foreground process 150 and can set the application to run as non-foreground process 152. When switching between foreground process 150 and non-foreground process 152, proxy interface component 126 can remove the graphical user interface components of the application that were initially in foreground process 150 and are rendered on display 148. Furthermore, proxy interface component 126 can invoke proxy application 110 using motion data structures generated by motion processor component 120. Instead of rendering user interface component 142, proxy interface component 126 can render the graphical user interface of proxy application 110 on display 148 of client device 104.

[0123] Running as a foreground process 150, agent application 110 can display the output of the action indicated in the address of the action data structure on the graphical user interface rendered in display 148. Agent application 110 can monitor user interaction events with the graphical user interface. When a user interaction event is detected on user interface component 142, agent application 110 can process the user interaction event (e.g., using an event handler) and can generate another output in response to the user interaction event. Agent application 110 can use the generated output to update the graphical user interface rendered in display 148 of client device 104.

[0124] By controlling the switching between foreground process 150 and non-foreground process 152, user involvement in the search agent application 110 can be eliminated, thereby improving HCI with applications on client device 104. Additionally, digital assistant application 108 can adjust and reduce the consumption of computing resources on client device 104 due to switching.

[0125] Figure 2 Describing the use of in Figure 1 The diagram shows a sequence of example data streams 200 for rendering visual components in the system 100. Data stream 200 can be composed of the above combinations. Figure 1 The system described is 100 or in combination with the following Figure 5 The system 500 described in detail is implemented or executed. The data stream 200 may include communication in the form of packets (e.g., HTTP messages) between the microphone 150, the client device 104, the data processing system 102, the proxy service 106 (via the network 114), and the display 148.

[0126] A local instance of a digital assistant application 108 running on client device 104 can receive audio data 205 (e.g., an input audio signal) from a microphone 146 communicatively coupled to client device 104. Digital assistant application 108 can execute on client device 104 in a non-foreground process 152, without any visual components rendered on display 148. Client device 104 may have another application in a foreground process 150, with a graphical user interface rendered on display 148. Digital assistant application 108 on client device 104 can perform initial processing (e.g., automatic speech recognition) on the audio data 205 to identify one or more words from the audio data 205. Digital assistant application 108 can provide one or more identified words from the audio data 205 as input 210 to a remote instance of digital assistant application 108 running on data processing system 102. NLP component 116 on data processing system 102 can then further process the one or more words of input 210 to identify a request and one or more parameters. The request can instruct agent application 110 to perform an action based on one or more parameters.

[0127] The action processor component 120 on data processing system 102 can use the request and one or more parameters to identify an action list 140 from data repository 128. The agent application 110 associated with the identified action list 140 may not execute on client device 104, or may run as one of the applications not in the foreground process 152. The action processor component 120 can generate an address based on the address template of the identified action list 140. The address may include a first substring (e.g., scheme or hostname) referencing the agent application to perform the action. The address may also include a second string corresponding to the action to be performed and one or more parameters on which the action is based. The action processor component 120 can package the address into an action data structure to be provided as input 215 to the agent service 106 of agent application 110.

[0128] Upon receiving input 215, the agent service 106 of agent application 110 can parse the action data structure to identify a first substring and a second substring. While processing input, client device 104 can maintain the application running as a foreground process 150 and a non-foreground process 152. Agent application 110, executing on agent service 106, can further parse the second substring to identify the action and one or more parameters. Based on this identification, agent application 110 can execute an action according to the parameters. While executing the action, agent application 110 can generate output 220 to provide to client device 104 via data processing system 102. Agent service 106 can provide output 220 to data processing system 102. Digital assistant application 108 on data processing system 102 can send output 220 as output 225 to client device 104.

[0129] In combination, proxy interface component 126 can determine that action list 140 specifies that user interface component 142 will be rendered when the action is performed. In response to this determination, proxy interface component 126 can identify user interface component 235 from the set of user interface components 142 (e.g., using an address generated using the address template of action list 140). Proxy interface component 126 can provide user interface component 235 to client device 104. Using user interface component 235, client device 104 can display rendered data 240 for use on display 148 of client device 104. Rendered data 240 may include user interface component 235 as an overlay on the graphical user interface of the application overlaid in foreground process 150. Client device 104 can maintain the application in foreground process 150 and non-foreground process 152 until audio data 205 is received from microphone 146.

[0130] Figure 3 It shows in Figure 1The diagram illustrates a use case diagram of an example client device 104 that displays input messages and provides user interface components in system 100. Client device 104 may have local instances of an application running as a foreground process 150 and a digital assistant application 108 running as a non-foreground process 152. The application running as foreground process 150 may have a graphical user interface 305 rendered on the display 148 of client device 104. As shown, the graphical user interface 305 may be used for a web browser application running as foreground process 150. In this case, the NLP component 116 of the digital assistant application 108 running on client device 104 may receive input audio signals via the microphone 146 of client device 104. The NLP component 116 may apply automatic speech recognition to recognize one or more words 310 from the input audio signal. For example, the NLP component 116 may recognize the word 310 “How many steps have I taken today?” from the input audio signal.

[0131] The NLP component 116 on client device 104 or data processing system 102 can perform further processing on word 310. Through further processing, NLP component 116 can identify a request from word 310 recognized from the input audio signal. In the depicted example, NLP component 116 can identify the request as a request to obtain the number of steps taken by a user operating client device 104 or associated with client device 104. Using the identified request, motion processor component 120 can identify agent application 110 capable of executing the request. Motion processor component 120 can also select an action list 140 from data repository 128. Action list 140 can be an action corresponding to the request to be performed by agent application 110. Using action list 140, motion processor component 120 can invoke agent application 110 to perform the action corresponding to the request.

[0132] In summary, proxy interface component 126 can recognize that user interface component 142 will be displayed when an action in action list 140 is performed. Action list 140 can specify whether user interface component 142 should be displayed and can identify which user interface component 142 should be presented. In response to this recognition, proxy interface component 126 can determine whether simultaneous presentation with the graphical user interface component 305 of the foreground application is allowed. This determination can be based on the authorization policy of proxy application 110. The authorization policy of proxy application 110 can specify which applications are allowed to be presented simultaneously with the user interface component 142 of proxy application 110. In the depicted example, proxy interface component 126 can determine that simultaneous presentation is allowed.

[0133] In response to this determination, the proxy interface component 126 can identify the user interface component 142 from the action list 140 to perform the action indicated by the word 310. After identification, the proxy interface component 126 can render the user interface component 142 as an overlay component 315 superimposed on top of the graphical user interface 305 of the foreground process 150. In the depicted example, the user interface component 142 rendered as the overlay component 315 may include the request word 310 "how many steps?" corresponding to the action. Additionally, the user interface component 142 may also have a result output from performing the action, such as "You have taken 3147 steps today". The user interface component 142 rendered as the overlay component 315 may also include a command button 320. When interacting with the command button 320, the proxy interface component 125 can close the overlay component 315, thereby causing the user interface component 142 to disappear from the rendering on the display 148 of the client device 104.

[0134] Figure 4 A block diagram of an example method 400 for rendering visual components in an application is shown. Method 400 can be combined with the above. Figures 1 to 3 The system described is 100 or below combined Figure 5 The system 500 described in detail is implemented or executed. In short, NLP component 116 can receive input audio signals (405). NLP component 116 can parse the input audio signals (410). Action processor component 120 can select action list 140 (415). Action processor component 120 can use action list 140 to generate addresses (420). Action processor component 120 can generate action data structures (425). Proxy interface component 126 can guide the action data structures (430). Proxy interface component 126 can determine whether user interface component 142 is authorized to be rendered with the foreground application (435). If it is determined to be authorized, proxy interface component 126 can render user interface component 142 (440). If it is determined to be unauthorized, proxy interface component 126 can call the proxy application 110 associated with the user interface component (445).

[0135] More specifically, NLP component 116 can receive input audio signals (405). NLP component 116 can receive one or more data packets, including input audio signals acquired at a sensor (e.g., microphone 146) on client device 104. For example, NLP component 116 of digital assistant application 108, which is at least partially executed by data processing system 102, can receive input audio signals. Input audio signals can include dialogue facilitated by digital assistant application 108. The dialogue can include one or more inputs and outputs. The dialogue can be audio-based, text-based, or a combination of audio and text. Input audio signals can include text input or other types of input that can provide dialogue information. When an input audio signal is received, client device 104 can make the application run as a foreground process 150 and a non-foreground process 152.

[0136] NLP component 116 can parse the input audio signal (410). By parsing the input audio signal, NLP component 116 can use various natural language processing techniques to identify a request and one or more parameters. The request can be an intent or request that can be fulfilled by digital assistant application 108 or agent application 110. Parameters can define the request. Based on the request, NLP component 116 can determine that the request corresponds to one of the functions of agent application 110.

[0137] Motion processor component 120 can select action list 140 (415). Based on a request identified from the input audio signal, motion processor component 120 can select action list 140 from a set maintained on data repository 128. For each action list 140, motion processor component 120 can compare the request with the request identifier of action list 140. This comparison can be performed using a semantic knowledge graph. Through this comparison, motion processor component 120 can determine that the request matches the request identifier of action list 140. In response to this determination, motion processor component 120 can select action list 140 from the set maintained on data repository 128. The selected action list can instruct user interface component 142 to be rendered for the agent application 110 that is not running as a foreground process 150.

[0138] Action processor component 120 can use action list 140 to generate an address (420). The address generation can be based on an address template in action list 140. This address template can have a first part and a second part. The first part can reference agent application 110 or agent service 106 for agent application 110 to perform the action. The second part can reference the action and can include one or more input variables for the action. Action processor component 120 can obtain the first and second parts from the address template when generating the address. Action processor component 120 can insert or populate parameters into the input variables of the second part of the address template.

[0139] Action processor component 120 can generate an action data structure (425). The action data structure may include a header and a body. Action processor component 120 may insert an address generated according to the address template of action list 140 into the header of the action data structure. Action processor component 120 may set the address as the destination address of the header to route the action data structure to proxy application 110 or proxy service 106 for proxy application 110.

[0140] The proxy interface component 126 can direct an action data structure (430). The proxy interface component 126 can direct the action data structure to the proxy application 110. The proxy interface component 126 can use the action data structure to invoke the proxy application 110 and can pass the action data structure to the proxy application 110. The proxy application 110 can then parse the action data structure to identify the action and one or more parameters from the address included in the action data structure. The proxy application 110 can then execute the action based on the parameters.

[0141] The proxy interface component 126 can determine whether the user interface component 142 is authorized to be rendered together with the foreground application (435). The determination of whether the user interface component 142 is authorized can be based on an authorization policy. The authorization policy can specify whether the output of the proxy application 110 is allowed or prohibited from being rendered simultaneously with the graphical user interface of the application running as the foreground process 150. The proxy interface component 126 can identify the application running in the foreground process 150. The proxy interface component 126 can compare the application with the specifications of the authorization policy. When the authorization policy allows simultaneous rendering, the proxy interface component 126 can determine that the user interface component 142 is authorized to be rendered. Otherwise, when the authorization policy restricts simultaneous rendering, the proxy interface component 126 can determine that the user interface component 142 is not authorized to be rendered together with the application running as the foreground process 150.

[0142] If authorized, the proxy interface component 126 can render the user interface component 142 (440). The proxy interface component 126 can identify the user interface component 142 from the selected action list 140 (e.g., using the address included therein). The proxy interface component 126 can render the identified user interface component 142 as an overlay on the graphical user interface of the application running in the foreground process 150. The proxy interface component 126 can include requests for actions and output generated by the proxy application 110 in the user interface component 142.

[0143] On the other hand, if it is determined that the authorization is unauthorized, the proxy interface component 126 can invoke the proxy application 110 (445) associated with the user interface component 142. The proxy interface component 126 can remove the original application running from the foreground process 150 and set the application as a non-foreground process 152, thereby removing the rendering of the graphical user interface of the application. The proxy interface component 126 can set the proxy application 110 as the foreground process 150. The proxy interface component 126 can also render the graphical user interface of the proxy application 110 on the display 148 of the client device.

[0144] Figure 5 This is a block diagram of an example computer system 500. The computer system or computing device 500 may include or be used to implement system 100 or its components (such as data processing system 102). The computing system 500 includes a bus 505 or other communication components for transmitting information and a processor 510 or processing circuitry coupled to the bus 505 to process information. The computing system 500 may also include one or more processors 510 or processing circuitry coupled to the bus to process information. The computing system 500 also includes a main memory 515, such as random access memory (RAM) or other dynamic storage devices, coupled to the bus 505 for storing information and instructions to be executed by the processor 510. The main memory 515 may be or may include a data repository 128. The main memory 515 may also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 510. The computing system 500 may further include a read-only memory (ROM) 520 or other static storage devices coupled to the bus 505 for storing static information and instructions for the processor 510. Storage device 525 (such as a solid-state device, disk, or optical disk) may be coupled to bus 505 for persistent storage of information and instructions. Storage device 525 may include data repository 128 or a portion thereof.

[0145] The computing system 500 can be coupled to a display 535 (such as a liquid crystal display or an active matrix display) via a bus 505 to display information to a user. An input device 530, such as a keyboard including alphanumeric and other keys, can be coupled to the bus 505 to transmit information and command selections to the processor 510. The input device 530 may include a touchscreen display 535. The input device 530 may also include cursor controls, such as a mouse, trackball, or arrow keys, to transmit directional information and command selections to the processor 510 and control cursor movement on the display 535. For example, the display 535 may be a data processing system 102, a client device 104, or... Figure 1 It is part of other components.

[0146] The processes, systems, and methods described herein can be implemented by a computing system 500 in response to a processor 510 executing instructions contained in main memory 515. Such instructions may be read into main memory 515 from another computer-readable medium, such as storage device 525. The arrangement for executing the instructions contained in main memory 515 causes the computing system 500 to perform the illustrative processes described herein. One or more processors in a multiprocessor arrangement may also be used to execute the instructions contained in main memory 515. Hardwired circuitry may be used in place of or in combination with software instructions in conjunction with the systems and methods described herein. The systems and methods described herein are not limited to any particular combination of hardware circuitry and software.

[0147] Although already Figure 5 The example computing system described herein can be implemented using other types of digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed herein and their equivalents), or in a combination of one or more of them (including the operations described herein).

[0148] In the context of systems discussed in this paper that collect or utilize personal information about users, the following opportunities can be provided to users: control whether programs or features collect user information (e.g., information about the user's social networks, social actions, or activities; user preferences; or information about the user's location) or control whether or how content that may be more relevant to the user is received from content servers or other data processing systems. Furthermore, specific data can be anonymized in one or more ways before storage or use, thereby removing personally identifiable information when generating parameters. For example, user identity can be anonymized to make it impossible to determine the user's personally identifiable information, or the user's geographic location information (such as city, postal code, or state / county level) can be generalized to make it impossible to determine the user's specific location. Thus, users can control how content servers collect and / or use information about them.

[0149] The subject matter and operations described herein can be implemented in digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed herein and their equivalents), or combinations thereof. The subject matter described herein can be implemented as one or more computer programs, i.e., one or more circuits encoding computer program instructions on one or more computer storage media for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof, or included therein. When the computer storage medium is not a propagating signal, it can be a source or destination of computer program instructions encoded in artificially generated propagating signals. A computer can also be one or more separate components or media (e.g., multiple CDs, disks, or other storage devices), or included therein. The operations described in this specification can be implemented as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.

[0150] The terms "data processing system," "computing device," "component," or "data processing equipment" encompass all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination thereof. The device may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may also include code that creates an execution environment for the computer program under consideration, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The device and execution environment can implement various computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures. Components of system 100 may include or share one or more data processing devices, systems, computing devices, or processors.

[0151] Computer programs (also known as programs, software, software applications, applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, declarative languages, or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs can correspond to files in a file system. A computer program can be stored as a part of a file that holds other programs or data (e.g., one or more scripts in a markup language document), or as a single file dedicated to the program in question, or as multiple collaborating files (e.g., a file storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0152] The processes and logic flows described herein can be performed by one or more programmable processors that execute one or more computer programs (e.g., components of data processing system 102) to perform actions by manipulating input data and generating outputs. Processes and logic flows can also be performed by special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)), and devices can also be implemented as special-purpose logic circuitry (e.g., FPGAs or ASICs). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROMs, and DVD-ROMs. Processors and memory can be supplemented by or incorporated into a special-purpose logic circuitry system.

[0153] The subjects described herein can be implemented in computing systems that include back-end components (e.g., as data processors), middleware components (e.g., application servers), front-end components (e.g., client computers with graphical user interfaces or web browsers through which users can interact with embodiments of the subjects described herein), or combinations of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0154] Computing systems such as System 100 or System 500 may include clients and servers. Clients and servers are generally geographically distant from each other and typically interact via a communication network (e.g., network 114). The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. In some implementations, the server sends data (e.g., data packets representing content items) to the client device (e.g., to display data to a user interacting with the client device and to receive user input from that user). Data generated at the client device (e.g., the result of user interaction) may be received from the client device at the server (e.g., received by data processing system 102 from client device 104 or proxy service 106).

[0155] Although the operations are depicted in a specific order in the accompanying drawings, it is not necessary to perform them in the specific order shown or in sequence, nor is it necessary to perform all the operations shown. The actions described herein can be performed in different orders.

[0156] Separation of system components is not required in all implementations, and the described program components may be included in a single hardware or software product. For example, NLP component 116, audio signal generator component 118, motion processor component 120, response selector component 122, proxy registry component 124, and proxy interface component 126 may be a single component, application or program, or a logical device having one or more processing circuits, or being part of one or more servers of data processing system 102.

[0157] Now that some illustrative embodiments have been described, it is apparent that the foregoing description is merely illustrative and not restrictive, and is presented by way of example. Specifically, while many of the examples presented herein relate to specific combinations of method actions or system elements, these actions and elements can be combined in other ways to achieve the same purpose. Actions, elements, and features discussed in connection with one embodiment are not intended to exclude similar effects in other embodiments.

[0158] The wording and terminology used herein are for descriptive purposes and should not be construed as restrictive. The use of “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof is intended to cover the items listed thereafter, their equivalents, other items, and alternative embodiments comprised of the items exclusively listed thereafter. In one embodiment, the systems and methods described herein consist of each or all of one or more of the described elements, actions, or components.

[0159] Any mention of an implementation, element, or action of a system or method referred to in the singular herein may also encompass an implementation that includes multiple such elements, and any mention of any implementation, element, or action referred to in the plural herein may also encompass an implementation that includes only one element. The singular or plural mentions are not intended to limit the currently disclosed systems or methods, their components, actions, or elements to a single or multiple configurations. A mention of any action or element based on any information, action, or element may include an implementation of that action or element that is at least partially based on any information, action, or element.

[0160] Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to “implementation,” “certain implementations,” “one implementation,” etc., are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with that implementation may be included in at least one implementation or embodiment. As used herein, such terms do not necessarily refer to the same implementation. Any implementation may be combined inclusively or exclusively with any other implementation in any manner consistent with the aspects and implementations disclosed herein.

[0161] A reference to “or” can be interpreted as inclusive, such that any term described using “or” can refer to a single term, more than one term, or any one of all the terms described. A reference to “at least one of 'A' and 'B'” can include only 'A', only 'B', and 'A' and 'B'. Such a reference used in conjunction with “includes” or other open-ended terms can include other items.

[0162] Technical features in the drawings, detailed descriptions, or any claims are followed by reference numerals, which are included to enhance the comprehensibility of the drawings, detailed descriptions, and claims. Therefore, the reference numerals, or their absence, do not limit the scope of any claim element.

[0163] The systems and methods described herein may be embodied in other specific forms without departing from their characteristics. The foregoing embodiments are illustrative and not limiting of the systems and methods described. Therefore, the scope of the systems and methods described herein is indicated by the appended claims rather than the foregoing description, and modifications within the equivalent meaning and scope of the claims are included therein.

Claims

1. A system for rendering a visual component on an application, comprising: A natural language processor executing on a data processing system having one or more processors, the natural language processor being configured to: receiving a data packet including an input audio signal detected by a sensor of a client device that displays a graphical user interface of a first application in a foreground process on the client device; as well as parsing the input audio signal of the data packets to identify a request; An action processor executed on the data processing system, for: an action list selecting to perform an action corresponding to the request by a second application, the second application being installed on the client device and not in the foreground process; as well as generating an action data structure according to the action list; as well as An agent interface executed on the data processing system, configured to: providing the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output; determining, based on an authorization policy of the second application, that the output from the second application performing the action is authorized to be presented with the graphical user interface of the first application; in response to determining that the output of the second application is authorized to be presented with the first application, identifying a user interface component of the second application for the action manifest; as well as The user interface component including the output from the second application is displayed on the client device along with a graphical user interface of the first application that is authorized to be presented with the second application.

2. The system according to claim 1, comprising: The action list has an agent identifier corresponding to the second application and a request identifier corresponding to the action; the natural language processor parsing the input audio signal of the data packet to identify an agent identifier corresponding to an agent, the agent corresponding to the second application installed on the client device; as well as The action handler selects the action list to perform the action having the agent identifier corresponding to the agent identifier identified from parsing the input audio signal.

3. The system according to claim 1, comprising the action processor, configured to: determining that an interface mode indicator of the action manifest specifies rendering of the user interface component when performing the action; and In response to determining that the interface mode indicator specifies rendering of the user interface component, the user interface component of the second application is identified for the action manifest.

4. The system according to claim 1, comprising the agent interface for: determining, for the action list for performing the action corresponding to the second request, that the client device is not authenticated by the second application; and In response to determining that the client device is not authenticated by the second application, presenting a prompt interface on the client device to authenticate the client device to the second application to perform the action corresponding to the second request.

5. The system of claim 1 , comprising the agent interface for providing the action data structure to the second application so that the second application: monitoring interaction events on the user interface component from the second application presented with the graphical user interface of the first application; and In response to detecting the interaction event, the interaction event is processed to update the user interface component.

6. The system according to claim 1, comprising: the natural language processor to receive the data packet via a digital assistant application, the data packet including the input audio signal detected by the sensor of the client device, the client device displaying the graphical user interface of the digital assistant application corresponding to the first application in the foreground process; as well as The agent interface is used to display within the graphical user interface of the digital assistant application a user interface component including a subcomponent indicating the request identified from parsing the input audio signal and a user interface component including the output from the agent corresponding to the second application.

7. The system according to claim 1, comprising: the natural language processor to receive the data packet, the data packet including the input audio signal detected by the sensor of the client device, the client device displaying a graphical user interface of a first agent corresponding to the first application in the foreground process; as well as The agent interface is configured to display a user interface component including the output from the second application as an overlay on a graphical user interface of the first agent.

8. The system of claim 1, comprising the action handler for providing the action data structure to the second application so that the second application: parsing the action data structure to identify an action to be performed; generating the output by performing the action identified from the action data structure; as well as The output is provided to the user interface component to be presented with a graphical user interface of the first application that is authorized to be presented with the second application.

9. The system according to claim 1, wherein: The action data structure includes an address for executing the action of the action list.

10. The system according to claim 9, wherein: The address includes a first substring corresponding to the second application.

11. The system according to claim 10, wherein: The address includes a second substring corresponding to the action.

12. The system according to claim 11, wherein: The second substring includes at least one input variable for performing the action.

13. A method for rendering a visual component on an application, comprising: Receiving, by a data processing system having one or more processors, a data packet including an input audio signal detected by a sensor of a client device that displays a graphical user interface of a first application in a foreground process on the client device; parsing, by the data processing system, the input audio signal of the data packet to identify a request; selecting, by the data processing system, a list of actions for performing an action corresponding to the request by a second application, the second application being installed on the client device and not in the foreground process; The data processing system generates an action data structure according to the action list; providing, by the data processing system, the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output; determining, by the data processing system, based on an authorization policy of the second application, that the output from the second application performing the action is authorized to be presented with the graphical user interface of the first application; identifying, by the data processing system, a user interface component of the second application for the action manifest in response to determining that the output of the second application is authorized to be presented with the first application; as well as The user interface component including the output from the second application is displayed, by the data processing system, on the client device along with a graphical user interface of the first application that is authorized to be presented with the second application.

14. The method according to claim 13, comprising: determining, by the data processing system, for the action list for performing the action corresponding to the second request, that the client device is not authenticated by the second application; as well as In response to determining that the client device is not authenticated by the second application, the data processing system presents a prompt interface on the client device to authenticate the client device to the second application to perform the action corresponding to the second request.

15. The method according to claim 13, comprising: receiving, by the data processing system, the data packet via a digital assistant application, the data packet including the input audio signal detected by the sensor of the client device, the client device displaying the graphical user interface of the digital assistant application corresponding to the first application in the foreground process; as well as The data processing system displays within the graphical user interface of the digital assistant application a user interface component including a subcomponent indicating the request identified from parsing the input audio signal and a user interface component including the output from the agent corresponding to the second application.

16. The method according to claim 13, comprising: The data processing system provides the action data structure to the second application so that the second application: monitoring interaction events on the user interface component from the second application presented with the graphical user interface of the first application; as well as In response to detecting the interaction event, the interaction event is processed to update the user interface component.

17. The method according to claim 13, wherein: The action data structure includes an address for executing the action of the action list.

18. The method according to claim 17, wherein: The address includes a first substring corresponding to the second application.

19. The method according to claim 18, wherein: The address includes a second substring corresponding to the action.

20. The method according to claim 19, wherein: The second substring includes at least one input variable for performing the action.

21. A system for rendering a visual component on an application, comprising: A natural language processor executing on a data processing system having one or more processors, the natural language processor being configured to: receiving a data packet including an input audio signal detected by a sensor of a client device that displays a graphical user interface of a first application running in a foreground process on the client device; as well as parsing the input audio signal of the data packets to identify a request; An action processor executed on the data processing system, for: an action list selecting to perform an action corresponding to the request by a second application, the second application being installed on the client device and not in the foreground process; as well as generating an action data structure according to the action list; as well as An agent interface executed on the data processing system, configured to: providing the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output; determining, based on an authorization policy, that the output from the second application performing the action is not authorized to be presented with the graphical user interface of the first application; as well as In response to determining that the output from the second application performing the action is not authorized to be presented with the graphical user interface of the first application: removing the first application from the foreground process on the client device, and setting the first application as a non-foreground process on the client device; Setting the second application to run in the foreground process on the client device; as well as A graphical user interface of the second application is displayed on the client device, including the output generated by the second application.

22. The system of claim 21, further comprising: the action list having an agent identifier corresponding to the second application and a request identifier corresponding to the action; the natural language processor to parse the input audio signal of the data packet to identify an agent identifier corresponding to an agent, the agent corresponding to the second application installed on the client device; as well as The action processor is configured to select the action list to perform the action having the agent identifier corresponding to the agent identifier identified from parsing the input audio signal.

23. The system of claim 21, comprising the agent interface for: determining, for the action list for performing the action corresponding to the second request, that the client device is not authenticated by the second application; and In response to determining that the client device is not authenticated by the second application, presenting a prompt interface on the client device to authenticate the client device to the second application to perform the action corresponding to the second request.

24. The system of claim 21, comprising the action handler for providing the action data structure to the second application so that the second application: parsing the action data structure to identify the action to be performed; and The output is generated by performing the action identified from the action data structure.

25. The system of claim 21, wherein: The action data structure includes an address for executing the action of the action list.

26. The system of claim 25, wherein: The address includes a first substring corresponding to the second application.

27. The system of claim 26, wherein: The address includes a second substring corresponding to the action.

28. The system of claim 27, wherein: The second substring includes at least one input variable for performing the action.

29. The system of claim 21, wherein: The authorization policy is associated with the second application.

30. The system of claim 29, wherein: The authorization policy includes a whitelist, and the first application is not included in the whitelist.

31. The system of claim 29, wherein: The authorization policy includes a blacklist, and the first application is not included in the blacklist.

32. A method for rendering a visual component on an application, comprising: Receiving, by a data processing system having one or more processors, a data packet including an input audio signal detected by a sensor of a client device that displays a graphical user interface of a first application in a foreground process on the client device; parsing, by the data processing system, the input audio signal of the data packet to identify a request; selecting, by the data processing system, a list of actions for performing an action corresponding to the request by a second application, the second application being installed on the client device and not in the foreground process; The data processing system generates an action data structure according to the action list; providing, by the data processing system, the action data structure to the second application so that the second application parses the action data structure and performs the action to generate an output; determining, by the data processing system, based on an authorization policy, that the output from the second application performing the action is not authorized to be presented with the graphical user interface of the first application; as well as In response to determining that the output from the second application performing the action is not authorized to be presented with the graphical user interface of the first application: The data processing system removes the first application from the foreground process on the client device and sets the first application as a non-foreground process on the client device; The data processing system sets the second application to run in the foreground process on the client device; as well as The data processing system displays, on the client device, a graphical user interface of the second application, including the output generated by the second application.

33. The method of claim 32, further comprising providing the action data structure to the second application so that the second application: parsing the action data structure to identify the action to be performed; and The output is generated by performing the action identified from the action data structure.

34. The method of claim 32, wherein: The action data structure includes an address for executing the action of the action list.

35. The method of claim 34, wherein: The address includes a first substring corresponding to the second application.

36. The method of claim 35, wherein: The address includes a second substring corresponding to the action.

37. The method of claim 36, wherein: The second substring includes at least one input variable for performing the action.

38. The method of claim 32, wherein: The authorization policy is associated with the second application.

39. The method of claim 38, wherein: The authorization policy includes a whitelist, and the first application is not included in the whitelist.

40. The method of claim 38, wherein: The authorization policy includes a blacklist, and the first application is not included in the blacklist.

Images