Using address templates to invoke functions of agent via digital assistant application

Abstract

The invention discloses invoking functionality of an agent via a digital assistant application using an address template. A system and method for invoking functionality of an agent via a digital assistant application is provided. Each action manifest may have an address template for the action of the proxy. The address template may include a portion having an input variable for performing an action. A data processing system may parse an input audio signal from a client device to identify a request and parameters to be executed by an agent. The data processing system may select an action manifest for actions corresponding to the request. The data processing system may generate an address using the address template. The address may include a substring having parameters for controlling execution of the action. The data processing system may steer the action data structure including the address to the agent to cause the agent to perform the action and provide an output for presentation.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980002159.8, with a filing date of May 6, 2019. Technical Field

[0003] The present disclosure relates to invoking the functionality of an agent via a digital assistant application using an address template. Background Art

[0004] An application can be installed on a computing device. The computing device can execute the application. The application can present digital content. Summary of the invention

[0005] According to a first aspect of the present disclosure, a system for calling a function of an agent via a digital assistant application is provided. The system may include a data processing system having one or more processors. The system may include an agent registry executed on the data processing system. The agent registry may maintain multiple action lists for multiple agents. Each action list may have an address template for an action of one of the multiple agents. The address template may define a first part and a second part. The first part may correspond to one of the multiple agents. The second part includes input variables for performing an action. The system may include a natural language processor executed on the data processing system. The natural language processor may receive a data packet including an input audio signal detected by a sensor of a client device. The natural language processor may parse the input audio signal of the data packet to identify a request and define parameters that the request will be performed by an agent in the multiple agents. The system may include an action handler executed on the data processing system. The action handler may select an action list of an agent that performs an action corresponding to the request from multiple action lists. The action handler may generate an address for performing an action of the action list according to the address template of the action list. The address may include a first substring and a second substring. The first substring may correspond to the agent. The second substring may have parameters for controlling the execution of the action. The action handler may generate an action data structure including an address generated by the action handler according to an address template. The system may include an agent interface executed on a data processing system. The agent interface may direct the action data structure to an agent referenced by the address so that the agent parses the action data structure, performs an action corresponding to the request according to the parameters, and provides an output for presentation on a client device.

[0006] An agent may, for example, lack natural language processing capabilities and may have predetermined input capabilities.

[0007] Each action list may have an agent identifier corresponding to one of the plurality of agents, a request identifier corresponding to the action, and an address template for the action corresponding to the request identifier. The natural language processor may parse the input audio signal of the data packet to identify the agent identifier corresponding to the agent in the plurality of agents. The action handler may select an action list having an action corresponding to the request identifier of the request from the plurality of action lists for the agent corresponding to the agent identification. The action handler may determine a correspondence between the agent identifier and the agent identifier. For example, the agent identifier may be an agent identifier.

[0008] The natural language processor may parse the input audio signal of the data packet to determine that the input audio signal lacks an agent identifier corresponding to any one of the plurality of agents. The action handler may, in response to determining that the input audio signal lacks the agent identifier, select an agent from the plurality of agents based on the request, the parameters, and an agent usage history of the client device, wherein the agent is capable of performing an action corresponding to the request according to the parameters.

[0009] At least one of the plurality of action lists may have a mapping of a second portion of an address template for the action, the second portion including a plurality of input variables for performing the action. The mapping may define a correspondence between one of the plurality of input variables and one of the plurality of words identified from parsing the input audio signal. The natural language processor may parse the input audio signal of the data packet to identify a plurality of parameters defining a request; and the action handler may generate an address for performing the action, wherein a second substring of the address has the plurality of parameters as the plurality of input variables defined by the mapping of the address template.

[0010] At least one of the plurality of action lists may have an address template for a plurality of actions to be performed by one of the plurality of agents. The natural language processor may parse a second input audio signal of a second data packet to identify a second request, a second parameter, and the agent, the second request being different from the request. The action handler may select an action list of the agent from the plurality of action lists to perform one of a plurality of actions corresponding to both the request and the second request.

[0011] At least one of the plurality of action lists may include a second portion of an address template, the address template having a plurality of input variables when performing the action, the plurality of input variables including input variables and auxiliary input variables. The action handler may determine that there are no identified parameters corresponding to the auxiliary input variables; and the action handler may generate an address for performing the action of the action list based on the address template of the action list, the second substring of the address having the parameters as the input variables and lacking the parameters as the auxiliary input variables.

[0012] The action handler can verify parameters against an address template of an action list for an action corresponding to the request; generate an address according to the address template in response to the request and verification of the parameters; and receive an output of the agent from executing the action corresponding to the request in response to directing the action data structure to the agent, the output indicating one of a success or a failure in executing the action.

[0013] The data processing system can verify a second address generated according to an address template of an action list for an action corresponding to a request; detect an error in verifying the second address, the error indicating that the second address format is incorrect; in response to detecting the error, terminate directing a second action data structure including the second address to the agent; and provide an indication of the error to an agent service of the agent.

[0014] The action handler may, in response to determining that the client device has not been authenticated by the proxy, terminate generating at least one of a second address or a second action data structure including the second address according to an address template for the action manifest; and present instructions to authenticate the client device to the proxy to perform an action corresponding to the second request.

[0015] The plurality of action lists may have a first subset of action lists including action lists defined by an administrator of the data processing system and a second subset of action lists including action lists provided by an agent service handling one of the plurality of agents.

[0016] The action handler can direct the action data structure to the agent referenced by the address so that the agent parses the address of the action data structure to identify the first substring and the second substring; identifies the action and parameters from the second substring of the address; and in response to identifying the action and parameters, executes the action according to the parameters.

[0017] The proxy interface can generate an address for executing an action in the action list based on an address template of the action list, wherein the address includes a first substring and a second substring, wherein the first substring has a host name of a proxy service that references the proxy, and the second substring has a path name and a string query, wherein the path name corresponds to the action to be executed, and the string query includes parameters as input variables.

[0018] According to a second aspect of the present invention, a method for calling a function of an agent via a digital assistant application is provided. The method may include a data processing system having one or more processors maintaining multiple action lists for multiple agents. Each action list may have an address template for an action of one of the multiple agents. The address template may define a first part and a second part. The first part may correspond to one of the multiple agents. The second part may include input variables for performing an action. The method may include a data processing system receiving a data packet including an input audio signal detected by a sensor of a client device. The method may include the data processing system parsing the input audio signal of the data packet to identify a request to be performed by an agent in the multiple agents, and defining parameters of the request. The method may include the data processing system selecting an action list of an agent that performs an action corresponding to the request from multiple action lists. The method may include the data processing system generating an address for performing an action of the action list according to the address template of the action list. The address may include a first substring and a second substring. The first substring may correspond to the agent. The second substring may have parameters for controlling the execution of the action. The method may include an action processor generating an action data structure including an address generated by the action processor according to the address template. The method may include the data processing system directing the action data structure to the proxy referenced by the address, so that the proxy parses the action data structure, performs an action corresponding to the request according to the parameters, and provides output for presentation on the client device.

[0019] The optional features of the system according to the first aspect described above and in detail below may be combined with the method of the second aspect.

[0020] In the following, these and other aspects and embodiments are discussed in detail. The foregoing information and the following detailed description include illustrative examples of the various aspects and embodiments, and provide an overview or framework for understanding the nature and character of the claimed aspects and embodiments. The accompanying drawings provide illustration and further understanding of the various aspects and embodiments, and are incorporated in and constitute a part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings are not intended to be drawn to scale. Like reference numbers and names in the various drawings represent like elements. For clarity, not every component may be labeled in every drawing. In the drawings:

[0022] Figure 1 A block diagram of an example system for invoking functionality of an agent via a digital assistant application using address templates according to examples of the present disclosure is shown.

[0023] Figure 2 An example according to the present disclosure is shown. Figure 1A sequence diagram of an example data flow in the system shown using address templates to invoke the functionality of an agent via a digital assistant application.

[0024] Figure 3 An example according to the present disclosure is shown. Figure 1 A use case diagram of an example client device in the illustrated system showing inputting a message and generating an address.

[0025] Figure 4 The use of an example according to the present disclosure is shown Figure 1 The illustrated example system is a flow chart of a method for invoking functionality of an agent via a digital assistant application using an address template.

[0026] Figure 5 is a block diagram of an example computer system that may be used to implement elements of the systems and methods described and illustrated herein, including, for example Figure 1 and Figure 3 The system shown, Figure 2 Flowchart diagram and Figure 4 The method of illustration. DETAILED DESCRIPTION

[0027] The following is a more detailed description of various concepts and implementations related to methods, devices, and systems for invoking the functionality of an agent via a digital assistant application using an address template. The various concepts introduced above and discussed in more detail below may be implemented in any of a number of ways.

[0028] Multiple applications can be installed on a client device to perform preconfigured functions such as those requested by a user operating the client device. At least one of these applications can be a digital assistant application. The digital assistant application can have natural language processing capabilities, and can obtain and parse input audio signals detected via a microphone of the client device to identify words from the input audio signal. According to the recognized words, the digital assistant application can determine the request to be executed and one or more parameters defining the request. The request can refer to one of the preconfigured functions (e.g., retrieving the results of a search query) with input variables compatible with the parameters that the digital assistant application can execute. For such requests, the digital assistant application itself can further process the request and present the audio or video output from the execution request. However, some requests may not correspond to any functions preconfigured on the digital assistant application (e.g., playing a video from an online video platform). For these types of requests, the digital assistant application can present audio or visual output, indicating that the application cannot execute the request, or in response, may not be able to take any action.

[0029] One of the other applications installed on the client device may have functions pre-configured thereon, which are not available through the digital assistant application. On the other hand, these applications (sometimes referred to as "agents" in this article) may lack the natural language processing capabilities configured by the digital assistant application. In addition, the digital assistant application may lack any integration or connection with other applications or services that manage these applications on the client device. In order to access the functions missing on the digital assistant application, the user can first close the digital assistant application, and then manually search for other applications with the required functions on the client device. Once found, the user can open to execute the application on the client device. Since the application may lack natural language processing capabilities, the user can use a keyboard, mouse, or touch-sensitive display to physically input on the application, or can use the camera on the client device to obtain an image as input. In turn, the application can process the input to execute the request indicated by the user.

[0030] The manual process of searching for an application, opening it, and then typing in order to execute a request may consume significant computing resources, such as processor and memory usage. In addition, switching back and forth between multiple applications to access different functions may cause delays to the user or result in a longer length of time, thereby reducing human-computer interaction (HCI). In addition, it may be technically difficult to integrate or connect between digital assistant applications with natural language processing capabilities and other applications that lack natural language processing capabilities to process input audio signals. Given the lack of such capabilities, passing a word recognized from an input audio signal to another application may not successfully execute the request indicated in the word.

[0031] Aspects of the disclosed subject matter provide an improved user interface. In order to address the technical challenges of the connection between a digital assistant application and other applications, the present subject matter provides an action list that can be used by a digital assistant application to access the functions of an application (agent). Specifically, the action list defines an address template that can be used to construct an address for accessing the functions of an application. The address template of the action list can be, for example, a uniform resource identifier (URI) template for constructing a web address with a host name, a path name, and a query to call the functions of an application. The host name (or scheme) can reference the location or domain name of the application. The path name can reference the function. The query can include one or more input arguments for executing a function. The address template can also define a mapping between a request parsed from an input audio signal and a function to be executed by the application.

[0032] After identifying the word from the input audio signal, the digital assistant application can determine that the request does not correspond to any preconfigured function of the digital assistant application. Instead of outputting that the digital assistant application cannot perform the indicated request, the digital assistant application can identify another application installed on the client device that can perform the function. Through this identification, the digital assistant application can identify the action list of the application for this function. Once identified, the digital assistant application can build, expand and generate an address for the function based on the action list. The address can include a host name that references the application, a path that references the function, and one or more queries containing parameters.

[0033] As the address is generated according to the address template, the digital assistant application can pass an action data structure (e.g., an API function call) including the address to the application. Since the address includes a host name that references the application, the action data structure can be directed to reach the referenced application. For example, the action data structure can be passed from the digital assistant application to the application by the operating system on the client device via an API function call. Upon receipt, the application can parse the action data structure to identify the address. The application can further parse the address to identify the path name, thereby identifying the function to be executed, and can parse the string query to identify one or more parameters that define the execution of the function. Based on the identification, the application can execute the function to generate output. The application can provide the output to the digital assistant application to be presented as audio or visual output. The application can also be called to be displayed as a foreground process on the client device to render the output on the display.

[0034] In this way, the digital assistant application can use addresses constructed using address templates to call functions of other applications, thereby extending the functions of the digital assistant application with the functions of other applications. Calls can be performed to achieve the desired functions without the need for the user to manually search, open, and enter. Using these action lists, the digital assistant application can reduce the consumption of computing resources and time spent on the user side. In addition, applications that are connected to the digital assistant application interface may continue to lack natural language processing capabilities, but the same configuration functions are now still provided through the digital assistant application. Therefore, these applications may not be separately configured with natural language processing capabilities, thereby saving capacity (e.g., disk space on the client device).

[0035] refer to Figure 1 , depicts an example system 100 that uses an address template to invoke a function of an agent via a digital assistant application. System 100 may include at least one data processing system 102, at least one client device 104, and at least one agent service 106. Data processing system 102, client device 104, and agent service 106 may be communicatively coupled to each other via at least one network 114.

[0036] The components of the system 100 can communicate via a network 114. The network 114 can include, for example, a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an ATM (asynchronous transfer mode) network, a SONET (synchronous optical network) network, an SDH (synchronous digital hierarchy) network, an NFC (near field communication) network, a local area network (LAN), a wireless network, or a wired network, or a combination thereof. The network 114 can include a wireless link such as an infrared channel or a satellite band. The topology of the network 114 can include a bus, a star, or a ring network topology. The network 114 can include a mobile phone network using any one or more protocols for communication between mobile devices including the Advanced Mobile Phone Protocol ("AMPS"), Time Division Multiple Access ("TDMA"), Code Division Multiple Access ("CDMA"), Global System for Mobile Communications ("GSM"), General Packet Radio Service ("GPRS"), or Universal Mobile Telecommunications System ("UMTS"). Different types of data can be transmitted via different protocols, or the same type of data can be transmitted via different protocols.

[0037] The data processing system 102 and the proxy service 106 can each include multiple logically grouped servers and contribute to distributed computing technology. The logical group of servers can be referred to as a data center, a server farm, or a machine farm. The servers can also be geographically dispersed. The data center or machine farm can be managed as a single entity, or the machine farm can include multiple machine farms. The servers in each machine farm can be heterogeneous-one or more servers or machines can operate according to one or more operating system platforms. The data processing system 102 and the proxy service 106 can respectively include servers in a data center stored in one or more high-density rack systems and associated storage systems, such as those located in an enterprise data center. In this way, the data processing system 102 or the proxy service 106 with integrated servers can improve the manageability of the system, data security, physical security of the system, and system performance by locating servers and high-performance storage systems on a localized high-performance network. The centralization of all or some data processing system 102 or proxy service 106 components including servers and storage systems and coupling them with advanced system management tools can allow more efficient use of server resources, thereby saving power and processing requirements and reducing bandwidth usage. Each component of data processing system 102 may include at least one processing unit, server, virtual server, circuit, engine, agent, device, or other logic device, such as a programmable logic array configured to communicate with data repositories 128 and 150 and other computing devices. Proxy service 106 may also include at least one processing unit, server, virtual server, circuit, engine, agent, device, or other logic device, such as a programmable logic array configured to communicate with data repositories and other computing devices.

[0038] The data processing system 102 may include at least one instance of a digital assistant application 108. The digital assistant application 108 may include at least one natural language processor (NLP) component 116 (sometimes referred to as a natural language processor in this article) to parse audio-based input. The digital assistant application 108 may include at least one audio signal generator component 118 (sometimes referred to as an audio signal generator in this article) to generate audio-based signals. The digital assistant application 108 may include at least one action processor component 120 (sometimes referred to as an action processor in this article) to generate an action data structure based on audio-based input. The digital assistant application 108 may include at least one response selector component 122 to select a response to an audio-based input signal. The digital assistant application 108 may include at least one proxy registry component 124 (sometimes referred to as a proxy registry in this article) to maintain an action list for generating an action data structure. The digital assistant application 108 may include at least one proxy interface component 126 (sometimes referred to as a proxy interface in this article) to communicate with one or more proxy applications 110.

[0039] The digital assistant application 108 may include at least one data repository 128. The data repository 128 of the data processing system 102 may include one or more local or distributed databases, and may include a database management system. The data repository 128 may include one or more regular expressions 130, one or more data parameters 132, one or more strategies 134, one or more response data 136, one or more templates 138, and one or more action lists 140. The NLP component 116, the audio signal generator component 118, the action processor component 120, the response selector component 122, the agent registry component 124, the agent interface component 126, and the data repository 128 may be separated from each other. The data repository 128 may include a computer data storage or memory, and may store one or more regular expressions 130, one or more data parameters 132, one or more strategies 134, response data 136, templates 138, and one or more action lists 140, etc. The data parameters 132, strategies 134, and templates 138 may include information such as rules for a voice-based conversation between the client device 104 and the data processing system 102. Regular expressions 130 may include rules regarding when a voice-based conversation between client device 104 and data processing system 102 will include proxy application 110 and proxy service 106. Response data 136 may include content items of audio output or related metadata, as well as input audio messages that may be part of one or more communication sessions with client device 104. Action manifest 140 may include information for invoking proxy application 110 or connecting with proxy application 110.

[0040] The data processing system 102 may include an instance of at least one agent application 110 (also referred to herein as an "agent" or "application") that performs various functions. The agent service 106 may include an instance of at least one agent application 110 that performs various functions. The agent application 110 may lack natural language processing capabilities (such as the capabilities provided by the NLP component 116). The agent application 110 may also have a preconfigured or predetermined set of input capabilities. The input capabilities of the agent application 110 may include, for example, input from a mouse, keyboard, touch screen (e.g., display 146) or camera of the client device 104. The input capabilities of the agent application 110 may lack audio (e.g., microphone 144) from the client device 104. The agent application 110 may have a graphical user interface (GUI) to receive input from the client device 104. Based on the input from the predetermined input capabilities, the agent application 110 may implement or execute one or more functions. For example, the agent application 110 may be a coffee purchasing application and may have a graphical user interface with one or more elements to specify an order for coffee. In the application, a user operating a client device 104 can manually enter coffee, size, and billing information on a graphical user interface rendered on a display 146 of the client device 104. Once set up, the agent application 110 of the client device 104 can send the input to the agent service 106 via the network 114. In turn, the agent service 106 can execute the purchase order indicated by the input entered through the graphical user interface of the agent application 110.

[0041] The components of data processing system 102 may each include at least one processing unit or other logic device, such as a programmable logic array engine or module configured to communicate with data storage repository 128. The components of data processing system 102 may be separate components, a single component, or part of multiple data processing systems 102. System 100 and its components, such as data processing system 102, may include hardware elements, such as one or more processors, logic devices, or circuits.

[0042] The functionality of data processing system 102, such as digital assistant application 108 or proxy application 110, may be included or otherwise accessible from one or more client devices 104. The functionality of data processing system 102 may correspond to the functionality or interface of digital assistant application 108 executed on client device 104. Client device 104 may respectively include and execute separate instances of one or more components of digital assistant application 108. Client device 104 may otherwise access the functionality of components of digital assistant application 108 on remote data processing system 102 via network 114. For example, client device 104 may include the functionality of NLP component 116 and may access the remaining components of digital assistant application 108 via network 114 to data processing system 102. The functionality of data processing system 102 may correspond to the functionality or interface of proxy application 110 executed on client device 104. Client device 104 may otherwise access the functionality of components of proxy application 110 or proxy service 106 on data processing system 102 via network 114. For example, client device 104 may pre-process input from a mouse, keyboard, touch screen, or camera for proxy application 110 and provide the input to proxy service 106 to perform the functions of proxy application 110. Digital assistant application 108 executing on client device 104 may include the functionality of digital assistant application 108 executing on data processing system 102.

[0043] Client devices 104 may each include at least one logical device, such as a computing device with a processor, to communicate with data processing system 102 via network 114. Client devices 104 may include instances of any of the components described with respect to data processing system 102 or proxy service 106. Client devices 104 may include instances of digital assistant application 108. Client devices 104 may include instances of proxy application 110. Client devices 104 may include desktop computers, laptop computers, tablet computers, personal digital assistants, smartphones, mobile devices, portable computers, thin client computers, virtual servers, speaker-based digital assistants, or other computing devices.

[0044] The client device 104 may include, execute, interface with, or otherwise communicate with one or more of at least one instance of the digital assistant application 108, at least one instance of the proxy application 110, and at least one communication interface 112, etc. In addition, the client device 104 may include, interface with, or otherwise communicate with at least one speaker 142, at least one microphone 144, and at least one display 146. The client device 104 may include an audio driver that provides a software interface for the speaker 142 and the microphone 144. The audio driver may execute instructions provided by the data processing system 102 to control the speaker 142 to generate corresponding sound waves or sound waves. The audio driver may execute an audio file or other instructions to convert the sound waves or sound waves obtained from the microphone 144 to generate audio data. For example, the audio driver may execute an analog-to-driver converter (ADC) to convert the sound waves or sound waves into audio data. The client device 104 may include a graphics driver to provide a software interface for the display 146. The graphics driver may execute instructions provided by the data processing system 102 or the proxy service 106 to control the display 146 to generate corresponding renderings thereon.

[0045] The instance of the digital assistant application 108 on the client device 104 may include or be executed by one or more processors, logic arrays, or memories. The instance of the digital assistant application 108 on the client device 104 may detect a keyword and perform an action based on the keyword. The digital assistant application 108 on the client device 104 may be an instance of a digital assistant application 108 executed at the data processing system 102, or may perform any function of the digital assistant application 108. The instance of the digital assistant application 108 on the client device 104 may filter out one or more terms or modify the terms (e.g., the instance of the digital assistant application 108 on the data processing system 102) for further processing before sending the terms as data to the data processing system 102. The instance of the digital assistant application 108 on the client device 104 may convert an analog audio signal detected by the speaker 142 into a digital audio signal, and send one or more data packets carrying the digital audio signal to the data processing system 102 via the network 114. The instance of the digital assistant application 108 on the client device 104 can transmit a data packet carrying some or the entire input audio signal in response to detecting an instruction to perform such a transmission. The instruction may include, for example, a trigger keyword or other keyword or permission to send a data packet including the input audio signal to the data processing system 102.

[0046] The instance of the digital assistant application 108 on the client device 104 can perform pre-filtering or pre-processing on the input audio signal to remove audio of certain frequencies. The pre-filtering can include filters such as low-pass filters, high-pass filters, or band-pass filters. The filter can be applied in the frequency domain. The filter can be applied using digital signal processing techniques. The filter can maintain frequencies corresponding to human speech or human pronunciation while eliminating frequencies outside the typical frequencies of human speech. For example, a band-pass filter can remove frequencies below a first threshold (e.g., 70Hz, 75Hz, 80Hz, 85Hz, 90Hz, 95Hz, 100Hz, or 105Hz) and above a second threshold (e.g., 200Hz, 205Hz, 210Hz, 225Hz, 235Hz, 245Hz, or 255Hz). Applying a band-pass filter can reduce computing resource utilization in downstream processing. The instance of the digital assistant application 108 on the client device 104 can apply a band-pass filter before transmitting the input audio signal to the data processing system 102, thereby reducing network bandwidth utilization. However, based on the computing resources available to the client device 104 and the available network bandwidth, it may be more efficient to provide the input audio signal to the data processing system 102 to allow the data processing system 102 to perform filtering. The instance of the digital assistant application 108 on the client device 104 may apply additional pre-processing or pre-filtering techniques, such as noise reduction techniques, to reduce the level of ambient noise that may interfere with the natural language processor. Noise reduction techniques may improve the accuracy and speed of the natural language processor, thereby improving the performance of the data processing system 102, and managing the rendering of the graphical user interface provided via the display.

[0047] The client device 104 may be operated by an end user who inputs a voice query as an audio input (via a microphone 144 or a speaker 142) into the client device 104, and receives an audio (or other) output from the data processing system 102 or the proxy service 106 to present, display, or render to the end user of the client device 104. The digital component may include a computer-generated voice that may be provided to the client device 104 from the data processing system 102 or the proxy service 106. The client device 104 may render the computer-generated voice to the end user via the speaker 142. The computer-generated voice may include a recording from a real person or computer-generated language. The client device 104 may provide a visual output via a display 146 communicatively coupled to the client device 104.

[0048] The data processing system 102 and the proxy service 106 may each include at least one server having at least one processor. For example, the data processing system 102 and the proxy service 106 may each include a plurality of servers located in at least one data center or server farm. The data processing system 102 may determine a request and a trigger keyword associated with the request from an audio input signal. Based on the request and the trigger keyword, the data processing system 102 may determine whether to forward the audio input signal to the proxy service 106 or to process the audio input signal internally. In response to determining that the audio input signal will be processed internally, the data processing system 102 may generate or select response data. The response data may be based on audio or text. For example, the response data may include one or more audio files, which, when rendered, provide audio output or sound waves. The data within the response data may also be referred to as content items. In addition to audio content, the response data may also include other content (e.g., text, video, or image content). In response to determining that the audio input signal is to be forwarded, the data processing system 102 may send the audio input signal to the proxy service 106. The proxy service 106 may parse the audio input signal to identify the command to be executed. Proxy service 106 may execute the command and return the results of the command to data processing system 102 or client device 104 .

[0049] The data processing system 102 may include at least one computing resource or server. The data processing system 102 may include, interface with, or otherwise communicate with at least one communication interface 112. The data processing system 102 may include, interface with, or otherwise communicate with at least one instance of a digital assistant application 108 on the data processing system 102. The instance of the digital assistant application 108 on the data processing system 102 may include, interface with, or otherwise communicate with at least one NLP component 116, at least one audio signal generator component 118, at least one action handler component 120, at least one response selector component 122, at least one proxy registry component 124, and at least one proxy interface component 126, etc. The data processing system 102 may include, interface with, or otherwise communicate with at least one data repository 128. At least one data repository 128 may include or store regular expressions 130, data parameters 132, policies 134, response data 136, templates 138, and action lists 140 in one or more data structures or databases. The data repository 128 may include one or more local or distributed databases and may include database management.

[0050] An instance of the digital assistant application 108 of the data processing system 102 may execute or run an instance of the proxy registry component 124 to maintain a set of action lists 140 on the data repository 128. The proxy registry component 124 may maintain and store a set of action lists 140 for one or more agent applications 110 on the data repository 128. A subset of action lists 140 may be defined by an administrator of the data processing system 102 or the digital assistant application 108 for an application type corresponding to the agent application 110. For example, an administrator of the digital assistant application 108 may configure or define a set of built-in action lists 140 for a purchase order application. Another subset of action lists 140 may be provided by one of the agent services 106 that handles resources for the corresponding agent application 110. At least one of the action lists 140 may be a preset action list 140 for performing an action in response to determining that a request does not match any other action list 140 of the agent application 110. For example, the agent service 106 may provide a set of customized action lists 140 to the data processing system 102 for storage on the data repository 128. The proxy registry component 124 can receive a subset of the action manifests 140 provided by the proxy service 106. Upon receipt, the proxy registry component 124 can store the received action manifests 140 on the data repository 128. The presence of the action manifests 140 for the actions of the proxy application 110 on the data repository 128 can indicate that the proxy application 110 is capable of performing the actions corresponding to the action manifests 140.

[0051] Each action list 140 can have at least one address template for at least one action of one of the proxy applications 110. The address template can be a single action for the proxy application 110 or multiple actions carried by the proxy application 110. The address template for the action may include a first part and a second part. The address template may be, for example, a uniform resource identifier (URI) template from which a URI is generated. The first part of the address template may correspond to one of the proxy applications 110 or a proxy service 106 that provides resources for the proxy application 110. For example, the first part of the address template may include a scheme in a custom deep link URI that references the proxy application 110. The first part of the address template may also include a host name that references the proxy service 106.

[0052] The second part of the address template may correspond to an action to be performed by the proxy application 110. The second part of the address template may also include one or more input variables (also referred to herein as input variables of a field) for performing the action. For example, the second part of the address template may include a path name in the URI corresponding to the action to be taken. The second part may also include one or more query strings after the path name in the URI. Each query string may correspond to one of the input variables for performing the action. A subset of the input variables may include optional or auxiliary input variables. The action may be performed by the proxy application 110 without auxiliary input variables in the address.

[0053] The action list 140 may include a mapping (sometimes referred to herein as a parameter mapping) of the second portion of the address template. The mapping may specify or define a correspondence between at least one input variable of the second portion and one or more words to be identified from the parsed input audio signal. One or more words from the parsed input audio signal may correspond to an input variable defining an action to be performed by the agent application 110. The mapping may define the insertion of words into the input variables of the address template. For example, a mapping for a coffee purchase order action may specify that the coffee name parsed from the audio signal is inserted into the coffee input variable of the address template. The action list 140 may specify or include a set of allowed values ​​(sometimes referred to as an entity list) for each input variable. The set of allowed values ​​may be part of the mapping of the action list 140. For example, for an input variable for a coffee name, the allowed values ​​for a coffee purchase order may include "latte", "coffee latte", and "latte coffe", etc. The action list 140 may also include a tag indicating that the corresponding input variable of the address template is optional or auxiliary. The tag may be part of the mapping of the second portion of the address template.

[0054] In addition to the address template, each action list 140 maintained by the agent registry component 124 may include information related to the action for the action list 140. The action list 140 may include at least one agent identifier corresponding to one of the agent applications 110 that perform the action. The agent identifier may include a collection of alphanumeric characters that reference the agent application 110. The agent identifier may be used to identify the agent application 110 associated with the action list 140 from the words parsed from the input audio signal. The action list 140 may include at least one request identifier corresponding to the action to be performed by the agent application 110. The request identifier (sometimes referred to as "intention identifier" or "intention name" in this article) may include an alphanumeric character representation of the action to be performed. The address template may be uniquely identified by one or more request identifiers. The request identifier may be used to identify the action list 140 from the words parsed from the input audio signal.

[0055] Each action list 140 may be implemented or specified using a markup language such as an extensible markup language (XML) file. For example, the action list 140 may be an XML file in the following form:

[0056]

[0057] In the example above, action manifest 140 may be for a ride order to be performed by a transportation application. The entry "actions.intent.ORDER_RIDE" may correspond to a request identifier of action manifest 140. The entry "https: / / ex_transport.appexample.com / order{?dropoffAddress}" may correspond to an address template, where "https: / / ex_transport.appexample.com / " corresponds to the first part and " / order{?dropoffAddress}" corresponds to the second part. Entries beginning with "parameter-mapping" may correspond to mappings.

[0058] The proxy registry component 124 may construct, create or generate a set of action lists 140 for each proxy application 110. The proxy service 104 of the proxy application 110 may provide a configuration file associated with the proxy application 110 to the proxy registry component 124 via the network 114. The configuration file may be, for example, an application binary interface (ABI) for the proxy application 110 submitted to the proxy registry component 124 by an application developer associated with the proxy application 110. The proxy service 104 of the proxy application 110 may provide metadata associated with the proxy application 110. The metadata may include, among other things, a proxy identifier and an application type of the proxy registry component 124. The application type may indicate the use of the proxy application 110. The proxy registry component 124 may receive information from the proxy service 106 to generate a set of action lists 140 for the proxy application 110.

[0059] The proxy registry component 124 can read, ingest and parse the configuration file for the proxy application 110 to identify one or more actions. For each identified function, the proxy registry component 124 can generate an action list 140 for the action. The action can be one of the actions in the set of action lists 140 defined by the administrator of the data processing system 102 or the digital assistant application 108 for the application type of the proxy application 110. The proxy registry component 1240 generates an address template for the action to be performed by the proxy application 110. The first part of the address template (e.g., the host name or scheme) can correspond to the proxy application 110. The second part of the address template can be predefined for the action (e.g., using a preset path name or query string). The proxy registry component 124 can also generate a mapping for the action list 140 for the identified function. The proxy registry component 124 can include or add a request identifier corresponding to the identified action to the action list 140. The mapping can include a set of allowed values ​​for the input variables of the address template. The proxy registry component 124 can include or add a proxy identifier corresponding to the proxy application 110 to the action list 140. By constructing the action manifest 140 , the proxy registry component 124 can store the action manifest 140 on the data repository 128 .

[0060] The data processing system 102 may include at least one communication interface 112. The communication interface 112 may be configured, constructed or used to receive and send information using, for example, data packets. The communication interface 112 may receive and send information using one or more protocols such as network protocols. The communication interface 112 may include a hardware interface, a software interface, a wired interface or a wireless interface. The communication interface 112 may be a data interface or a network interface that enables the components of the system 100 to communicate with each other. The communication interface 112 of the data processing system 102 may provide or send one or more data packets including action data structures, audio signals or other data to the client device 104 or the proxy service 106 via the network 114. For example, the data processing system 102 may provide an output signal from the data repository 128 or from the audio signal generator component 118 to the client device 104.

[0061] The data processing system 102 may also instruct the client device 104 to perform the function indicated in the action data structure via data packet transmission. The output signal may be obtained, generated, converted, or transmitted to the client device 104 from the data processing system 102 (or other computing device) as one or more data packets (or other communication protocols). The communication interface 112 may facilitate the conversion or formatting of data from one format to another. For example, the communication interface 112 may include an application programming interface ("API") that includes definitions for communication between various components such as software components. An application, script, program, or other component associated with the data processing system 102 may be installed at the client device 104. The application may enable the client device 104 to transmit the input audio signal (and other data) to the communication interface 112 of the data processing system 102.

[0062] The data processing system 102 may include an application, script, or program installed on the client device 104, such as an instance of a digital assistant application 108 on the client device 104, to transmit the input audio signal to the communication interface 112 of the data processing system 102, and drive the components of the client computing device to render the output audio signal or visual output. The data processing system 102 may receive data packets, digital files, or other signals including or identifying an input audio signal (or multiple input audio signals). The client device 104 may detect the audio signal via a speaker 142, and convert the analog audio signal to a digital file via an analog-to-digital converter. For example, the audio driver may include an analog-to-digital converter component. The preprocessor component may convert the audio signal to a digital file that can be transmitted via data packets on the network 114.

[0063] The digital assistant application 108 of the data processing system 102 or an instance of the client device 104 can execute or run the NLP component 116 to receive or obtain data packets including input audio signals detected by the microphone 144 of the client device 104. The data packets can provide digital files. The NLP component 116 can receive or obtain digital files or data packets including audio signals and parse the audio signals. For example, the NLP component 116 can provide interaction between humans and machines. The NLP component 116 can be configured with technologies for understanding natural language and enabling the data processing system 102 to derive meaning from human or natural language input. The NLP component 116 can include or be configured with machine learning-based technologies, such as statistical machine learning. The NLP component 116 can parse the input audio signal using a decision tree, a statistical model, or a probability model.

[0064] The NLP component 116 can perform functions such as named entity recognition (e.g., given a stream of text, determine which items in the text map to names, such as people or places, and what the type of each such name is, such as person, location (e.g., "home"), or organization), natural language generation (e.g., converting information from a computer database or semantic intent into understandable human language), natural language understanding (e.g., converting text into a more formal representation, such as a first-order logic structure that a computer module can process), machine translation (e.g., automatically translating text from one human language to another), morpheme segmentation (e.g., dividing words into individual morphemes and identifying classes of morphemes based on the morphological or structural complexity of the words in the language under consideration, which can be challenging), question answering (e.g., determining answers to human language questions, which can be specific or open-ended), and semantic processing (e.g., processing performed after identifying a word and encoding its meaning to relate the identified word to other words with similar meanings).

[0065] The NLP component 116 can parse the input audio signal and convert it into a recognized string by comparing the input signal with a representative audio waveform set stored (e.g., in the data repository 128) and selecting the closest match. The NLP component 116 can also divide or subdivide the input audio signal into one or more audio segments of duration (e.g., 15 seconds to 2 minutes) to process each segment. The audio waveform set can be stored in the data repository 128 or other databases accessible to the data processing system 102. The representative waveform is generated across a large set of users and can then be enhanced by a voice sample from the user. After the audio signal is converted into recognized text, the NLP component 116 matches the text with words associated with actions that the data processing system 102 can serve, for example, via training between users or by manual specification. The NLP component 116 can determine that the input audio signal obtained from the microphone 144 does not contain any recognizable string. When determining that the input audio signal does not contain any recognizable string, the NLP component 116 can determine that the input audio signal contains silence (e.g., the maximum amplitude is less than 0dB). In addition, the NLP component 116 can determine a signal-to-noise ratio (SNR) of the input audio signal. The NLP component 116 can compare the SNR of the input audio signal to a threshold SNR (e.g., -20 dB). In response to determining that the SNR of the input audio signal is greater than the threshold SNR, the NLP component 116 can determine that the input audio signal does not contain any recognizable character strings.

[0066] In addition to or in lieu of an input audio signal, the data processing system 102 may receive an image or video input signal. The NLP component 116 may convert the input image or video into a text or digital file. The NLP component 116 may process, analyze, or interpret the image or video input to perform an action, generate a request, or select or identify a data structure. The data processing system 102 may process the image or video input signal using, for example, image interpretation technology, computer vision, a machine learning engine, or other technology to identify or interpret the image or video to convert the image or video into a digital file. One or more image interpretation technologies, computer vision technologies, or machine learning technologies may be collectively referred to as imaging technologies. In addition to or in lieu of audio processing technology, the data processing system 102 (e.g., the NLP component 116) may be configured with imaging technology.

[0067] By parsing the input audio signal, the NLP component 116 can determine or identify at least one request. The input audio signal may include, for example, a query, a question, a command, an instruction, or other statements in a natural language. The request may correspond to at least one trigger keyword, which is identified from the identified string converted from the input audio signal. The request may indicate an action to be taken. For example, the NLP component 116 may parse the input audio signal to identify at least one request to leave home for dinner and watch a movie at night. The trigger keyword may include at least one word, phrase, root or part of a word, or a derivative indicating an action to be taken. For example, the trigger keyword "go" or "to go to" from the input audio signal may indicate a request for transportation. In this example, the input audio signal (or the identified request) does not directly express the intention of transportation, but the trigger keyword indicates that transportation is an auxiliary action of at least one other action indicated by the request.

[0068] To identify the request, the NLP component 116 may apply semantic processing techniques to the input audio signal to identify a trigger keyword corresponding to the request. The NLP component 116 may apply semantic processing techniques to the input audio signal to identify a trigger phrase including one or more trigger keywords, such as a first trigger keyword and a second trigger keyword. For example, the input audio signal may include the sentence "Find the nearest café". The NLP component 116 may determine that the input audio signal includes the trigger keyword "find". The NLP component 116 may determine that the request is to search for a location near the client device 104.

[0069] In addition to the request, the NLP component 116 can also identify at least one parameter that defines the request. The parameter can define the request, acting as a supplement or constraint to the action corresponding to the request to be taken. The parameter can further specify the action to be taken. The parameter may include a subset of the identified string converted from the input audio signal (excluding the trigger keyword). The NLP component 116 can apply semantic processing technology to the input audio signal to identify one or more descriptor words related to the identified trigger keyword. From the example "Find the nearest café", the NLP component 116 may have identified the term "find" as a trigger keyword. Combined, the NLP component 116 can identify "nearest" as a first parameter and "café" as a second parameter defining the request "find".

[0070] In addition, the NLP component 116 can identify an application identifier from a string identified from the input audio signal. The application identifier can correspond to one of the proxy applications 110. In order to identify the application identifier, the NLP component 116 can perform a named entity recognition algorithm on the string identified from the input audio signal. When using the named entity recognition algorithm, the NLP component 116 can maintain a list of proxy identifiers for the proxy applications 110. The list may include proxy identifiers of the proxy applications 110 installed on the client device 104 that receives the input audio signal including the request. By applying the named entity recognition algorithm, the NLP component 116 can determine that the input audio signal includes an application identifier corresponding to the proxy application 110. For example, the NLP component 116 can determine that the input audio signal "Order car with ride sharing service XYZ (using carpooling service XYZ to book a car)" includes an explicit proxy application 110 "ride sharing service XYZ (carpooling service XYZ)". On the contrary, by applying the named entity recognition algorithm, the NLP component 116 can determine that the input audio lacks an application identifier corresponding to any proxy application 110.

[0071] Based on the request, the NLP component 116 can determine whether the request corresponds to a function of the digital assistant application 108 or a function of the proxy application 110. To determine, the NLP component 116 can access a list of requests for functions of the digital assistant application 108. The list of requests for functions of the digital assistant application 108 can be retained in the data repository 128. The request list can include a string set of trigger keywords and requests corresponding to requests that are predetermined to be associated with the functions of the digital assistant application 108. For example, the list can include "get", "show", "search", and "find", etc. With the identification of the request, the NLP component 116 can compare the request with the list of requests for functions of the digital assistant application 108.

[0072] The NLP component 116 can also use a regular expression 130 maintained on the data repository 128 to determine whether a request corresponds to a function of the digital assistant application 108. The regular expression 130 can define a pattern for matching to determine whether a keyword identified from the input audio signal references at least one function of the digital assistant application 108. The regular expression 130 can also specify which keywords are used to execute the command indicated in the input audio signal. For example, the regular expression 130 can take the form of {[request], [auxiliary parameters]}}. For keywords of the input audio signal that are determined to reference a function of the digital assistant application 108, the regular expression 130 can specify one or more keywords including a request for the digital assistant application 108 and auxiliary variables. The regular expression 130 can specify the order of the request and the referenced keyword in one or more keywords identified from the input audio signal.

[0073] Based on the comparison, the NLP component 116 can determine that the request identified from the input audio signal matches one of the request list for functions of the digital assistant application 108. In response to the determination, the NLP component 116 can determine that the request corresponds to one of the functions of the digital assistant application 108. For example, the NLP component 116 can parse the words "What is the weather?" from the input audio signal, and can identify that the input audio signal includes a request for the weather. The request list for functions of the digital assistant application 108 can specify that the request for the weather is one of the functions of the digital assistant application 108. In this example, the NLP component 116 can determine a match between the two requests, and can determine that the request references one of the functions of the digital assistant application 108.

[0074] In response to determining that the request corresponds to the function of the digital assistant application 108, the digital assistant application 108 can execute and fulfill the request identified from the parsed input audio signal. In addition, the digital assistant application 108 can execute the request according to the parameters defining the request. For example, the request can be a definition of a search word (e.g., "abnegation"). The digital assistant application 108 can perform an Internet search to retrieve the definition of a word. When fulfilling the request, the digital assistant application 108 can call a response selector component 122. The response selector component 122 can use a strategy 134 or response data 136 maintained on a data repository 128 to select or identify a response phrase. The strategy 134 can be specific to the request and can specify the response data 136 of the request. The response selector component 122 can use the request type of the response to search for the strategy 134 to generate an output when fulfilling the request. Once the strategy 134 is identified, the response selector component 122 can select a response phrase. The response phrase can include a string set such as a word or phrase. The digital assistant application 108 can display the response phrase on the display 146 of the client device 104. For example, the digital assistant application 108 may display a content item including a response phrase on a graphical user interface of the digital assistant application 108 on the display 146 of the client device 104 .

[0075] By identifying the response phrase, the digital assistant application 108 can call the audio signal generator component 118. The instance of the digital assistant application 108 running on the data processing system 102 or the client device 104 can execute the audio signal generator component 118. Using the selected response phrase, the audio signal generator component 118 can generate an output audio signal. Based on one or more words included in the response phrase, the audio signal generator component 118 can generate an output audio signal. The audio signal generator component 118 can play the output audio signal on the speaker 142 of the client device 104.

[0076] On the contrary, from the comparison, the NLP component 116 can determine that the request parsed from the input audio signal does not match any of the list of requests for functions of the digital assistant application 108. In response to this determination, the NLP component 116 can determine whether the request corresponds to one of the functions of the agent application 110. In order to determine, the NLP component 116 can access a list of requests for functions of the agent application 110. The list of requests for functions can be for the agent application 110 installed on the client device 104. The list of requests for functions of the agent application 110 can be maintained on the data repository 128. The request list can include a set of strings for triggering keywords and requests corresponding to requests predetermined to be associated with functions of the agent application 110. The NLP component 116 can compare the request with the list of requests for functions of the agent application 110.

[0077] By comparison, the NLP component 116 can determine that the request identified from the input audio signal matches one of the request lists for the functions of the agent application 110. In response to the determination, the NLP component 116 can determine that the request corresponds to one of the functions of the agent application 110. For example, the NLP component 116 can parse the word "Buy melaundry detergent" from the input audio signal, and can identify that the input audio signal includes buying detergent. The request list for the functions of the agent application 110 can specify that the request for weather is one of the functions of the agent application 110. In this example, the NLP component 116 can determine a match between the two requests, and can determine that the request references one of the functions of the agent application 110, namely, a function for such a purchase. In addition, in response to determining that the input audio signal includes an application identifier of the agent application 110, the NLP component 116 can determine that the request does not correspond to a function of the digital assistant application 108.

[0078] On the other hand, the NLP component 116 can determine from the comparison that the request does not correspond to any functionality of any proxy application 110. After determining that the request does not correspond to the digital assistant application 108 or the proxy application 110, the response selector component 122 can generate or identify a response phrase indicating that the request does not correspond to any functionality available on the client device 104. For example, the response phrase can include "Sorry, I don't understand your request." The response phrase can be selected from the response data 136 based on the determination. In addition, the audio signal generator component 118 can use one or more words of the response phrase to generate a corresponding output audio signal. The audio signal generator component 118 can play the output audio signal on the speaker 142 of the client device 104.

[0079] An instance of the digital assistant application 108 running on the data processing system 102 or the client device 104 can execute the action handler component 120 to generate an action data structure based on the determination. The action handler component 120 can execute a script or program based on the input received from the NLP component 116. The proxy service 106 can provide the script or program. The proxy service 106 can make the script or program available to the data processing system 102 through an API. The action handler component 120 can determine the parameters or responses to the input fields and can package the data into an action data structure. The action data structure can be provided to the proxy application 110 through an API. The action handler component 120 can transmit the action data structure to the proxy service 106 to implement the request recognized from the input audio signal parsed by the NLP component 116.

[0080] In response to determining that the request corresponds to a function of the proxy application 110, the action handler component 120 may identify or select an action list 140 from the data repository 128 for an action corresponding to the request. The action handler component 120 may select at least one action list 140 from a set of action lists 140 maintained on the data repository 128. The action handler component 120 may search the data repository 128 for an action list 140 using the request identified from the input audio signal. The action handler component 120 may also search the data repository 128 for an action list 140 using one or more parameters identified from the input audio signal. The action handler component 120 may traverse the set of action lists 140 maintained on the data repository 128. For each action list 140, the action handler component 120 may identify a request identifier for the action list 140.

[0081] The action handler component 120 can compare the request identified from the input audio signal with the request identifier of the action list 140. The comparison of the request identifier from the input audio signal with the request identifier of the action list 140 can be based on the semantic knowledge graph. The action handler component 120 can call the NLP component 116 to use the semantic knowledge graph to determine whether the request is semantically related to the request identifier. The semantic knowledge graph may include a collection of nodes and edges. Each node may correspond to one or more words or phrases. Each edge may define the semantic distance between a pair of words represented by the corresponding words on the node. From the semantic knowledge graph, the NLP component 116 can identify the node corresponding to the request, the node corresponding to the request identifier, and the semantic distance indicated in the pair of nodes. The NLP component 116 can compare the semantic distance with the threshold distance. When it is determined that the semantic distance does not meet (e.g., greater than) the threshold distance, the NLP component 116 can determine that the request does not match the request identifier of the action list 140. On the other hand, when it is determined that the semantic determination satisfies (e.g., is less than or equal to) the threshold distance, the NLP component 116 can determine that the request does match the request identifier of the action manifest 140. Based on the comparison, the action handler component 120 can determine that the request does not match the request identifier of the action manifest 140. In response to this determination, the action handler component 120 can identify the next action manifest 140 in the set maintained on the data repository 128 and can repeat the comparison.

[0082] On the contrary, from the comparison, the action handler component 120 can determine that the request matches the request identifier of the action list 140. In response to the determination, the action handler component 120 can select an action list 140 for the action corresponding to the request. Because as described above, multiple requests can correspond to the same action list 140, the action handler component 120 can select an action list 140 that was or will be selected for another request. Before selecting an action list 140, the action handler component 120 can also compare one or more parameters identified from the input audio signal with the input variables of the action list 140. The action handler component 120 can identify the input variables from the parameter mapping or address template of the action list 140. When comparing the identified parameters with the input variables of the action list 140, the action handler component 120 can exclude the auxiliary parameters indicated in the action list 140. The action handler component 120 can determine that the identified parameters match the input variables of the action list 140 based on the allowed value set of each input variable. In response to the determination, the action handler component 120 can select an action list 140 for an action corresponding to the request defined by one or more parameters. The action handler component 120 can also stop traversing the collection of action lists 140 in the data repository 128. Otherwise, the action handler component 120 can determine that the identified parameters do not match the input variables of the action list 140. The action handler component 120 can identify another action list 140 for the action corresponding to the same request, and can repeat the comparison. After comparing with all action lists 140, the action handler component 120 can determine that the identified parameters do not match the input variables in any action list 140 of the proxy application 110. In response to the determination, the action handler component 120 can select a preset action list 140 for the proxy application 110 to respond to the request.

[0083] The action handler component 120 can also use the application identifier identified from the input audio signal to search the action list 140 in the data repository 128. Before searching using the request, the action handler component 120 can identify a subset of the action list 140 whose application identifier matches the application identifier. The application identifier can correspond to one of the proxy applications 110. Each action list 140 of the identified subset can include an application identifier determined to match the application identifier identified from the input audio signal. In order to identify the subset, the action handler component 120 can compare the application identifier with the application identifier of each action list 140. When it is determined that the application identifier matches the application identifier of the action list 140, the action handler component 120 can include the action list 140 in the subset. Otherwise, when it is determined that the application identifier does not match the application identifier of the action list 140, the action handler component 120 can exclude the action list 140 from the subset. The action handler component 120 can use the request to traverse the application identifier in the identified subset of the action list 140 to select the action list 140 of the action corresponding to the request.

[0084] When it is determined that the input audio signal lacks an application identifier, the action handler component 120 can select an action list 140 from a collection maintained on the data repository 128 based on the request, one or more parameters, and the agent usage history. The agent usage history can indicate usage statistics of the agent application 110 on the client device 104 running the digital assistant application 108 from which the input audio signal is received. The agent usage history can also indicate the functions of the agent application 110 called on the client device 104. The agent usage history can be limited to a defined time window before the request is identified in the input audio signal detected on the client device 104. The action handler component 120 can identify a subset of the action list 140 by the request identifier that matches the request and one or more parameters. The subset may include action lists 140 with different application identifiers corresponding to multiple agent applications 110. The action handler component 120 can use the agent usage history of the client device 104 to select the action list 140 from the subset. The action handler component 120 can identify at least one application identifier from the agent usage history. The action handler component 120 may compare the application identifier with the application identifier of each action list 140 in the identified subset. Through the comparison, the action handler component 120 may determine that the application identifier of the action list 140 matches the application identifier from the agent usage history. In response to the determination, the action handler component 120 may select an action list 140 for the action to be performed by the agent application 110 based on the parameter. Conversely, through the comparison, the action handler component 120 may determine that the application identifier of the action list 140 does not match the application identifier from the agent usage history. In response to the determination, the action handler component 120 may identify another action list 140 and may repeat the comparison using the application identifier.

[0085] By identifying the action list 140, the action handler component 120 can verify one or more parameters identified from the input audio signal against the address template of the action list 140. The verification of the action handler component 120 may check whether there are necessary parameters specified by the action list 140 to perform the action corresponding to the request. For example, for a carpool request, the action handler component 120 can check whether there are parameters corresponding to the destination parsed from the input audio signal. The action handler component 120 can use the parameter mapping for the address template of the action list 140 when verifying the parameters. The action handler component 120 can identify the correspondence between the parameters defined in the mapping for the action list 140 and the input variables. The action handler component 120 can identify the corresponding subset of input variables that are not specified as auxiliary through the mapping of the action list 140 for the address template.

[0086] From each input variable that is not represented as auxiliary, the action handler component 120 can determine whether at least one of the parameters corresponds to the input variable specified by the mapping. The determination of the correspondence can be performed using a semantic analysis algorithm (e.g., a semantic knowledge graph) provided by the NLP component 116. For each input variable, the action handler component 120 can call the NLP component 116 to determine whether the input variable corresponds to any parameter, or vice versa. In response to determining that all input variables correspond to at least one parameter, the action handler component 120 can determine that the parameter is successfully verified. The action handler component 120 can also continue to use the address template of the action list 140 to generate an address to implement the action corresponding to the request.

[0087] On the other hand, in response to determining that at least one input variable (a variable not indicated as auxiliary) does not correspond to any parameter, the action handler component 120 can determine that the parameter was not successfully verified. The action handler component 120 can stop or terminate further processing of the request and parameters (e.g., generating an address using an address template of the action list 140). The action handler component 120 can detect or determine an error when verifying one or more parameters identified from the input audio signal. The digital assistant application 108 can present an indication that the request is invalid or unsuccessful (e.g., using a visual content item or audio). The action handler component 120 can provide an indication of the error to the agent service 106 or an administrator of the agent application 110 associated with the action list 140.

[0088] After identifying the action list 140, the action handler component 120 can determine whether the client device 104 is authenticated by the proxy application 110 associated with the action list 140. The action handler component 120 can identify the proxy application 110 associated with the action list 140. Once identified, the action handler component 120 can use the proxy application 110 to check the authentication status of the client device 104. The check of the authentication status can be performed using the account identifier of the client device 104 or the digital assistant application 108. The action handler component 120 can send a request for the authentication status to the proxy service 106 that manages the resources of the proxy application 110. The request may include the account identifier of the client device 104 or the digital assistant application 108. The proxy service 106 can provide a response to the action handler component 120 indicating whether the client device 104 is authenticated by the proxy application 110. When the response indicates that the client device 104 has been authenticated, the action handler component 120 can determine that the client device 104 has been authenticated by the proxy application 110. The action handler component 120 may also continue to process the request. On the other hand, when the response indicates that the client device 104 is not authenticated, the action handler component 120 may determine that the client device 104 is not authenticated by the proxy application 110. The action handler component 120 may stop or terminate further processing of the request (e.g., generating an address using an address template of the action list 140). The digital assistant application 108 may also present an indication to authenticate with the proxy application 110 to perform an action corresponding to the request. For example, the digital assistant application 108 may render a prompt for logging into the proxy application 110 on the display 146 of the client device 104. The prompt may include fields for the user's account identifier and password.

[0089] In addition, the action handler component 120 can also determine whether the proxy application 110 associated with the action list 140 has been installed on the client device 104. The action handler component 120 can identify the proxy application 110 associated with the action list 140. Once identified, the action handler component 120 can access the client device 104 to determine whether the proxy application 110 has been installed on the client device 104. For example, the action handler component 120 can access a list of installed applications on the client device 104 to determine whether the proxy application 110 is installed. The action handler component 120 can check the application identifier or the proxy identifier of the list. In response to determining that the proxy application 110 is installed, the action handler component 120 can continue to process the request. On the other hand, in response to determining that the proxy application 110 is not installed, the action handler component 120 can terminate the processing of the request (for example, using the address template of the action list 140 to generate an address). The digital assistant application 108 can also present an instruction to install the proxy application 110 to perform the action corresponding to the request. For example, the digital assistant application 108 may render a prompt indicating that the corresponding agent application 110 is not installed on the display 146 of the client device 104. The prompt may also include a link to install the agent application 110.

[0090] The action handler component 120 can generate an address based on the address template of the action list 140. The action handler component 120 can identify the address template of the action list 140 selected from the collection maintained in the data repository 128. As described above, the address template of the action list 140 may include: a first part referencing the proxy application 110; and a second part referencing the action to be performed and one or more input variables for performing the action. For example, the address template may be a URI template. The first part may include a scheme referencing the proxy application 110 or a host name referencing the proxy service 106 or the proxy application 110. The second part may include a path name referencing the action and one or more query strings corresponding to one or more input variables in the URI. For example, for a transport order, the address template of the action list 140 may be "https: / / ex_transport.app example.com / order{?dropoffAddress}". The first part may correspond to "https: / / ex_trans port.appexample.com / ", and the second part may correspond to "order{?dropoffAddress}".

[0091] The address generated according to the address template may include a first substring and a second substring. The first substring of the generated address may correspond to the first part of the address template. The generated address may be part of a URI, and the first substring may include a scheme referencing the proxy application 110 or a host name referencing the proxy service 106 or the proxy application 110. The scheme and host name may be obtained by the proxy processor component 120 from the first part of the address template. The second substring of the generated address may correspond to the second part of the second template. The generated address may also be part of a URI, and the second substring may include a path name referencing the action performed by the proxy application 1110 and a query string including parameters identified from the input audio signal. For example, the address generated according to the address template may be "https: / / ex_transport.appexample.com / order?dropoffaddress=Anywhere,USA". The first substring of the address may include "https: / / ex_trans port.appexample.com / " and the second substring of the address may include "order?dropoffaddress=Anywhere,USA".

[0092] When generating an address, the action handler component 120 can use or set the first portion of the address template as a first substring of the address to reference the proxy service 106 or the proxy application 110 associated with the proxy service 106. The action handler component 120 can use the second portion of the address template as a second substring of the address to correspond to the action and the parameters for performing the action. The action handler component 120 can populate the input variables of the second portion of the address template with one or more parameters identified from the input audio signal.

[0093] The action handler component 120 can be filled according to the parameter mapping of the action list 140 defined for the address template. As described above, the parameter mapping can define the correspondence between the input variables of the second part and the words that will be recognized from the parsed input audio signal. The action handler component 120 can use the parameters as the words recognized from the input audio signal to insert them into the second part of the address template when filling the input variables. The filling of the second part of the address template can be based on the expression expansion in the URI template. For each input variable specified in the address template, the action handler component 120 can identify the corresponding parameter specified by the mapping of the action list 140. Through this identification, the action handler component 120 can insert, replace or set the query string by the name of the input variable and the identified parameter. For example, the input variable can be "{?dropoffAddress}" in the address template, and the identified parameter can be "Anywhere,USA". In this example, the action handler component 120 can replace "?dropoffAddress=Anywhere_USA" in the second substring of the address according to the expression expansion of the URI template.

[0094] When filling the input variables of the address template, the action handler component 120 can determine whether the input variable is auxiliary as specified in the mapping of the address template for the action list 140. In response to determining that the input variable is auxiliary, the action handler component 120 can also determine whether there are identified parameters corresponding to the auxiliary input variables. When it is determined that there are no parameters corresponding to the auxiliary input variables, the action handler component 120 can omit or remove the auxiliary input variables from the address. Therefore, the generated address may lack one or more parameters corresponding to the auxiliary input variables in the address template. Otherwise, when it is determined that there are parameters corresponding to the auxiliary input variables, the action handler component 120 can insert or fill the auxiliary input variables with the parameters. Conversely, in response to determining that the input variable is not auxiliary, the action handler component 120 can identify the parameters to be included in the input variable from the identified parameters.

[0095] As the address is generated according to the address template, the action handler component 120 can verify the address. The action handler component 120 verifying the address can be checking whether the address is in the correct format. For verification, the action handler component 120 can check the address against a regular expression 130 for an address (e.g., a URI or URL). The regular expression 130 for the address can include, for example, a scheme, a host name, a path name, and a query string separated by delimiters of a valid address. The action handler component 120 can compare the address with the regular expression 130 to determine whether the address matches the pattern of the address specified by the regular expression 130. In response to determining that the address matches, the action handler component 120 can determine that the generated address has been successfully verified.

[0096] In contrast, in response to determining that the address does not match, the action handler component 120 can determine that the generated address was not successfully verified. Based on this determination, the action handler component 120 can also detect an error when verifying the generated address. The error can indicate that the address format is incorrect. In addition, the action handler component 120 can stop or terminate further processing of the address (e.g., generating an action data structure). The action handler component 120 can provide an indication of the error to the agent service 106 or an administrator of the agent application 110 associated with the action list 140.

[0097] The action processor component 120 can use the address for performing the action to package or generate an action data structure. The generated action data structure can be according to an application programming interface (API). For example, the API can be specified by the operating system running on the client device 104, and can be used by both the digital assistant application 108 and the proxy application 110. The action data structure can be transmitted between the digital assistant application 108 and the proxy application 110 via the API. The generated action data structure can be according to the hypertext transfer protocol (HTTP). For example, the action data structure generated by the action processor component 120 can be an HTTP request (e.g., GET or POST function). The action data structure can include at least one header and at least one body. The head can include the device modality (e.g., smart phone, smart speaker, tablet computer, laptop computer and desktop machine) of the client device 104 that executes the digital assistant application 108 and the proxy application 110. The head can include a source address set to the client device 104. The head can include setting the generated address to the destination address. The body of the action data structure can include other information related to the request identified from the parsing input audio signal. For example, the subject may include an identifier that references the digital assistant application 108 or the client device 104 that runs the digital assistant application 108.

[0098] An instance of the digital assistant application 108 on the data processing system 102 or the client device 104 can execute the agent interface component 126 to provide the action data structure to the agent application 110. As the action data structure is generated, the agent interface component 126 can provide or direct the action data structure to the agent application 110 or the agent service 106 that manages the resources of the agent application 110. The agent interface component 126 can transmit the action data structure to the agent service 106 that manages the resources of the agent application 110 through the network 114 (e.g., as an HTTP request). For example, an HTTP request corresponding to the action data structure can be sent by the agent interface component 126 through the network 114. The agent interface component 126 can use the action data structure (e.g., via an API) to call the agent application 110. For example, the agent interface component 126 can make a function call to the agent application 110 according to the API and facilitated by the operating system of the client device 104. In the call, the agent interface component 126 can pass the action data structure to the agent application 110. When the proxy application 110 is called, the client device 104 can render and display the proxy application 110 on the display 148. For example, in response to the call, the proxy application 110 can enter the foreground of the proxy application 110 when the digital assistant application 108 or other application is switched to background processing.

[0099] By switching in this manner, user involvement in searching for the agent application 110 on the client device 104 may be eliminated, thereby improving the HCI of the digital assistant application 108. In addition, since the NLP component 116 has processed the input audio signal and the action processor component 120 has generated the action data structure, the natural language processing functionality may not be incorporated into the agent application 110, thereby saving processing power and disk space on the client device 104.

[0100] The data processing system 102, the client device 104, or the proxy service 106 may execute an instance of the proxy application 110. Because the destination address of the data structure may reference the proxy application 110 or the proxy service 106, the proxy application 110 may receive the action data structure from the proxy interface component 126. For example, the action data structure may have been sent via the network 114, and the destination address may include a host name that references the proxy application 110 or the proxy service 106. Because the destination address of the action data structure references the proxy application 110 or the proxy service 106, the action data structure may be routed to the proxy application 110 or the proxy service 106 by the network 114. In another example, the destination address of the action data structure may be a deep link with a solution that references the proxy application 110. By calling the destination address, the action data structure may be directed by the client device 104 or the network 114 to the proxy application 110 referenced by the solution.

[0101] Once the action data structure is received, the proxy application 110 can parse the action data structure to identify the address. The identified address can come from the destination address header of the action data structure. The proxy application 110 can parse the address to identify a first substring of the address of the proxy service 106 that references the proxy application 110 itself or manages resources for the proxy application 110. The proxy application 110 can parse the address of the second substring of the address. The proxy application 110 can further parse the second substring of the address to identify the action and one or more parameters for performing the action. The proxy application 110 can parse the path name of the URI to identify the action, and parse one or more query strings to identify one or more parameters for performing the action. For example, the proxy application 110 can parse the address "https: / / ex_trans port.appexample.com / order?dropoffaddress=Anywhere,USA" to identify the second substring "order?dropoffaddress=Anywhere,USA." The proxy application 110 can further identify "order" as a shipping request and "Anywhere,USA" as the destination location of the shipping request.

[0102] Using this identification, the agent application 110 can complete, implement or perform an action based on one or more parameters. Using the previous example, the agent application 110 can perform a series of processes and routines to perform a transportation request for the destination "Anywhere, USA". The agent service 106 of the agent application 110 can identify one or more available vehicles for the destination in the request, and can send a notification to the computing device (e.g., a smartphone) of one of the available vehicles to request transportation. In executing the request, the agent application 110 can generate an output to be presented on the client device 104. The generated output can include text, audio, or visual content items. In the previous example, the agent application 110 can use audio or visual content items to generate the output "Found a vehicle arriving in 5 minutes".

[0103] The proxy application 110 may also generate an output to indicate whether the execution of the action was successful or failed to send to the digital assistant application 108. When executing the proxy, the proxy application 110 may fail to complete the action or terminate before completion. For example, when processing a transportation request, the proxy service 106 may determine that there are no available vehicles near the client device 104. Based on this determination, the proxy application 110 may create an output indicating failure. In response to the failure, the proxy application 110 may generate an output indicating failure. Conversely, the proxy application 110 may complete the execution of the action. In response to completion, the proxy application 110 may generate an output indicating success.

[0104] When generated, the proxy application 110 can provide the output indicating success or failure to the digital assistant application 108. In turn, the action handler component 120 can receive the output from the proxy application 110. The digital assistant application 108 can also present the output as text, visual content items, or audio (e.g., using the audio signal generator component 118). The output presented by the digital assistant application 108 can indicate whether the proxy application 110 executes the request successfully or failed. When the output from the proxy application 110 indicates failure, the digital assistant application 108 can present the output indicating failure. On the contrary, when the output from the proxy application 110 indicates success, the digital assistant application 108 can present the output indicating success.

[0105] Figure 2 Depicted in Figure 1 In the system shown in FIG. 1 , a sequence diagram of an example data flow 200 for calling the functions of the agent application 110 via the digital assistant application 108 is shown. The data flow 200 may be a sequence diagram of an example data flow 200 for calling the functions of the agent application 110 via the digital assistant application 108. Figure 1 The system 100 described below may be combined with Figure 5 The detailed description is implemented or performed by system 500. Data flow 200 includes communications in the form of data packets (eg, HTTP messages or other forms) between microphone 144, client device 104, data processing system 102, and proxy service 106 via network 114.

[0106] The local instance of the digital assistant application 108 running on the client device 104 can receive audio data 205 (e.g., an input audio signal) from a microphone 144 that is communicatively coupled to the client device 104. The digital assistant application 108 on the client device 104 can perform initial processing (e.g., automatic speech recognition) on the audio data 205 to recognize one or more words from the audio data 205. The digital assistant application 108 can provide the one or more recognized words from the audio data 205 as input to the remote instance of the digital assistant application 108 running on the data processing system 102. In turn, the NLP component 116 on the data processing system 102 can further process the one or more words of the input 210 to recognize a request and one or more parameters. The request can indicate an action that the proxy application 110 is to perform based on the one or more parameters.

[0107] The action handler component 120 on the data processing system 102 can use the request and one or more parameters to identify the action list 140 from the data repository 128. The action handler component 120 can generate an address based on the address template of the identified action list 140. The address may include a first substring (e.g., a scheme or host name) that references the proxy application to perform the action. The address may additionally include a second string corresponding to the action to be performed and one or more parameters based on which the action is to be performed. The action handler component 120 can package the address into an action data structure to provide as input 215 to the proxy service 106 for the proxy application 110.

[0108] After receiving input 215, the proxy service 106 for the proxy application 110 can parse the action data structure to identify the first substring and the second substring. The proxy application 110 executed on the proxy service 106 can further parse the second substring to identify the action and one or more parameters. Through identification, the proxy application 110 can perform actions according to the parameters. In performing the action, the proxy application 110 can generate output 220 to provide to the digital assistant application 108. The proxy service 106 can provide the output 220 to the digital assistant application 108 on the data processing system 102. In turn, the digital assistant application 108 on the data processing system 102 can send the output 220 as output 225 to the client device 104. The digital assistant application 108 or the proxy application 110 running on the client device can present the output 225 from the proxy service 106.

[0109] Figure 3 Shown in Figure 1A use case diagram of an example client device 104 in which an input message is displayed and an address is generated in the system shown. The NLP component 116 of the digital assistant application 108 running on the client device 104 can receive an input audio signal via the microphone 144 of the client device 104. The NLP component 116 can apply automatic speech recognition to recognize one or more words from the input audio signal. For example, the NLP component 116 can recognize the words "Get me amedium flat white from the Café app" from the input audio signal. The digital assistant application 108 can display the recognized words in the content item 305 on the display 146 of the client device 104. Continuing with the previous example, the digital assistant application 108 can display the content item 305 with the text "Get me a medium flat white from the Café app" recognized by the NLP component 116.

[0110] The NLP component 116 on the client device 104 or the data processing system 102 may perform further processing on the words. Through further processing, the NLP component 116 may identify a request, one or more parameters describing the request, and an agent identifier corresponding to the agent application 110 that executes the request. In the illustrated example, the NLP component 116 may identify the word "Get" as a request, the words "medium" and "flat white" as parameters, and "Café app" as an agent identifier.

[0111] Using the identified request, parameters, and agent identifier, the action handler component 120 can select an action list 140 from the data repository 128. The action list 140 can be for actions corresponding to the request to be performed by the agent application 110 corresponding to the agent identifier according to the parameters. Corresponding to the agent identifier through the agent application 110. As shown, the action list 140 may include an address template "https: / / order-actions.cafeapp.com / order{?drinkname,?size}". The address template can be divided into a first part and a second part. The first part can reference the agent application 110 or the agent service 106 ("https: / / order-actions.cafeapp.com"). The second part can reference the path name (" / order"), the action corresponding to the request, and the parameters using the query string ("{?drinkname,?size}") for the drink name and size, respectively.

[0112] Based on the address template of the action list 140, the action handler component 120 can generate an address 310. The action handler component 120 can fill the second part of the address template with the identified parameters to generate the address. The address 310 may include a first substring and a second substring. The first substring may include the first part of the address template in the action list 140 ("https: / / order-actions.cafeapp.com / "). The second substring may include the path name of the second part of the action (" / order"). The second substring may also include query strings for the drink name and size, respectively (?drinkname="flat_white"&size="medium"). The action handler component 120 can generate an action data structure including the address 310. As the action data structure is generated, the action handler component 120 can direct the action data structure to the proxy application 110. The direction may include sending the action data structure (e.g., a URI for containing a domain name) via the network 114 or passing the action data structure by calling the proxy application 110 (e.g., a deep link with a scheme that references the proxy application 110).

[0113] Figure 4 A block diagram of an example method 400 for generating a voice activation thread in a networked computer environment is shown. The method 400 may be combined with the above Figure 1-3 The system 100 described below may be combined with Figure 5 The system 500 described in detail can be implemented or performed. In brief overview, the NLP component 116 can receive an input audio signal (405). The NLP component 116 can parse the input audio signal (410). The action handler component 120 can select an action list 140 (415). The action handler component 120 can use the action list 140 to generate an address (420). The action handler component 120 can generate an action data structure (425). The proxy interface component 126 can guide the action data structure (430).

[0114] In more detail, the NLP component 116 can receive an input audio signal (405). The NLP component 116 can receive one or more data packets including an input audio signal acquired at a sensor (e.g., microphone 144) on the client device 104. For example, the NLP component 116 of the digital assistant application 108 executed at least in part by the data processing system 102 can receive the input audio signal. The input audio signal can include a conversation facilitated by the digital assistant application 108. The conversation can include one or more inputs and outputs. The conversation can be audio-based, text-based, or a combination of audio and text. The input audio signal can include text input, or other types of input that can provide conversation information.

[0115] The NLP component 116 can parse the input audio signal (410). By parsing the input audio signal, the NLP component 116 can use various natural language processing techniques to identify a request and one or more parameters. The request can be an intent or request that the digital assistant application 108 or the agent application 110 can fulfill. The parameters can define the request. Based on the request, the NLP component 116 can determine that the request corresponds to one of the functions of the agent application 110.

[0116] The action handler component 120 may select an action list 140 (415). Based on the request identified from the input audio signal, the action handler component 120 may select an action list 140 from a collection maintained on the data repository 128. For each action list 140, the action handler component 120 may compare the request with a request identifier of the action list 140. The comparison may be performed using a semantic knowledge graph. Based on the comparison, the action handler component 120 may determine that the request matches the request identifier of the action list 140. In response to the determination, the action handler component 120 may select an action list 140 from a collection maintained on the data repository 128.

[0117] The action handler component 120 may generate an address using the action list 140 (420). The address may be generated according to an address template of the action list 140. The address template may have a first part and a second part. The first part may reference the proxy application 110 or a proxy service 106 for the proxy application 110 to perform the action. The second part may reference the action and may include one or more input variables of the action. The action handler component 120 may obtain the first part and the second part from the address template when generating the address. The action handler component 120 may insert or fill parameters into the input variables of the second part of the address template.

[0118] The action handler component 120 may generate an action data structure (425). The action data structure may include a header and a body. The action handler component 120 may insert an address generated according to an address template of the action manifest 140 into the header of the action data structure. The action handler component 120 may set the address as the destination address of the header to route the action data structure to the proxy application 110 or the proxy service 106 of the proxy application 110.

[0119] The proxy interface component 126 may direct the action data structure (430). The proxy interface component 126 may direct the action data structure to the proxy application 110. The proxy interface component 126 may use the action data structure to call the proxy application 110, and may pass the action data structure to the proxy application 110. In turn, the proxy application 110 may parse the action data structure to identify the action and one or more parameters from the address included in the action data structure. The proxy application 110 may perform the action based on the parameters.

[0120] Figure 5 5 is a block diagram of an example computer system 500. The computer system or computing device 500 may include or be used to implement the system 100 or its components, such as the data processing system 102. The computing system 500 includes a bus 505 or other communication component for transmitting information and a processor 510 or processing circuit coupled to the bus 505 to process information. The computing system 500 may also include one or more processors 510 or processing circuits coupled to the bus to process information. The computing system 500 also includes a main memory 515, such as a random access memory (RAM) or other dynamic storage device, which is coupled to the bus 505 to store information, and instructions to be executed by the processor 510. The main memory 515 may also be or include the data repository 128. The main memory 515 may also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 510. The computing system 500 may further include a read-only memory (ROM) 520 or other static storage device, which is coupled to the bus 505 to store static information and instructions for the processor 510. A storage device 525, such as a solid-state device, magnetic disk, or optical disk, may be coupled to bus 505 for persistent storage of information and instructions. Storage device 525 may include or be a part of data repository 128.

[0121] The computing system 500 may be coupled to a display 535 such as a liquid crystal display or an active matrix display via the bus 505 for displaying information to a user. An input device 530 such as a keyboard including alphanumeric and other keys may be coupled to the bus 505 for communicating information and command selections to the processor 510. The input device 530 may include a touch screen display 535. The input device 530 may also include a cursor control such as a mouse, trackball, or cursor direction keys for communicating direction information and command selections to the processor 510 and for controlling cursor movement on the display 535. For example, the display 535 may be a Figure 1 Part of data processing system 102, client device 104, or other component.

[0122] The processes, systems, and methods described herein can be implemented by the computing system 500 in response to the processor 510 executing an arrangement of instructions contained in the main memory 515. Such instructions can be read into the main memory 515 from another computer-readable medium such as the storage device 525. The arrangement of executing the instructions contained in the main memory 515 causes the computing system 500 to perform the exemplary processes described herein. One or more processors in a multi-processing arrangement can also be employed to execute the instructions contained in the main memory 515. Hard-wired circuits can be used in place of software instructions or in combination with software instructions in conjunction with the systems and methods described herein. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0123] although Figure 5 An example computing system has been described in the specification, but the subject matter including the operations described in this specification may be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them.

[0124] For situations where the systems discussed herein collect personal information about users or can make use of personal information, users may be provided with the opportunity to control whether programs or features can collect personal information (e.g., information about the user's social network, social actions or activities, the user's preferences, or the user's location) or control whether or how to receive content that may be more relevant to the user from a content server or other data processing system. In addition, certain data may be anonymized in one or more ways before being stored or used so that personally identifiable information is removed when generating parameters. For example, the identity of a user may be anonymized so that personally identifiable information about the user cannot be determined, or the geographic location of a user from which location information is obtained may be summarized (e.g., to the city, zip code, or state level) so that the user's specific location cannot be determined. , The user may control how the content server collects and uses information about him or her.

[0125] The subject matter and operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware including the structures disclosed in this specification and their structural equivalents or a combination of one or more thereof. The subject matter described in this specification may be implemented as one or more computer programs, such as one or more circuits of computer program instructions encoded on one or more computer storage media, for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Although a computer storage medium is not a propagation signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagation signal. A computer storage medium may also be or be included in one or more separate components or media (e.g., multiple CDs, disks or other storage devices). The operations described in this specification may be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0126] The terms "data processing system," "computing device," "component," or "data processing apparatus" encompass various apparatuses, devices, and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or multiple systems on a chip, or a combination of the foregoing. The apparatus may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. The components of system 100 may include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0127] A computer program (also referred to as a program, software, software application, application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for a computing environment. A computer program may correspond to a file in a file system. A computer program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0128] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs (e.g., components of data processing system 102) to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and the apparatus may also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated into, special purpose logic circuitry.

[0129] The subject matter described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or a combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), inter-networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0130] A computing system such as system 100 or system 500 may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network (e.g., network 114). The relationship between the client and the server is generated by computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., data packets representing content items) to a client device (e.g., in order to display data to a user interacting with the client device or to receive user input from it). Data generated at the client device (e.g., the result of a user interaction) may be received from the client device at the server (e.g., received by the data processing system 102 from the client device 104 or the proxy service 106).

[0131] Although operations are depicted in the drawings in a particular order, they need not be performed in the particular order shown or in sequential order, and not all operations shown need be performed. The actions described herein may be performed in a different order.

[0132] The separation of various system components need not require separation in all embodiments, and the described program components may be included in a single hardware or software product. For example, the NLP component 116, the audio signal generator component 118, the action processor component 120, the response selector component 122, the agent registry component 124, and the agent interface component 126 may be a single component, application or program, or a logical device having one or more processing circuits, or part of one or more servers of the data processing system 102.

[0133] Now that some exemplary embodiments have been described, it is apparent that the foregoing is exemplary and not restrictive, having been presented by way of example. In particular, although many of the examples presented herein relate to specific combinations of method actions or system elements, these actions and those elements may be combined in other ways to achieve the same goals. The actions, elements, and features discussed in conjunction with one embodiment are not intended to be excluded from similar roles in other embodiments.

[0134] The words and terms used herein are for descriptive purposes and should not be considered limiting. The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized by," and variations thereof are meant to encompass the items listed thereafter, their equivalents and additional items, and alternative embodiments consisting of the items listed thereafter. In one embodiment, the systems and methods described herein include one, more than one of each combination, or all of the described elements, actions, or components.

[0135] Any reference to the embodiments, elements or actions of the systems and methods herein mentioned in the singular may also encompass embodiments including a plurality of these elements, and any reference to any embodiments, elements or actions herein in the plural may also encompass embodiments including only one element. Reference in the singular or plural form is not intended to limit the currently disclosed systems or methods, their components, actions or elements to a single or multiple configurations. Reference to any action or element based on any information, action or element may include an embodiment where the action or element is based at least in part on any information, action or element.

[0136] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment," "some embodiments," "an embodiment," etc. are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in conjunction with an embodiment may be included in at least one embodiment or example. These terms as used herein do not necessarily all refer to the same embodiment. Any embodiment may include or be exclusively combined with any other embodiment in any manner consistent with the aspects and embodiments disclosed herein.

[0137] References to "or" may be interpreted as inclusive, such that any term described using "or" may indicate a single, more than one, and any of the described terms. References to at least one of "A" and "B" may include only "A," only "B," and both "A" and "B." Such references used in conjunction with "includes" or other open terms may include additional items.

[0138] Where technical features in the drawings, detailed description or any claims are followed by reference numerals, the reference numerals have been included to increase the intelligibility of the drawings, detailed description and claims. Therefore, neither the reference numerals nor their absence will have any limiting effect on the scope of any claim element.

[0139] The systems and methods described herein may be implemented in other specific forms without departing from their characteristics. The foregoing embodiments are exemplary rather than limiting of the described systems and methods. Therefore, the scope of the systems and methods described herein is indicated by the appended claims rather than the preceding description, and changes within the meaning and range of equivalents of the claims are hereby encompassed.

Claims

1. A system for calling an application function via a digital assistant application, comprising: A natural language processor executing on a data processing system having one or more processors for: receiving a data packet including an input audio signal detected by a sensor of a client device; as well as parsing the input audio signal of the data packets to identify a request to be performed by an application and to define parameters of the request; An action processor executed on the data processing system, for: An address for executing the request is generated according to an address template corresponding to the request and including input variables and auxiliary input variables, wherein: the action processor determining that there is no identified parameter corresponding to the auxiliary input variable; The address is at least a portion of a uniform resource identifier (URI); and the address comprises a first substring and a second substring, the first substring corresponding to the application and the second substring comprising one or more query strings, the one or more query strings comprising the parameter identified from the input audio signal as the input variable and lacking a parameter as the auxiliary input variable; and generating an action data structure including the address generated by the action handler according to the address template; and An interface executed on the data processing system for directing the action data structure to the application referenced by the address so that the application parses the action data structure, executes the request according to the parameters, and provides output for presentation on the client device.

2. The system of claim 1, wherein: The natural language processor is further configured to parse the input audio signal of the data packet to identify an application identifier corresponding to the application; and The action processor is used to select the address template based on the application identifier.

3. The system of claim 1, wherein: The natural language processor is further configured to parse the input audio signal of the data packet to determine that the input audio signal lacks an application identifier corresponding to any one of a plurality of applications; and The action handler is further configured to, in response to determining that the input audio signal lacks the application identifier, select the application from the plurality of applications based on the request, the parameters, and an application usage history of the client device, the application being capable of executing the request according to the parameters.

4. The system according to claim 1, wherein: The action processor is further configured to: validating the parameters against the address template corresponding to the request; In response to the request and the verification of the parameters, generating the address according to the address template; as well as In response to directing the action data structure to the application, receiving the output of the application from the execution of the request, the output indicating one of a success or a failure of the execution of the request.

5. The system according to claim 1, wherein: The data processing system is further configured to: verifying a second address generated according to the address template corresponding to the request; detecting an error when verifying the second address, the error indicating that the second address is incorrectly formatted; responsive to detecting the error, terminating directing a second action data structure including the second address to the application; as well as An indication of the error is provided to a proxy service for the application.

6. The system according to claim 1, wherein: The action processor is further configured to: in response to determining that the client device is not authenticated by the application, terminating generating at least one of a second address or a second action data structure including the second address according to the address template corresponding to the request; as well as Instructions are presented to authenticate the client device to the application to perform the second request.

7. The system according to claim 1, wherein: The action handler is further configured to direct the action data structure to the application referenced by the address so that the application: parsing the address of the action data structure to identify the first substring and the second substring; identifying the request and the parameters from the second substring of the address; as well as In response to identifying the request and the parameters, the request is executed according to the parameters.

8. The system according to claim 1, wherein: The interface is further configured to generate the address for executing the request according to the address template, the address including the first substring and the second substring, the first substring having a host name referencing a proxy service of the application.

9. The system according to claim 1, wherein: The first substring includes a scheme in a custom deep link URI that references the application.

10. The system according to claim 1, wherein: The second substring further includes a pathname referencing the request to be executed by the application.

11. A method for invoking an application function via a digital assistant application, comprising: receiving, by a data processing system having one or more processors, a data packet including an input audio signal detected by a sensor of a client device; parsing, by the data processing system, the input audio signal of the data packets to identify a request to be performed by an application and to define parameters of the request; The data processing system generates an address for executing the request according to an address template corresponding to the request and including input variables and auxiliary input variables, wherein: Generating the address for executing the request includes determining that there is no identified parameter corresponding to the auxiliary input variable; The address is at least a portion of a uniform resource identifier (URI); and the address comprises a first substring and a second substring, the first substring corresponding to the application and the second substring comprising one or more query strings, the one or more query strings comprising the parameters identified from the input audio signal as the input variables and lacking parameters as the auxiliary input variables; generating, by the data processing system according to the address template, an action data structure including the address generated by the action processor; and The action data structure is directed, by the data processing system, to the application referenced by the address, so that the application parses the action data structure, executes the request according to the parameters, and provides output for presentation on the client device.

12. The method according to claim 11, further comprising: parsing the input audio signal of the data packet to identify an application identifier corresponding to the application; as well as The address template is selected based on the application identifier.

13. The method according to claim 11, further comprising: parsing the input audio signal of the data packet to determine that the input audio signal lacks an application identifier corresponding to any one of a plurality of applications; as well as In response to determining that the input audio signal lacks the application identifier, the application is selected from the plurality of applications based on the request, the parameters, and an application usage history of the client device, the application being capable of executing the request according to the parameters.

14. The method according to claim 11, further comprising: validating the parameters against the address template corresponding to the request; In response to the request and the verification of the parameters, generating the address according to the address template; as well as In response to directing the action data structure to the application, receiving the output of the application from the execution of the request, the output indicating one of a success or a failure of the execution of the request.

15. The method according to claim 11, further comprising: verifying a second address generated according to the address template corresponding to the request; detecting an error when verifying the second address, the error indicating that the second address is incorrectly formatted; responsive to detecting the error, terminating directing a second action data structure including the second address to the application; as well as An indication of the error is provided to a proxy service for the application.

16. The method according to claim 11, further comprising: in response to determining that the client device is not authenticated by the application, terminating generating at least one of a second address or a second action data structure including the second address according to the address template corresponding to the request; as well as Instructions are presented to authenticate the client device to the application to perform the second request.

17. The method of claim 11, further comprising directing the action data structure to the application referenced by the address so that the application: parsing the address of the action data structure to identify the first substring and the second substring; identifying the request and the parameters from the second substring of the address; and In response to identifying the request and the parameters, the request is executed according to the parameters.

18. The method of claim 11, further comprising generating the address for executing the request according to the address template, the address including the first substring and the second substring, the first substring having a host name of a proxy service referencing the application.

19. A method implemented by one or more processors, the method comprising: receiving, by a data processing system, a data packet including an input audio signal detected by a sensor of a client device; parsing, by the data processing system, the input audio signal of the data packet to identify a request and parameters associated with the request; selecting, by the data processing system, an agent to perform the request, wherein the agent is selected based on being able to perform an action corresponding to the request according to the parameter and further based on a history of agent usage by the client device; selecting, by the data processing system, an address template, the address template being selected based on the agent selected by the data processing system and further based on the action corresponding to the request; generating, by the data processing system, an address according to the address template selected by the data processing system, the address being at least a portion of a uniform resource identifier (URI), and the address being generated based on (i) the agent selected by the data processing system, (ii) the action corresponding to the request, and (iii) the parameters associated with the request; and The data processing system uses the address generated according to the address template to cause the agent to perform the action corresponding to the request according to the parameters.

20. The method according to claim 19, wherein: The proxy is not specified in the input audio signal.

21. The method of claim 19, further comprising validating the parameters against the address template selected by the data processing system, in, Generating the address is responsive to validating the parameters against the address template selected by the data processing system.

22. The method of claim 21, further comprising receiving an output of the agent performing the action corresponding to the request according to the parameter, the output indicating one of a success or a failure of the execution of the request.

23. The method of claim 19, further comprising: verifying a second address generated according to the address template selected by the data processing system; detecting an error when verifying the second address, the error indicating that the second address is incorrectly formatted; based on detecting the error in validating the second address, refraining from using the second address to cause the proxy to perform the action corresponding to the request according to the parameter; as well as An indication of the error is provided to the proxy service.

24. The method according to claim 19, wherein: The agent is an application or a service.

25. A computer program product, the computer program product comprising one or more computer-readable storage media having program instructions stored collectively on the one or more computer-readable storage media, the program instructions being executable to: receiving, by a data processing system, a data packet including an input audio signal detected by a sensor of a client device; parsing, by the data processing system, the input audio signal of the data packet to identify a request and parameters associated with the request; selecting, by the data processing system, an agent to perform the request, wherein the agent is selected based on being able to perform an action corresponding to the request according to the parameter and further based on a history of agent usage by the client device; selecting, by the data processing system, an address template, the address template being selected based on the agent selected by the data processing system and further based on the action corresponding to the request; generating, by the data processing system, an address according to the address template selected by the data processing system, the address being at least a portion of a uniform resource identifier (URI), and the address being generated based on (i) the agent selected by the data processing system, (ii) the action corresponding to the request, and (iii) the parameters associated with the request; and The data processing system uses the address generated according to the address template to cause the agent to perform the action corresponding to the request according to the parameters.

26. The computer program product of claim 25, wherein: The proxy is not specified in the input audio signal.

27. The computer program product of claim 25, wherein: The program instructions are further executable to validate the parameters against the address template selected by the data processing system; and Generating the address is responsive to validating the parameters against the address template selected by the data processing system.

28. The computer program product of claim 27, wherein: The program instructions are further executable to receive an output of the agent performing the action corresponding to the request according to the parameter, the output indicating one of a success or a failure of the execution of the request.

29. The computer program product of claim 25, wherein: The program instructions are further executable to: verifying a second address generated according to the address template selected by the data processing system; detecting an error when verifying the second address, the error indicating that the second address is incorrectly formatted; based on detecting the error in validating the second address, refraining from using the second address to cause the proxy to perform the action corresponding to the request according to the parameter; as well as An indication of the error is provided to the proxy service.

30. The computer program product of claim 25, wherein: The agent is an application or a service.

31. A system comprising: A processor, a computer readable memory, one or more computer readable storage media, and program instructions stored together on the one or more computer readable storage media, the program instructions being executable to: receiving, by a data processing system, a data packet including an input audio signal detected by a sensor of a client device; parsing, by the data processing system, the input audio signal of the data packet to identify a request and parameters associated with the request; selecting, by the data processing system, an agent to perform the request, wherein the agent is selected based on being able to perform an action corresponding to the request according to the parameter and further based on a history of agent usage by the client device; selecting, by the data processing system, an address template, the address template being selected based on the agent selected by the data processing system and further based on the action corresponding to the request; generating, by the data processing system, an address according to the address template selected by the data processing system, the address being at least a portion of a uniform resource identifier (URI), and the address being generated based on (i) the agent selected by the data processing system, (ii) the action corresponding to the request, and (iii) the parameters associated with the request; and The data processing system uses the address generated according to the address template to cause the agent to perform the action corresponding to the request according to the parameters.

32. The system of claim 31, wherein: The proxy is not specified in the input audio signal.

33. The system of claim 31, wherein: The program instructions are further executable to validate the parameters against the address template selected by the data processing system; and Generating the address is responsive to validating the parameters against the address template selected by the data processing system.

34. The system of claim 33, wherein: The program instructions are further executable to receive an output of the agent performing the action corresponding to the request according to the parameter, the output indicating one of a success or a failure of the execution of the request.

35. The system of claim 31, wherein: The program instructions are further executable to: verifying a second address generated according to the address template selected by the data processing system; detecting an error when verifying the second address, the error indicating that the second address is incorrectly formatted; based on detecting the error in validating the second address, refraining from using the second address to cause the proxy to perform the action corresponding to the request according to the parameter; as well as An indication of the error is provided to the proxy service.

36. The system of claim 31, wherein: The agent is an application or a service.

Images