Interacting with applications via dynamically updating natural language processing

Abstract

The invention relates to interacting with applications via dynamically updated natural language processing. For example, a system receives a first input audio signal. The system processes a first input audio signal via a natural language processing technique to identify an application. A system activates an application for execution on a client computing device. The application declares a function that the application is configured to perform. The system modifies the natural language processing technique in response to the functionality declared by the application. The system receives a second input audio signal. The system processes the second input audio signal via the modified natural language processing technique to detect one or more parameters. The system determines that the one or more parameters are compatible with the input of the input field of the application. A system generates an action data structure for an application. The system inputs the action data structure into an application executing the action data structure.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980101548.6, filed on November 27, 2019. Background Art

[0003] The computing device can execute applications. The application can provide a user interface that can receive input information, perform functions, and output information. Summary of the invention

[0004] The technology solution generally involves interacting with applications such as third-party (3P) applications via dynamically updated natural language processing ("NLP") or natural language understanding. By dynamically updating NLP, the technology solution can facilitate voice-based interaction with applications (including 3P applications) via a digital assistant.

[0005] Interacting with an application to perform actions on a computing device with or using limited interfaces can be challenging. Using additional interfaces to communicate with an application can increase computing resource consumption, require the use of excessive interfaces, or delay or otherwise introduce latency into the execution and completion of the application's actions.

[0006] Thus, the technical solution may utilize a protocol to receive declarations from an application. The technical solution may use the declarations to update natural language understanding or processing technology used to parse or process voice input in real time (e.g., in response to invocations by the application and receipt of the declarations). The technical solution may use updated NLP technology to process subsequent voice inputs to improve the accuracy, reliability, and likelihood of detecting input parameters used by the application to perform an action.

[0007] At least one aspect of the present technical solution relates to a system for interacting with an application. The system may include a digital assistant component executed by a client computing device. The digital assistant component may include one or more processors and memory. The digital assistant component may execute a natural language processor component, an application manager component, and a direct action application programming interface ("API"). The NLP component may receive a first input audio signal detected by a microphone of the client computing device. The NLP component may process the first input audio signal via a natural language processing technique to identify an indication of an application. The application manager component may activate the application to execute on the client computing device in response to the indication. The application may declare to the digital assistant via a protocol the function that the application is configured to execute. The NLP component may modify the natural language processing technique in response to the function declared by the application. The NLP component may receive a second input audio signal detected by a microphone of the client computing device. The NLP component may process the second input audio signal via the modified natural language processing technique to detect one or more parameters. The NLP component may determine that the one or more parameters are compatible for input to an input field of an application activated by the digital assistant on the client computing device based on a comparison between the one or more parameters and the function declared by the application. The direct action API may generate an action data structure for the application in response to the determination of compatibility and based on the one or more parameters. The direct action API may input an action data structure into an application via a protocol to cause the application to execute the action data structure and update a user interface element of the application.

[0008] At least one aspect of the present technical solution relates to a method for dynamically interacting with an application. The method may be performed by one or more digital assistant components executed by a client computing device. The method may include the digital assistant component receiving a first input audio signal detected by a microphone of the client computing device. The method may include the digital assistant component processing the first input audio signal via a natural language processing technique to identify an indication of an application. The method may include the digital assistant component activating the application to execute on the client computing device in response to the indication. The application declares to the digital assistant via a protocol the function that the application is configured to execute. The method may include the digital assistant component modifying the natural language processing technique in response to the function declared by the application. The method may include the digital assistant component receiving a second input audio signal detected by a microphone of the client computing device. The method may include the digital assistant component processing the second input audio signal via the modified natural language processing technique to detect one or more parameters. The method may include the digital assistant component determining that one or more parameters are compatible for input to an input field of an application activated by a digital assistant on the client computing device based on a comparison between the one or more parameters and the function declared by the application. The method may include the digital assistant component generating an action data structure for the application in response to the determination of compatibility and based on the one or more parameters. The method may include the digital assistant component inputting the action data structure into the application via the protocol, so that the application executes the action data structure and updates the user interface element of the application.

[0009] At least one aspect relates to a system for dynamically interacting with an application. The system may include one or more processors and memory. For example, the system may include a data processing system executed by a client computing device. The system may include a digital assistant component executed by one or more processors and memory. The digital assistant component may activate an application for execution on a client computing device in response to a first input audio. The activation of the application may cause the application to declare to the digital assistant component via a protocol the function that the application is configured to execute. The digital assistant component may modify a natural language processing technique in response to the function declared by the application. The digital assistant component may receive a second input audio signal detected by a microphone of the client computing device. The digital assistant component may process the second input audio signal via the modified natural language processing technique to detect one or more parameters. The digital assistant component may determine that one or more parameters are compatible for input to an input field of an application activated by a digital assistant on a client computing device based on a comparison between the one or more parameters and the function declared by the application. The digital assistant component may generate an action data structure for the application in response to a determination of compatibility and based on the one or more parameters. The digital assistant component may input the action data structure into the application via a protocol so that the application executes the action data structure and updates a user interface element of the application. The digital assistant component can transmit a request for content to the remote data processing system. The digital assistant component can present a content item selected by the remote data processing system in response to receiving the content item from the remote data processing system.

[0010] At least one aspect relates to a method for dynamically interacting with an application. The method may be performed by one or more processors and memories. For example, the method may be performed by a data processing system executed by a client computing device. The method may be performed by a digital assistant component executed by one or more processors and memories. The method may include a digital assistant component activating an application for execution on a client computing device in response to a first input audio. The activation of the application may cause the application to declare to the digital assistant component via a protocol the function that the application is configured to execute. The method may include the digital assistant component modifying a natural language processing technique in response to the function declared by the application. The method may include the digital assistant component receiving a second input audio signal detected by a microphone of the client computing device. The method may include the digital assistant component processing the second input audio signal via the modified natural language processing technique to detect one or more parameters. The method may include the digital assistant component determining that one or more parameters are compatible for input to an input field of an application activated by a digital assistant on a client computing device based on a comparison between the one or more parameters and the function declared by the application. The method may include the digital assistant component generating an action data structure for the application in response to a determination of compatibility and based on the one or more parameters. The method may include the digital assistant component inputting the action data structure into the application via a protocol so that the application executes the action data structure and updates a user interface element of the application. The method may include the digital assistant component transmitting a request for content to the remote data processing system. The method may include the digital assistant component presenting a content item selected by the remote data processing system in response to receiving the content item from the remote data processing system.

[0011] These and other aspects and embodiments are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of the various aspects and embodiments, and provide an overview or framework for understanding the nature and character of the claimed aspects and embodiments. The accompanying drawings provide illustration and further understanding of the various aspects and embodiments, and are incorporated into and constitute a part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings are not intended to be drawn to scale. Like reference numbers and names in the various drawings indicate like elements. For clarity, not every component may be labeled in every drawing. In the drawings:

[0013] Figure 1 is an illustration of a system for dynamically interacting with applications, according to an implementation.

[0014] Figure 2 is an illustration of the operation of a system for dynamically interacting with applications, according to an embodiment.

[0015] Figure 3is an illustration of a method of dynamically interacting with an application according to an implementation.

[0016] Figure 4 The diagram can be used to implement Figure 1 and Figure 2 The components of the system depicted in Figure 3 A block diagram of the general architecture of a computer system in which the method is depicted. DETAILED DESCRIPTION

[0017] The following is a more detailed description of various concepts and implementations related to methods, devices, and systems for dynamic application interaction. The various concepts introduced above and discussed in more detail below can be implemented in any of a variety of ways.

[0018] This technical solution generally involves interacting with third-party applications. The system and method of this technical solution can interact with (3P) applications by dynamically updating natural language processing technology (or natural language understanding). In a voice-based computing environment, such as when the input mode is based on voice input (e.g., an audio input signal detected by a microphone of a computing device), it may be challenging to interact with 3P applications to perform actions. For example, a digital assistant executed on a mobile computing device (e.g., a smart phone) can receive voice input. The digital assistant can call or start an application for execution on a smart phone (or at least partially executed on a smart phone). However, the application may not be configured with a voice interface (e.g., a user interface that allows voice input or natural language processing) to allow input parameters to be received via voice input. In the absence of input parameters, the application may not be able to perform an action, or may not be able to perform an action accurately and reliably. With a partial set of input parameters, the application may perform an action incorrectly (e.g., output an error or an unexpected result), or perform an action incorrectly (e.g., perform an incorrect action). In some cases, an application may request additional parameters in order to perform an action using a non-voice based interface (e.g., a display device, a touch input interface, a gesture interface, an icon, a button, a drop-down menu, a keyboard, or a mouse). Thus, without a complete set of input parameters, the application may perform the wrong action, or activate an additional interface, which may result in wasted computing resource utilization (e.g., additional power consumption, computing processing) and introduce delays in the execution or completion of the action by the application (e.g., additional requests, prompts, remote procedure calls to obtain input parameters).

[0019] Therefore, the systems and methods of the technical solution generally involve interacting with third-party ("3P") applications via dynamically updating natural language processing ("NLP") or natural language understanding. By dynamically updating NLP, the technical solution can facilitate voice-based interaction with 3P applications via digital assistants. By providing voice-based interaction, technical limitations associated with alternative interaction methods (e.g., touch) can be overcome for 3P applications: for example, users who may have difficulty handling touch interfaces due to physical constraints may benefit. The systems and methods of the technical solution can utilize a protocol to receive declarations from an application. The technical solution can use the declaration to update the natural language understanding or processing technology used to parse or process voice input in real time (e.g., in response to the call of the application and the receipt of the declaration). The technical solution can use the updated NLP technology to process subsequent voice input in order to improve the accuracy, reliability, and possibility of detecting input parameters used by the application to perform actions.

[0020] For example, a voice input to a smart phone may include "book a table at Restaurant_Booking_App". The digital assistant may recognize the mobile application Restaurant_Booking_App in the voice input and then launch the Restaurant_Booking_App mobile application on the smart phone. The Restaurant_Booking_App mobile application may declare to the digital assistant via the protocol what functions the application may perform or the state of the user interface of the application. Using this information, the digital assistant may better understand subsequent voice input or voice queries related to the Restaurant_Booking_App application. For example, if the subsequent voice input is "at Example_Italian_Restaurant", the digital assistant may determine that the voice input is a continuation of a reservation via the Restaurant_Booking_App application. The digital assistant may then generate an action data structure with information or parameters to make the reservation and transmit the action data structure to the Restaurant_Booking_App application via the protocol.

[0021] The technical solution can determine the context of the application based on the current state of the application, modify the NLP technology to improve the ability of the digital assistant to detect input parameters in subsequent voice input that are compatible with the current state or context of the application, and then input these parameters into the application to allow the application to perform actions efficiently, accurately and reliably. By providing input parameters used by the application to perform actions, the technical solution can reduce or eliminate the number of prompts or requests for input made by the application, while limiting or eliminating the use of different user interfaces (e.g., display devices, touch screens, gestures, keyboards, or mice). Therefore, the system and method of the technical solution can provide a seamless process of launching an application using voice input and performing actions by the application. In this way, the beneficial aspects of a voice interface can be provided to the application.

[0022] Figure 1 An exemplary system 100 that dynamically interacts with an application is illustrated. The system 100 may include a content selection infrastructure. The system 100 may include a data processing system 102. The data processing system 102 may include a computing device 128 or execute on a computing device 128. The data processing system 102 may communicate with one or more of a 3P content provider device 140, a remote data processing system 142, or a 3P application server 148 via a network 105. The network 105 may include a computer network such as the Internet, a local area network, a wide area network, a metropolitan area network, or other regional networks, an intranet, a satellite network, and other communication networks such as a voice or data mobile phone network. The network 105 may be used to access information resources such as web pages, websites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed on at least one computing device 128 (such as a laptop, a desktop computer, a tablet computer, a personal digital assistant, a smart phone, a portable computer, or a speaker). For example, via the network 105, a user of the computing device 128 may access information or data provided by the 3P content provider device 140. Computing device 128 may or may not include a display; for example, a computing device may include a limited type of user interface, such as a microphone and a speaker. In some cases, the primary user interface of computing device 128 may be a microphone and a speaker. Computing device 128 may interface with or be included in a voice-based computing environment.

[0023] Network 105 may be used by data processing system 102 to access information resources, such as applications, web pages, websites, domain names, or uniform resource locators, that may be presented, output, rendered, or displayed by client computing device 128. For example, via network 105, a user of client computing device 128 may access information or data provided by 3P content provider device 140. Network 105 may include or constitute a subnetwork of information resources available on the Internet that are associated with a content placement or search engine results system or that are eligible to include a third-party digital component as part of a digital component placement activity.

[0024] The network 105 may be any type or form of network, and may include any of the following: a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an ATM (asynchronous transfer mode) network, a SONET (synchronous optical network) network, an SDH (synchronous digital hierarchy) network, a wireless network, and a wired network. The network 105 may include a wireless link, such as an infrared channel or a satellite band. The topology of the network 105 may include a bus, a star, or a ring network topology. The network may include a mobile phone network using any one or more protocols for communicating between mobile devices, including an Advanced Mobile Phone Protocol ("AMPS"), a Time Division Multiple Access ("TDMA"), a Code Division Multiple Access ("CDMA"), a Global System for Mobile Communications ("GSM"), a General Packet Radio Service ("GPRS"), or a Universal Mobile Telecommunications System ("UMTS"). Different types of data may be transmitted via different protocols, or the same type of data may be transmitted via different protocols.

[0025] System 100 may include, interface with, communicate with, or otherwise access 3P application server 148. Data processing system 102 may communicate with 3P application server 148 via network 105. 3P application server 148 may be remote and distinct from data processing system 102, remote data processing system 142, 3P digital content provider device 140, and computing device 128. 3P application server 148 may be associated with a developer of application 118. 3P application server 148 may facilitate execution of application 118. For example, 3P application server 148 may perform back-end processing for application 118. Application 118 executing on computing device 128 may execute front-end components of application 118, and 3P application server 148 may execute back-end components of application 118. Applications 118 executing on computing device 128 may communicate remote procedure calls or other requests or data to 3P application server 148. 3P application server 148 may communicate data, information, or requests to applications 118 to perform functions or execute actions. 3P application server 148 may modify or change the state of application 118, which may result in the application declaring functionality of the application or invocation of digital assistant component 106.

[0026] System 100 may include at least one remote data processing system 142. Remote data processing system 142 may include at least one logical device, such as a computing device having a processor that communicates with, for example, data processing system 102, 3P content provider device 140 (e.g., content provider) via network 105. Remote data processing system 142 may include at least one computing resource, server, processor, or memory. For example, remote data processing system 142 may include multiple computing resources or servers located in at least one data center. Remote data processing system 142 may include multiple logically grouped servers and facilitate distributed computing techniques. Logical groups of servers may be referred to as data centers, server farms, or machine farms. Servers may also be geographically dispersed. A data center or machine farm may be managed as a single entity, or a machine farm may include multiple machine farms. Servers within each machine farm may be heterogeneous—one or more of the servers or machines may operate according to one or more types of operating system platforms.

[0027] The servers in the machine farm can be stored in a high-density rack system along with the associated storage systems and located in an enterprise data center. Consolidating servers in this manner can improve system manageability, data security, physical security of the system, and system performance by locating the servers and high-performance storage systems on a localized high-performance network, for example. Centralization of all or some of the remote data processing system 142 components, including servers and storage systems, and coupling them with advanced system management tools allows for more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage.

[0028] The remote data processing system 142 may include a digital assistant server 144 and a content selector component 146. The digital assistant server 144 may be designed, constructed, and operated to perform one or more functions of communicating with the digital assistant component 106 of the computing device 128. The content selector component 146 may be designed, constructed, and operated to select digital component items (e.g., content items) provided by the 3P digital content provider device 140. The content selector component 146 may select content items in response to a request for content from the computing device 128. The content selector component 146 may transmit the selected content items to the computing device 128 for presentation (e.g., audio output, visual output, or audio-visual output).

[0029] System 100 may include, access, or otherwise interact with at least one 3P content provider device 140. 3P content provider device 140 may include at least one logic device such as a computing device with a processor to communicate with, for example, computing device 128, data processing system 102, or remote data processing system 142 via network 105. 3P content provider device 140 may include at least one computing resource, server, processor, or memory. For example, 3P content provider device 140 may include multiple computing resources or servers located in at least one data center. 3P content provider device 140 may include or refer to a service provider device or a commodity provider device.

[0030] The 3P digital content provider device 140 can provide an audio-based digital component to be presented as an audio output digital component by the computing device 128. The digital component can include a response to a search query or request. The digital component can include information from a database, a search engine, or a networked resource. For example, the digital component can include news information, weather information, sports information, an encyclopedia entry, a dictionary entry, or information from a digital textbook. The digital component can include an offer for a product or service, such as a voice-based message that states: "Would you like me to order you a taxi?" The 3P content provider device 140 can include a memory to store a series of audio digital components that can be provided in response to a voice-based query. The 3P content provider device 140 can also provide an audio-based digital component (or other digital components) to the data processing system 102, where they can be stored in the data repository 122. The data processing system 102 can select an audio digital component and provide (or instruct the content provider computing device 140 to provide) the audio digital component to the client computing device 128. The audio-based digital component can be exclusively audio, or can be combined with text, image, or video data. A digital component or content item may include images, text, video, multimedia, or other types of content in one or more formats.

[0031] The remote data processing system 142 may include a content placement system having at least one computing resource or server. The remote data processing system 142 may include, interface with, or otherwise communicate with at least one content selector component 146. The remote data processing system 142 may include, interface with, or otherwise communicate with at least one digital assistant server 144.

[0032] The content selector component 146 and the digital assistant server 144 may each include at least one processing unit or other logic device, such as a programmable logic array engine, or a module configured to communicate with each other or with other resources or databases. The content selector component 146 and the digital assistant server 144 may be separate components, a single component, or part of a remote data processing system 142. The system 100 and its components (such as the remote data processing system 142) may include hardware elements, such as one or more processors, logic devices, or circuits.

[0033] Remote data processing system 142 may obtain anonymous computer network activity information associated with multiple computing devices 128. A user of computing device 128 may affirmatively authorize remote data processing system 142 to obtain network activity information corresponding to the user's computing device 128. For example, remote data processing system 142 may prompt the user of computing device 128 to consent to obtaining one or more types of network activity information. The identity of the user of computing device 128 may remain anonymous, and computing device 128 may be associated with a unique identifier (e.g., a unique identifier of the user or computing device provided by the data processing system or the user of the computing device). Remote data processing system 142 may associate each observation with a corresponding unique identifier.

[0034] 3P digital content provider device 140 can establish electronic content activities. Electronic content activities can be stored as content data in the data repository of content selector component 146. Electronic content activities can refer to one or more content groups corresponding to a common theme. Content activities can include a hierarchical data structure, which includes content groups, digital component data objects, and content selection criteria. In order to create a content activity, 3P digital content provider device 140 can specify the value of the activity level parameter of the content activity. The activity level parameter can include, for example, the name of the activity, the preferred content network for placing digital component objects, the value of the resources to be used for the content activity, the start and end dates of the content activity, the duration of the content activity, the schedule for the placement of digital component objects, the language, the geographic location, and the type of computing device on which the digital component objects are provided. In some cases, an impression can refer to when a digital component object is obtained from its source (e.g., a remote data processing system 142 or a 3P digital content provider device 140) and is countable. In some cases, robot activities can be filtered and excluded as impressions due to the possibility of click fraud. Thus, in some cases, a flash may refer to a measurement of a response from a web server to a page request from a browser, filtered from robot activity and error codes, and recorded at a point as close as possible to the opportunity to render a digital component object for display on computing device 128. In some cases, a flash may refer to a visual or audible flash; for example, a digital component object is at least partially (e.g., 20%, 30%, 30%, 40%, 50%, 60%, 70%, or more) viewable on a display device of client computing device 128, or audible via speaker 136 of computing device 128. A click or selection may refer to a user interaction with a digital component object, such as a voice response to an audible flash, a mouse click, a touch interaction, a gesture, a shake, an audio interaction, or a keyboard click. A conversion may refer to a user taking a desired action with reference to a digital component object; for example, purchasing a product or service, completing a survey, visiting a physical store corresponding to a digital component, or completing an electronic transaction.

[0035] The 3P digital content provider device 140 may also establish one or more content groups for a content campaign. A content group includes one or more digital component objects and corresponding content selection criteria, such as keywords, words, terms, phrases, geographic locations, types of computing devices, time of day, interests, themes, or verticals. Content groups under the same content campaign may share the same campaign-level parameters, but may have customized specifications for specific content group-level parameters, such as keywords, negative keywords (e.g., preventing the placement of digital components in the presence of negative keywords on the primary content), bidding for keywords, or parameters associated with bidding or content campaigns.

[0036] To create a new content group, the 3P digital content provider device 140 can provide values ​​for content group level parameters of the content group. The content group level parameters include, for example, a content group name or content group theme, and bids or results (e.g., clicks, impressions, or conversions) for different content placement opportunities (e.g., automatic placement or managed placement). The content group name or content group theme can be one or more terms that the 3P digital content provider device 140 can use to capture the topic or theme for which the digital component objects of the content group are to be selected for display. For example, a car dealer can create a different content group for each brand of vehicle it sells, and can also create a different content group for each model of vehicle it sells. Examples of content group themes that car dealers can use include, for example, "Make A sportscar", "Make B sports car", "Make C sedan", "Make C truck", "Make C hybrid", or "Make D hybrid". An exemplary content campaign theme may be "hybrid" and include content groups for both "Make C hybrid" and "Make D hybrid," for example.

[0037] 3P digital content provider device 140 can provide one or more keywords and digital component objects to each content group. Keywords can include terms related to products or services associated with or identified by digital component objects. Keywords can include one or more terms or phrases. For example, a car dealer can include "sports car", "V-6 engine", "four-wheel drive", "fuel efficiency" as keywords for content groups or content activities. In some cases, negative keywords can be specified by the content provider to avoid, prevent, block or disable content placement for certain terms or keywords. The content provider can specify a match type for selecting a digital component object, such as an exact match, a phrase match or a broad match.

[0038] The 3P digital content provider device 140 may provide one or more keywords to be used by the remote data processing system 142 to select digital component objects provided by the 3P digital content provider device 140. The 3P digital content provider device 140 may identify one or more keywords to bid on and further provide bid amounts for the various keywords. The 3P digital content provider device 140 may provide additional content selection criteria to be used by the remote data processing system 142 to select digital component objects. Multiple 3P digital content provider devices 140 may bid on the same or different keywords, and the remote data processing system 142 may run a content selection process or an advertising auction in response to receiving an indication of the keywords of the electronic message.

[0039] 3P digital content provider device 140 can provide one or more digital component objects for selection by remote data processing system 142. Remote data processing system 142 (e.g., via content selector component 146) can select digital component objects that match resource allocation, content scheduling, maximum bid, keywords, and other selection criteria specified for a content group when a content placement opportunity becomes available. Different types of digital component objects can be included in a content group, such as voice digital components, audio digital components, text digital components, image digital components, video digital components, multimedia digital components, or digital component links. When a digital component is selected, data processing system 102 can transmit the digital component object for presentation via computing device 128, rendering on computing device 128 or a display device of computing device 128. Rendering can include displaying the digital component on a display device, or playing the digital component via a speaker of computing device 128. Remote data processing system 142 can provide instructions to computing device 128 to render the digital component object. The remote data processing system 142 may instruct the digital assistant component 106 of the computing device 128 or the audio driver 138 of the computing device 128 to generate an audio signal or sound wave. The remote data processing system 142 may instruct the application 118 executed by the computing device 128 to render the selected digital component object. For example, the application 118 may include a slot (e.g., a content slot) (e.g., an audio slot or a visual slot) in which the digital component object may be rendered.

[0040] The data processing system 102 may include, execute or otherwise communicate with a content selector component 146 to receive a query, keyword or trigger keyword identified by a natural language processor, and select a digital component based on the trigger keyword. The content selector component 146 may select a digital component via a real-time content selection process. The content selection process may include, for example, performing a search via a search engine, or accessing a database stored on a remote server or device such as a 3P content provider device 140. The content selection process may refer to or include selecting a sponsored digital component object provided by a third-party content provider 140. The real-time content selection process may include a service in which digital components provided by multiple content providers are parsed, processed, weighted or matched in order to select one or more digital components to provide to a computing device 128. The content selector component 146 may perform a content selection process in real time. Performing a content selection process in real time may refer to performing a content selection process in response to a request for content received via a client computing device 128. The real-time content selection process may be performed (e.g., initiated or completed) within a time interval (e.g., 1 second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, or 20 minutes) of receiving the request. The real-time content selection process may be performed during the communication session with the client computing device 128, or within a time interval after the communication session is terminated. The data processing system 102 may select the audio ringtone to be provided after completing the digital component selection process or other digital task.

[0041] For example, the data processing system 102 may include a content selector component 146 designed, constructed, configured or operated to select digital component objects. In order to select digital components for display in a speech-based environment, the data processing system 102 (e.g., via the NLP component 108) may parse the input audio signal to identify a query, a keyword (e.g., a trigger keyword), and use the keyword to select a matching digital component. The data processing system 102 may select a matching digital component based on a broad match, an exact match, or a phrase match. For example, the content selector component 146 may analyze, parse, or otherwise process the subject of a candidate digital component to determine whether the subject of the candidate digital component corresponds to the subject of a keyword or phrase of an input audio signal detected by a microphone of the client computing device 128. The content selector component 146 may use image processing techniques, character recognition techniques, natural language processing techniques, or database lookups to identify, analyze, or recognize the voice, audio, word, character, text, symbol, or image of the candidate digital component. The candidate digital component may include metadata indicating the subject of the candidate digital component, in which case the content selector component 146 may process the metadata to determine whether the subject of the candidate digital component corresponds to the input audio signal.

[0042] 3P digital content provider 140 can provide additional indicators when setting content activities including digital components. Content providers can provide information that content selector component 146 can identify by performing a search using information about candidate digital components at a content activity or content group level. For example, candidate digital components can include unique identifiers that can be mapped to content groups, content activities, or content providers. Content selector component 146 can determine information about 3P digital content provider device 140 based on information in a content activity data structure stored in data repository 122.

[0043] Remote data processing system 142 may receive, via a computer network, a request for content for presentation on computing device 128. Data processing system 102 may identify the request by processing input audio signals detected by a microphone of client computing device 128. The request may include selection criteria for the request, such as device type, location, and keywords associated with the request.

[0044] In response to the request, remote data processing system 142 may select a digital component object from a data repository or database associated with 3P digital content provider device 140 and provide the digital component via network 105 for presentation via computing device 128. Computing device 128 may interact with the digital component object. Computing device 128 may receive an audio response to the digital component. Computing device 128 may receive an indication of a selection of a hyperlink or other button associated with the digital component object, which causes or allows computing device 128 to identify a service provider, request a service from a service provider, instruct a service provider to perform a service, transmit information to a service provider, or otherwise query a service provider device.

[0045] Computing device 128 (or client computing device or client device) may include or execute data processing system 102. Data processing system 102 may include, interface with, or otherwise communicate with at least one interface 104. Data processing system 102 may include, interface with, or otherwise communicate with at least one digital assistant component 106. Data processing system 102 may include, interface with, or otherwise communicate with at least one natural language processor (“NLP”) component 108. Data processing system 102 may include, interface with, or otherwise communicate with at least one model 110. Data processing system 102 may include, interface with, or otherwise communicate with at least one technology 112. Data processing system 102 may include, interface with, or otherwise communicate with at least one application manager component 114. Data processing system 102 may include, interface with, or otherwise communicate with at least one direct action application programming interface (“API”) 116. Data processing system 102 may include, interface with, communicate with, or execute at least one application 118. Data processing system 102 may execute at least one application 118, at least in part (e.g., an application may include a client component and a server component). Data processing system 102 may include, interface with, or otherwise communicate with at least one data repository 122.

[0046] The data repository 122 may include one or more local or distributed databases, and may include a database management system. The data repository 122 may include computer data storage or memory, and may store applications 124, application data, profiles, indexes 126, preferences, and other data. An application 124 in the data repository 122 may refer to or include a data structure or data file that stores information associated with an application, such as declarations, input fields, states, context information, user interfaces, input parameters, or preferences. An application 118 may refer to an activated, launched, invoked, or executed application 118, which may correspond to information about the application 118 stored in an application data structure 124 in the data repository 122. An application 124 may include an executable file, an application package file, a configuration file, or other data that facilitates the execution of an application.

[0047] The data repository 122 may include, store, manage or maintain an index 126. The index 126 may include a mapping, association or link that associates a keyword or term with an application. The index 126 may include a table with keywords or terms mapped to candidate applications. The index 126 may include a list of applications installed on the computing device 128. The index 126 may include a list of applications authorized or accessible by the computing device 128 or its account. The index 126 may include credentials or authentication information for accessing the application. For example, the index may include a list of applications accessible to the computing device 128, and account information for obtaining access to the application. The index 126 may link keywords or terms to applications or facilitate identification of other information that an application is called, activated or started in response to an input audio signal or voice input.

[0048] The interface 104, digital assistant component 106, NLP component 108, application manager component 114, direct action API, or other components of the data processing system 102 may each include at least one processing unit or other logic device, such as a programmable logic array engine or a module configured to communicate with each other or with other resources or databases. The interface 104, digital assistant component 106, NLP component 108, application manager component 114, direct action API, or other components of the data processing system 102 may be separate components, a single component, or part of the data processing system 142. The system 100 and its components (such as the data processing system 102) may include hardware elements, such as one or more processors, logic devices, or circuits.

[0049] Computing device 128 may include, interface with, or otherwise communicate with at least one sensor 136, transducer 132, audio driver 138, preprocessor 130, or display device 134. Sensor 136 may include, for example, an ambient light sensor, a proximity sensor, a temperature sensor, an accelerometer, a gyroscope, a motion detector, a GPS sensor, a position sensor, a microphone, or a touch sensor. Transducer 132 may include a speaker or a microphone. Audio driver 138 may provide a software interface to hardware transducer 132. The audio driver may execute an audio file or other instructions provided by data processing system 102 to control transducer 132 to generate corresponding sound waves or sound waves. Display device 134 may include Figure 4 . The preprocessor 130 may be configured to detect a trigger keyword, a predetermined hot word, an initiation keyword, or an activation keyword. In some cases, the trigger keyword may include a request to perform an action. In some cases, the trigger keyword may include a predetermined action keyword to enable or activate the computing device 128, and the request keyword may follow the trigger keyword or hot word. The preprocessor 130 may be configured to detect keywords and perform actions based on keywords. The preprocessor 130 may detect a wake-up word or other keywords or hot words, and in response to the detection, call the digital assistant component 106 of the data processing system 102 executed by the computing device 128. In some cases, the preprocessor 130 may filter out one or more terms or modify the terms before sending the terms to the remote data processing system 142 for further processing. The preprocessor 130 may convert an analog audio signal detected by a microphone into a digital audio signal, and transmit or provide one or more data packets carrying the digital audio signal to the data processing system 102 or the remote data processing system 142 via the network 105. In some cases, the pre-processor 130 may provide a data packet carrying some or all of the input audio signal to the digital assistant component 106 or the remote data processing system 142 in response to detecting an instruction to perform such a transmission. The instruction may include, for example, a trigger keyword or other keyword or authorization to transmit the data packet including the input audio signal to the data processing system 102 or the remote data processing system 142.

[0050] The client computing device 128 may be associated with an end user who enters a voice query as audio input into the client computing device 128 (via the sensor 136) and receives audio output in the form of computer-generated speech, which may be provided from the data processing system 102 (or 3P content provider device 140) to the client computing device 128 as output from the transducer 132 (e.g., a speaker). The computer-generated speech may include a recording of speech from a real person or generated by a computer.

[0051] The data processing system 102 may include an interface 104 designed, configured, constructed, or operated to receive and transmit information using, for example, data packets. The interface 104 may receive and transmit information using one or more protocols, such as a network protocol. The interface 104 may include a hardware interface, a software interface, a wired interface, or a wireless interface. The interface 104 may facilitate conversion or formatting of data from one format to another. For example, the interface 104 may include an application programming interface that includes definitions for communicating between various components, such as software components. The interface 104 may facilitate communication between one or more components of the system 100, such as a digital assistant component 106, an application 118, and a data repository 122.

[0052] The data processing system 102 may include an application, script, or program installed at the client computing device 128, such as an application for communicating an input audio signal to the interface 104 of the data processing system 102 and driving components of the client computing device to render an output audio signal. The data processing system 102 may receive a data packet or other signal that includes or identifies an audio input signal. For example, the data processing system 102 may execute or run an NLP component 108 to receive or obtain an audio signal and parse the audio signal. For example, the NLP component 108 may provide interaction between a person and a computer. The NLP component 108 may be configured with technology for understanding natural language and allowing the data processing system 102 to derive meaning from human or natural language input. The NLP component 108 may include or be configured with technology based on machine learning (such as statistical machine learning). The NLP component 108 may utilize a decision tree, a statistical model, or a probabilistic model to parse the input audio signal. The NLP component 108 can perform, for example, functions such as named entity recognition (e.g., given a stream of text, determining which items in the text map to appropriate names, such as people or places, and the type of each such name, such as person, location, or organization), natural language generation (e.g., converting information from a computer database or semantic intent into understandable human language), natural language understanding (e.g., converting text into more formal representations, such as first-order logic structures that computer modules can manipulate), machine translation (e.g., automatically translating text from one human language to another), morphological segmentation (e.g., breaking words into individual morphemes and identifying categories of morphemes, which can be challenging based on the complexity of the morphology or structure of the words of the language under consideration), question answering (e.g., determining answers to human language questions, which can be specific or open-ended), semantic processing (e.g., processing that can occur after a word has been identified and its meaning has been encoded in order to associate the identified word with other words of similar meaning).

[0053] The NLP component 108 converts the audio input signal into recognized text by comparing the input signal with a stored representative audio waveform set (e.g., model 110 or technology 112) and selecting the closest match. The audio waveform set can be stored in the model 110 or other databases accessible to the data processing system 102. The representative waveform is generated on a large user set and can then be enhanced with a voice sample from the user. After the audio signal is converted into recognized text, the NLP component 108 matches the text to words associated with actions that the data processing system 102 can serve, for example, via the use of a model 110 that has been trained across users or by manual specification.

[0054] The audio input signal may be detected by a sensor 136 or transducer 132 (e.g., a microphone) of the client computing device 128. Via the transducer 132, the audio driver 138, or other components, the client computing device 128 may provide the audio input signal to the data processing system 102, where it may be received (e.g., via the interface 104) and provided to the NLP component 108 or stored in the data repository 122.

[0055] The NLP component 108 can obtain an input audio signal. According to the input audio signal, the NLP component 108 can identify at least one request or at least one trigger keyword corresponding to the request. The request may indicate the intention or theme of the input audio signal. The trigger keyword may indicate the type of action that may be taken. For example, the NLP component 108 can parse the input audio signal to identify at least one request to call an application. The NLP component 108 can parse the input audio signal to identify at least one request, such as a request to leave home at night to attend dinner and a movie. The keyword may include at least one word, phrase, root word, or partial word or derivative word indicating the action to be taken. For example, the keyword "go" or "to go to" from the input audio signal may indicate the need for traffic. In this example, the input audio signal (or the identified request) does not directly express the intention of traffic, but the keyword indicates that traffic is an auxiliary action of at least one other action indicated by the request.

[0056] The NLP component 108 can parse the input audio signal to identify, determine, retrieve or otherwise obtain requests and keywords. For example, the NLP component 108 can apply a semantic processing technique (e.g., technique 112) to the input audio signal to identify keywords or requests. The NLP component 108 can apply the semantic processing technique 112 to the input audio signal to identify a trigger phrase including one or more trigger keywords (such as a first trigger keyword and a second trigger keyword). For example, the input audio signal may include the sentence "I need someone to do my laundry and my dry cleaning." The NLP component 108 can apply the semantic processing technique 112 or other natural language processing techniques to a data packet including a sentence to identify the trigger phrases "do my laundry" and "my drycleaning". The NLP component 108 can also identify multiple trigger keywords, such as clothes and dry cleaning. For example, the NLP component 108 can determine that the trigger phrase includes a trigger keyword and a second trigger keyword.

[0057] The NLP component 108 may identify a search query or trigger phrase corresponding to performing a search or other request for information. The NLP component 108 may determine that the input audio signal corresponds to a request for information about a topic, event, current event, news event, dictionary definition, historical event, person, place, or thing. For example, the NLP component 108 may determine that the input audio signal corresponds to a search query for information about historical events, such as "When did the American Revolution take place?" The NLP component 108 may determine that the input audio signal corresponds to a request to invoke, launch, or activate an application 118.

[0058] The NLP component 108 may filter the input audio signal to identify keywords, search queries, or trigger keywords. For example, a data packet carrying the input audio signal may include "It would be great if I could get someone that could help me go to the airport", in which case the NLP component 108 may filter out one or more of the following terms: "it", "would", "be", "great", "if", "I", "could", "get", "someone", "that", "could", or "help". By filtering out these terms, the NLP component 108 may more accurately and reliably identify trigger keywords, such as "go to the airport", and determine that this is a request for a taxi or ride-sharing service.

[0059] Based on processing or parsing the input audio signal, the NLP component 108 may identify an application 118 to be invoked, launched, opened, or otherwise activated. The NLP component 108 may identify the application 118 based on parsing the input audio signal to identify a term, keyword, trigger keyword, or phrase. The NLP component 108 may perform a lookup in the index 126 using the identified term, keyword, trigger keyword, or phrase to identify the application 118. In some cases, the keyword may include an identifier of the application 118, such as "Application_Name_A" or "Application_Name_B". In some cases, the keyword may indicate a type or category of the application 118, such as a ride-sharing application, a restaurant reservation application, a movie ticket application, a news application, a weather application, a navigation application, a streaming music application, a streaming video application, a restaurant review application, or other types or categories of applications 118.

[0060] If the input audio signal includes an identifier corresponding to an application, such as an alphanumeric identifier, the NLP component 108 may perform a lookup in the index 126 using the alphanumeric identifier to identify the application 118 and provide an indication to the application manager component 114 for activation. The indication may include a unique identifier, reference, pointer, link, or deep link to the application 118. If the input audio signal includes an indication of a type or category of the application 118, the NLP component 108 may perform a lookup in the index 126 using the type or category of the application 118 to identify whether a corresponding application 118 is installed on the computing device 128. For example, if the category or type of application is a restaurant reservation application, the index 126 may include a reference or identifier of an application associated with the category, such as "Restaurant_Booking_App". If the input audio signal includes a query or request, the NLP component 108 may determine the type of the query or request and then identify an application 118 in the index 126 that is configured or capable of satisfying the query or request. In some cases, NLP component 108 may communicate with remote data processing system 142 to identify and select an application in response to an input audio signal.

[0061] Even if the digital assistant component 106 cannot identify an application 118 installed on the computing device 128 that can satisfy the input query or request using the index 126, the digital assistant component 106 can generate an alert or prompt. The alert or prompt can indicate that the application is not installed and that the digital assistant component 106 cannot satisfy the request or query. In some cases, the prompt can include an indication of an application 118 that is configured to satisfy the query or request but is not installed on the computing device 128 or is not otherwise accessible by the computing device 128. The prompt can include a reference to a resource that can provide the application 118 for installation on the computing device 128.

[0062] The NLP component 108 may use one or more models 110 and techniques 110 to process the input audio signal. The model 110 may refer to or include a machine learning model 110 trained based on data. The model 110 may include various weights or configurations to facilitate natural language processing or understanding. The model 110 may include, for example, certain weights, calibrations, or configurations to facilitate natural language understanding. The model 110 may be calibrated to be more likely to detect a certain type of term. For example, if the NLP component 108 detects that the input request is to activate an application, the NLP component 108 may use a model 110 with a list of all applications installed on the computing device 128 (e.g., index 126). The NLP component 108 may determine which application from the application list corresponds to the input request. By using a model 110 with a limited set of applications installed on the computing device 128, the NLP component 108 may increase the likelihood of accurately detecting the requested application 118 compared to a set of all applications that may be provided by an online application market. Furthermore, by using a model 110 with a limited set of applications 118 installed on a computing device, as compared to a set of all applications, the NLP component 108 can reduce computing resource utilization (e.g., using a smaller model 110 can take up less memory and use less processing) while improving the accuracy and reliability of identifying applications.

[0063] The NLP component 108 may use one or more techniques 112 to parse or process the input audio signal. The techniques 112 may include rule-based techniques 112 or statistical techniques 112. The techniques 112 may utilize machine learning or deep learning. Exemplary techniques 112 may include named entity recognition, sentiment analysis, text summarization, aspect mining, or topic mining. The techniques 112 may include or be based on text embedding (e.g., real-valued vector representations of strings), machine translation (e.g., language analysis and language generation), or dialogue and conversation (e.g., models used by artificial intelligence). The techniques 112 may include determining or utilizing grammatical techniques (e.g., the arrangement of words in a sentence based on grammar), such as lemmatization, morphological segmentation, word segmentation, part-of-speech tagging, parsing, sentence decomposition, or stemming. The techniques 112 may include determining or utilizing semantic techniques, such as named entity recognition (e.g., determining portions of text that can be identified and classified into the current group, such as the name, person, or place of application 118), word sense disambiguation, or natural language generation. Thus, the NLP component 108 may receive a first input audio signal detected by a microphone (sensor 136 ) of the client computing device 128 , and process the first input audio signal via the natural language processing technique 112 to identify an indication of an application.

[0064] The computing device 128 (or the digital assistant component 106) may include an application manager component 114 designed, constructed, and operated to activate an application 118. The application manager component 114 may receive an indication of an application 118 identified by the NLP component 108. The application manager component 114 may receive an alphanumeric identifier or other reference or indication of the application 118. The application manager component 114 may perform a lookup in the index 126 to verify that the application 118 is available or compatible for activation on the computing device 128. The application manager component 114 may activate the application 118. Activating the application 118 may refer to or include bringing the application 118 to the foreground of the computing device 128. Activating the application 118 may refer to executing the application. Activating the application 118 may refer to bringing the application from a background process to a foreground process. The application manager component 114 may generate a command, script, process, or other instruction to call, launch, or otherwise activate the application 118.

[0065] The application manager component 114 may perform a lookup in the index 126 of the client computing device 128 to determine that the application 118 is installed on the client computing device 128. The application manager component 114 may activate the application 118 in response to determining that the application 118 is installed on the client computing device 128.

[0066] Launching application 118 may cause the application to declare information to digital assistant component 106. Application 118 may declare the function that application 118 is configured to perform. Declaring the function that application 118 is configured to perform may include providing an indication of the type of parameters or the type of parameters that application 118 may receive as input in order to perform the function or task. Application 118 may declare the state of application 118, such as the current state. Application 118 may declare the current context of application 118. Application 118 may declare the type of function that application 118 may perform, the type of input for performing a function, or other information or data that facilitates application 118 to perform a task or perform an action.

[0067] For example, application 118 may include an application for making reservations at a restaurant. The first input audio signal may include a query or request to make a reservation, such as "Make a reservation using Restaurant_Booking_App_1". NLP component 108 using NLP technique 112 and model 110 (e.g., model 110 configured with applications installed on computing device 128) may identify application 118, and application manager component 114 may activate application 118 (e.g., Restaurant_Booking_App_1).

[0068] Application 118 can declare information using protocol 150. Protocol 150 can include a two-way communication protocol. Protocol 150 can include, for example, a transport protocol, a TCP / IP protocol, an interprocess communication protocol, a messaging protocol, or any other communication technology. Protocol 150 can include an interprocess communication protocol, by which application 118 can declare functions or intents supported by application 118, parameter types for intents or functions, and supported parameter values. For example, function "BOOK_A_TABLE" can have a TableType parameter and a BookingTime parameter. The TableType parameter can be associated with a value of Windowside or Lakeside. The BookingTime parameter can be associated with the type "Time". The digital assistant component 106 can receive this information via protocol 150 in order to modify the processing performed by the NLP component 108 to improve NLP understanding and determine the fulfillment of the user query. The digital assistant component 106 can send the fulfillment back to the application 118 using the same protocol 150 (e.g., "Set TableType=Lakeside").

[0069] Application 118 can use an API (e.g., a direct action API or a software development kit (“SDK”)) to declare information. Application 118 can be configured with the API or SDK to interface with digital assistant component 106 via interface 104. Application can use interface 104 to communicate with one or more components of digital assistant component 106, and digital assistant component 106 can use interface 104 to communicate with application 118.

[0070] The application 118 may be configured with a declaration 120. The developer of the application 118 may provide the declaration 120. The application 118 may be constructed, designed, and operated to determine the current state of the application 118. The state of the application 118 may refer to what input the application 118 is currently able to receive. The state of the application 118 may refer to what actions the application 118 is able to perform. The state of the application 118 may refer to what functions the application 118 is able to perform. Thus, the state of the application 118 may refer to, include, or be based on input that the application 118 may receive, functions that the application 118 may perform, or actions that the application 118 is configured to perform.

[0071] Exemplary declarations may include functions, actions, formats of input values, types of input values, categories or types of text, names, semantic information, values, or other information that facilitates detecting input parameters in voice input. Application 118 may declare that application 118 is expecting to receive queries, requests, instructions, or commands. Application 118 may declare that application 118 is expecting to receive values ​​or words, phrases, proper nouns, geographic locations, zip codes, states, cities, or towns. Declaration 120 may include points of interest. Declaration 120 may include currency. Therefore, application 118 may be configured to provide declarations 120 related to what functions application 118 can perform, or the current context or state of application 118, which may indicate what input parameters application 118 is currently able to receive and process to perform functions or actions. Exemplary functions may include making reservations at restaurants, making reservations at hotels, booking flights, renting cars, purchasing goods or services, ordering takeout food, booking ride sharing, or booking a taxi.

[0072] For example, application 118 may be designed, constructed, and operated to make reservations at restaurants. When launched, called, or otherwise activated, application 118 may enter a state. Application 118 may enter a default or main state. In this default, main, or initial state, application 118 may include an input text box. The input text box may be an input search box configured to receive a query. The query may include a query for a restaurant type, location, or restaurant name. Therefore, the current state of application 118 may indicate that application 118 is capable of receiving an input query in a search field of a user interface of application 118.

[0073] The application 118 may declare to the NLP component 108 via the protocol 150 that the functions that the application 118 can perform include making a reservation at a restaurant. In some cases, the application 118 may declare what type of input the application 118 can receive to perform the function, such as an input query related to searching for restaurants. The application 118 may further declare the type or format of the input, such as a keyword, text, a name, a category or type of restaurant, or the name of the restaurant. By declaring to the NLP 108 that the application 118 can perform the function of making a reservation at a restaurant, or is expecting to receive the name of a restaurant, the NLP 108 may modify the model 110 or technique 112 of the NLP 108 to improve the likelihood, accuracy, and reliability of detecting input information in the speech input that facilitates the application 118 to perform the function of making a reservation at a restaurant.

[0074] The NLP component 108 can modify the natural language processing technology 112 or the model 110 in response to the functions declared by the application. The NLP component 108 can perform dynamic real-time modifications or updates to the NLP processing technology 112 or the model 110. The NLP component 108 can perform real-time updates based on the declaration of the NLP technology 112 or the model 110. The NLP component 108 can use the declaration to perform real-time natural language understanding based on the state or context of the third-party application. Real-time can refer to updating the NLP processing in response to the activation of the application. Real-time can refer to updating the NLP processing after activating the application and before receiving the second audio input signal. Real-time can refer to updating the NLP processing within 1 second of activating the application 118, 2 seconds of activating the application 118, 3 seconds of activating the application 118, 5 seconds of activating the application 118, 10 seconds of activating the application 118, 15 seconds of activating the application 118, or 30 seconds of activating the application 118.

[0075] In some cases, the model 110 can be updated based on the function, and the NLP technology 112 can use the updated model 110. The NLP component 108 can provide the updated model 110 to the NLP technology 112. The NLP component 112 can select the type of model 110 or technology 112 based on the declaration. The NLP component 112 can select a model 110 or technology that is adjusted or calibrated to detect input parameters for the function declared by the application 118 in the subsequent input audio signal. For example, if the declaration indicates that the function performed by the application 118 uses an input including a geographic area, the NLP component 112 can select a model 110 or technology 112 that is calibrated, optimized or adjusted to detect a geographic area (such as a state, city, town, square or other geographic area). For example, the NLP component 108 can select a dictionary for the declared function. In another example, the model 110 can be adjusted to more heavily weight the terms corresponding to the declaration, rather than the numerical value. In another example, the technology 112 can be optimized to detect the geographic area, rather than the numerical value or semantic processing. Thus, the digital assistant component 106 can modify the natural language processing technology 112 after activating the application 118 and based on the functionality declared by the application 118 to increase the likelihood of detecting one or more parameters relative to the natural language processing technology 112 that has not been modified based on the functionality (e.g., technology 112 using an updated technology 112 or technology 112 using an updated model 110 may provide improvements in the accuracy and reliability or likelihood of detecting input parameters for a functionality relative to using a default technology 112 or a default model 110 to detect input parameters).

[0076] In another example, if the declared functionality includes making a reservation at a restaurant, the NLP component 108 may select a model 110 or technique 112 that is optimized to detect nouns, such as the name of a restaurant. Thus, based on the functionality declared by the application, the NLP component 108 may select a model 110 or technique 112 that is calibrated, tuned, optimized, or otherwise configured to more reliably detect input parameters for performing or executing the functionality, such as proper nouns, numbers, words, phrases, questions, numerical values, monetary amounts, or geographic locations.

[0077] The digital assistant component 106 (e.g., via the NLP component 108) can receive a second input audio signal detected by a microphone (e.g., sensor 136) of the computing device 128. The digital assistant component 106 can process the second input audio signal via the modified natural language processing technique 112 or model 110 to detect one or more parameters corresponding to the functions declared by the application 118. The NLP component 108 can perform real-time natural language processing of the second input audio signal to identify one or more parameters using state information declared by the application 118 (e.g., a declaration indicating functions that the application 118 can perform in the current state). Performing real-time natural language processing can refer to processing the second input audio signal using current context information declared by the application 118 to modify or update the NLP technique 112 or model 110 to process the input signal to detect the input parameters.

[0078] The digital assistant component 106 can determine that one or more parameters are compatible for input into the input field of the application 118 based on a comparison between one or more parameters and the function declared by the application. The digital assistant component 106 can determine that the detected input parameters are compatible with the declared function. The digital assistant component 106 can use strategies, matching techniques or other techniques to determine compatibility. The digital assistant component 106 can compare the type of the input parameter with the type of the input parameter that the function is configured to receive. The type can include, for example, a numerical value, a geographic location, a proper noun, a question, a word or a phrase. The type can include or be based on semantics, such as the meaning of the input parameter. The type can include or be based on grammar, such as the format or grammar of the input parameter. For example, the input parameter can be a geographic location. The digital assistant component 106 can determine that the input parameter is a geographic location, such as an address, and then compare the type "address" with the type of the input value configured for the declared function to determine a match or compatibility.

[0079] In another example, the digital assistant component 106 can determine compatibility based on the number of input parameters, the range of possible input values, or consistency between multiple input parameters.

[0080] In some cases, the digital assistant component 106 can use a template to determine compatibility. The digital assistant component 106 can identify a template associated with the declared function. The template can indicate the type of input parameters, the number of input parameters, the semantics and syntax associated with the input parameters, or any other information associated with the expected input parameters. The digital assistant component 106 can determine whether the detected input parameters are compatible using the template. For example, if the template indicates that the expected input parameter is a geographic location or address, but the detected input parameter is a product type, the digital assistant component 106 can determine the incompatibility between the input parameter and the declared function. In response to determining incompatibility, the digital assistant component 106 can determine not to create an action data structure with incompatible detected input parameters. The digital assistant component 106 can determine not to input incompatible parameters into the application 118 because it may cause a failure or error or an unexpected result of the application 118. Therefore, by detecting incompatibility and not inputting wrong input parameters, the digital assistant component 106 can reduce or prevent errors and wasteful computing resource utilization. In some cases, in response to detecting an error, the digital assistant component 106 may generate a prompt or alert requesting compatible input parameters.

[0081] The NLP component 108 may receive state information declared by the application 118 via the protocol 150. The NLP component 108 may map the second input audio signal to the state information via the modified natural language processing technology 112 (or model 110) to identify the request. For example, the state information may indicate that the application 118 is in a ready state and is open to receive a request to perform a type of action. The action type may range from performing a search for a restaurant, being added to a waiting list for a restaurant, making a reservation at a restaurant, or leaving a review of a restaurant. The application may declare the types of available actions to the NLP component 108. The NLP component 108 may then parse the second input audio signal to determine a request to leave a review of the restaurant (e.g., "leaveRestaurant_A 5 stars"). The NLP component 108 may detect that this is a request to leave a review of the restaurant using the technology 112 or model 110 (which is adjusted to more reliably and accurately detect that this is a request to leave a review based on the declared state information indicating that leaving a review is one of the compatible requests). The NLP component 108 can detect the request and then provide detected input parameters, including the function type (e.g., review), the name of the restaurant being reviewed (e.g., Restaurant_A), and what the review or rating is (e.g., 5 stars). The direct action API 116 can generate an action data structure that includes these three input parameters. Thus, the digital assistant component 106 can prevent or avoid the need for multiple dialogues or screens in order to perform the action by using the NLP component 108, which can modify the NLP technology 112 or model 110 in real time based on the declared context or state of the activated application 118.

[0082] The digital assistant component 106 may include a direct action API 116 designed and constructed to generate an action data structure for the application 118 in response to a determination of compatibility and based on one or more parameters. The direct action API 116 may input the action data structure into the application 118 via the protocol 150 so that the application 118 executes the action data structure and updates the user interface elements of the application 118. The processor of the computing device 128 may call the direct action API 116 to execute a script that generates a data structure to the application 118 to request or book a service or product, such as a car from a car sharing service or making a reservation at a restaurant. The direct action API 116 may obtain data from the NLP component 108 and the data repository 122, as well as data received from the client computing device 128 with the consent of the end user, to determine the location, time, user account, logistics or other information to allow the application 118 to perform an action or operation in communication with the 3P application server 148, such as making a reservation at a restaurant or reserving a car from a car sharing service. Using the direct action API 116, the digital assistant component 106 can also communicate with the 3P application server 148 to complete the conversion by making a reservation or making a car sharing pickup reservation in this example.

[0083] The direct action API 116 may perform specified actions to provide input to the application 118, thereby causing the application to perform actions to perform the function declared by the application 118. Depending on the action specified in its input, the direct action API 116 may execute code or a dialog script that identifies the parameters required to implement the function or user request. Such code may, for example, look up additional information in the data repository 122, such as the name of a home automation service, or it may provide audio output to be rendered at the client computing device 128 to ask the end user a question such as the intended destination of the requested taxi. The direct action API 116 may determine the necessary parameters and may package the information into an action data structure, which may then be sent to the application 118 for execution.

[0084] Direct action API 116 can receive instructions or commands from other components of NLP component 108 or digital assistant component 106 to generate or construct action data structure. Direct action API 116 can determine the type of action in order to select a template stored in data repository 122. The type of action can include, for example, service, product, reservation or ticket. The type of action can further include the type of service or product. For example, the type of service can include car sharing service, food delivery service, laundry service, maid service, repair service or home service. The type of product can include, for example, clothes, shoes, toys, electronic products, computers, books or jewelry. The type of reservation can include, for example, dinner reservation or hair salon appointment. The type of ticket can include, for example, movie ticket, stadium ticket or airplane ticket. In some cases, the type of service, product, reservation or ticket can be classified based on price, location, transportation type, availability or other attributes.

[0085] Upon identifying the type of request, the direct action API 116 may access a corresponding template from a template repository stored in a data repository (e.g., stored in an application 124 data structure or database). The template may include fields in a structured data set that may be populated by the direct action API 116 to further the requested operation of the 3P application server 148 (such as an operation of sending a taxi to pick up the end user at the pick-up location and transport the end user to the destination location). The direct action API 116 may perform a search in the template repository (e.g., stored in the application 124 data structure or database) to select a template that matches one or more features of the declared function. For example, if the declared function corresponds to requesting a car or ride to a destination, the data processing system 102 may select a car sharing service template. The car sharing service template may include one or more of the following fields: device identifier, pick-up location, destination location, number of passengers, or service type. The direct action API 116 may populate the fields with values. To populate the fields with values, the direct action API 116 may ping, poll, or otherwise obtain information from one or more sensors 136 of the computing device 128 or a user interface of the computing device 128. For example, the direct action API 116 may use a location sensor such as a GPS sensor to detect the source location. The direct action API 116 may obtain further information by submitting a survey, prompt, or query to a user terminal of the computing device 128. The direct action API may submit the survey, prompt, or query via the interface 104 of the data processing system 102 and a user interface (e.g., an audio interface, a voice-based user interface, a display, or a touch screen) of the computing device 128. Thus, the direct action API 116 may select a template for an action data structure based on a trigger keyword or request, populate one or more fields in the template with information detected by one or more sensors 136 or obtained via a user interface, and generate, create, or otherwise construct an action data structure to facilitate the 3P application server 148 to perform an operation.

[0086] The data processing system 102 can select a template from a template data structure stored in the data repository 122 (e.g., stored in an application 124 data structure or database) based on various factors, including, for example, one or more of a statement from the application, a keyword, a request, a 3P application server 148, an application type, a category to which the application 118 belongs (e.g., taxi service, laundry service, flower service, reservation, or food delivery), a location, or other information.

[0087] To select a template based on a declaration, the data processing system 102 (e.g., via the direct action API 116) can use the declaration or input parameters from the application 118 to perform a lookup or other query operation on a template database in a data repository (e.g., stored in an application 124 data structure or database) to identify a template data structure that maps or otherwise corresponds to the declaration or input parameters. For example, each template in the template database can be associated with one or more declarations to indicate that the template is configured to generate an action data structure in response to the declaration that the application 118 can process to perform an action. In some cases, the data processing system 102 can identify the type of application 118 based on information in the index 126 or the application 124.

[0088] To construct or generate the action data structure, the data processing system 102 may identify one or more fields in the selected template to be populated with values ​​detected in the input audio signal by the NLP component 108 using the updated or modified NLP technique 112 or model 110. The fields may be populated with numeric values, strings, Unicode values, Boolean logic, binary values, hexadecimal values, identifiers, location coordinates, geographic regions, timestamps, or other values. The fields or data structures themselves may be encrypted or masked to maintain data security.

[0089] When determining the fields in the template, the data processing system 102 can identify the values ​​of the fields to fill in the fields of the template to create an action data structure. The data processing system 102 can obtain, retrieve, determine or otherwise identify the values ​​of the fields by parsing or processing the input audio signal using the updated or modified NLP technology 112 or model 110. In some cases, the data processing system 102 can determine the values ​​from other sources (such as data storage libraries 122 or sensors 136). For example, the data processing system 102 can obtain or obtain information by querying or polling one or more available sensors of the client computing device 128, prompting the end user of the client computing device 128 for information, or accessing an online web-based resource using the HTTP protocol. For example, the data processing system 102 can determine that it does not have the current location of the client computing device 128, which may be a required field of the template. The data processing system 102 can query the client computing device 128 for location information. Data processing system 102 may request client computing device 128 to provide location information using one or more location sensors 136 , such as global positioning system sensors, WiFi triangulation, cell tower triangulation, Bluetooth beacons, IP addresses, or other location sensing technologies.

[0090] The direct action API 116 may use the protocol 150 to input the action data structure into the application 118. The direct action API 116 may input the action data structure via a user interface of the application 118 or other interface of the application 118. The direct action API may input the action data structure via the interface 104. The direct action API may input the action data structure via a headless interface of the application 118. The application 118, upon receiving the action data structure, may execute the action data structure to perform an action or function.

[0091] The digital assistant component 106 may receive additional declarations from the application 118. After the action data structure is input into the application 118, the current state of the application 118 may change based on processing or executing the action data structure. The application 118 may provide an updated or new declaration in response to executing the action data structure and changing the state of the application. The new declaration or updated declaration may indicate the current state of the application after receiving the action data structure, executing the action data structure, or partially executing the action data structure. The updated declaration may indicate a change to the user interface, such as an additional input field in the user interface. The NLP component 108 may modify (e.g., re-modify or modify again) the natural language processing technology 112 or model 110 previously modified in response to the function declared by the application in response to the indication of the change in state. The type of modification may be similar to the first modification, but is suitable for the new declaration (e.g., a new function declared or an input for executing a new function, or requesting additional input to facilitate the execution of the original function or action). The digital assistant component 106 may receive a third input audio signal after re-modifying the NLP technology 112 or model 110. The NLP component 108 may process the third input audio signal via the natural language processing technique 112 modified in response to the indication of the state change (e.g., the modified technique 112 or the same technique 112 using a modified or different model 110) to identify a second one or more parameters. The data processing system 102 may generate a second action data structure based on the second one or more parameters.

[0092] In some cases, the application 118 may call the digital assistant component 106. For example, the application 118 may provide an indication of a change in the state of the application 118. The application 118 may provide an indication via the protocol 150. The application 118 may provide a statement indicating a change in the state of the application 118, or a request for input or a request for an action data structure. The statement may indicate that the application 118 is in a ready state or can be used to execute an action data structure. The indication or statement may call the digital assistant component 106. For example, the indication may wake up the digital assistant component 106 so that the digital assistant component detects or processes an input audio signal. The call to the digital assistant component 106 may refer to or include a hot word or wake-up word detected by bypassing the preprocessor 130. The call may cause the digital assistant component 106 to generate a prompt or output, such as an audio output requesting input from a user of the computing device 128.

[0093] The call may cause the digital assistant component 106 to parse the current user interface of the application 118 to determine the context of the application or context information of the application or the current state of the application. In addition to determining the state or context of the application based on a declaration from the application 118, the digital assistant component 106 (e.g., via the application manager component 114) may also parse the user interface (e.g., graphical user interface) of the application 118 to determine the state of the application. Determining the state of the application 118 may refer to or include determining the type of input that the application 118 is able to receive, a change in the available input, or the type of action or function that the application 118 is able to perform. The application manager component 114 may parse the current user interface of the application 118 in response to an indication of a change to identify one or more input fields provided via a user interface element of the application 118. The digital assistant component 106 may receive an input audio signal (e.g., a third input audio signal) after the declaration call by the application 118 sent via the protocol 150. The digital assistant component 106 can process the third input audio signal using natural language processing techniques based on the declared one or more input fields to identify second one or more parameters of the third input audio signal for input into the application 118 in response to a change in the state of the application 118.

[0094] The digital assistant component 106 can receive an indication to deactivate an application. Deactivation can refer to an application 118 exiting, being closed, disabled, shut down, removed or uninstalled, entering a standby state, becoming a background process (e.g., no longer actively used in the foreground), or a different application entering the foreground or becoming a foreground process. Deactivation of an application 118 can indicate that the application 118 is not the primary application currently being used, or is no longer the primary application being used or executed on the computing device 128.

[0095] The digital assistant component 106 can return the modified natural language processing technology to the default natural language processing technology. The digital assistant component can return the modified NLP technology 112 or model 110 to the default or initial technology 112 or model 110 in response to the deactivation of the application 118. The digital assistant component 106 can remove the modifications made to the technology 112 or model 110 based on the functions declared by the application 118 to the digital assistant component 106 from the natural language processing technology 112 (or model 110). The digital assistant component 106 can restore the NLP to a previous state or a default state. The digital assistant component 106 can delete the modifications or adjustments made. The digital assistant component 106 can deactivate the modified technology 112 or model 110, but store them in the data repository 122 for future use. The digital assistant component 106 can reset the technology 112 or model 110 to an initial state or a default state to facilitate NLP of subsequent input audio signals that are no longer provided to the application 118 because the application 118 has been deactivated. By resetting the NLP, the digital assistant component 106 can prevent or reduce erroneous or inaccurate NLP by not using a technique 112 or model 110 that is tuned or calibrated for a specific functionality declared by the application 118. The NLP can revert to using a technique 112 or model 110 that is tuned for detecting queries or inputs generally, rather than a technique 112 or model 110 that is tuned to detect a specific query or input for the functionality of the application 118.

[0096] The digital assistant component 106 can transmit a request for content to the remote data processing system 142. The digital assistant component 106 can transmit a request for content based on or in response to an action data structure, a function declared by the application 118, a statement 120, a modified NLP technique, one or more parameters detected using the modified NLP technique in the second input audio signal, or a user interface element of the updated application. For example, the request can include information about the function declared by the application 118, the statement 120, the modified NLP technique, one or more parameters detected using the modified NLP technique in the second input audio signal, or a user interface element of the updated application. The request can include or provide a function type or function name to the remote data processing system 142. The request can include or provide a statement 120 to the remote data processing system 142. The request can include or provide an indication of how to modify the NLP to the remote data processing system 142, such as the weight applied to the NLP technique or the type of NLP technique being used. The request may include or provide information about parameters or values ​​detected in the second input audio signal using the modified NLP processing technology to the remote data processing system 142. The request may include or provide information about the user interface element of the application, such as a content item slot (e.g., the size or position of the content item slot that the content item will present via a visual display, or the duration of the slot, such as the duration of an audio or video content item) to the remote data processing system 142. Therefore, generating a request based on information may refer to or include generating a request using this information. In some cases, generating a request based on the user interface element of the updated application may include generating a request in response to identifying an available content slot in the user interface element of the application.

[0097] In some cases, the application 118 can generate a request for content and transmit the request for content directly to the remote data processing system 142. The application 118 can generate a request for content in response to the input of the action data structure. The application 118 can generate a request for content and transmit the request to the digital assistant component 106 to transmit, forward or communicate the request to the remote data processing system 142.

[0098] The content selector component 146 of the remote data processing system 142 can select content items (e.g., digital component objects) based on or in response to a request. The content selector component 146 can select content items based on or in response to an action data structure, a function declared by the application 118, a statement 120, a modified NLP technique, or a user interface element of the updated application. The content selector component 146 can input information associated with the action data structure, the function declared by the application 118, the statement 120, the modified NLP technique, the parameter or value detected in the input audio signal using the modified NLP technique, or the user interface element of the updated application into the real-time content selection process. The content selector component 146 can select content items that are related to or otherwise match information about the action data structure, the application, the function declared by the application 118, the statement 120, the modified NLP technique, the parameter or value detected in the input audio signal using the modified NLP technique, or the user interface element of the updated application. For example, if the application is an e-commerce application that sells shoes, the content selector component 146 may select content items for a type of athletic shoes. In another example, if the second input audio signal detected using the modified input audio signal includes a parameter or value for a restaurant type or a table type in a restaurant, the remote data processing system 142 may select content items related to restaurants (e.g., coupons for restaurants).

[0099] The remote data processing system 142 may provide the selected content item to the digital assistant component 106 for presentation. The remote data processing system 142 may provide the selected content item to the application 118 for presentation. The remote data processing system 142 may provide the selected content item to the digital assistant component 106 for presentation to the application 118. The content item may be presented via a user interface of the application 118, such as in a visual content slot or an audio output.

[0100] The digital assistant component 106 can present the content item separately or independently from the application 118. For example, the digital assistant component 106 can call a separate user interface, such as a pop-up window or banner content to be displayed via the display device 134 of the computing device 128. In some cases, the digital assistant component 106 can provide the content item as an audio output. The audio output can be presented before, during, or after the audio output of the application 118. In the case where the application 118 does not provide an audio output, the digital assistant component 106 can provide the content item via an audio output independent of the interface used by the application. Therefore, the data processing system 102 can present the content item via a user interface (e.g., an audio interface) of the digital assistant component 106 or a user interface (e.g., a graphical user interface) of the application 118.

[0101] Figure 2 is a diagram of operation 200 of dynamically interfacing the system 100 with an application. The system may include Figure 1 The system 100 depicted in or Figure 4 . In operation 200, at action 202, a preprocessor 130 (e.g., performed by a computing device 128) may receive or detect a first voice query. The first voice query may include an input audio signal. The preprocessor 130 may detect a trigger keyword, hot word, wake word, alarm, or other indication to invoke the digital assistant component 106. In response to detecting the wake word, the preprocessor 130 may invoke the digital assistant component 106 at action 204.

[0102] At action 206, the NLP component 108 of the digital assistant component 106 can parse the first voice query to identify the application. The NLP component 108 can use one or more NLP techniques or models to identify the application. The NLP component 108 can provide an indication of the application to the application manager component 114 at action 208. The application manager component 114 can determine that the application 118 is installed on the computing device 128 or is otherwise accessible by the computing device 128. The application manager 114 can start or activate the application 118 at action 210.

[0103] At act 212, application 118 may identify a state of application 118. For example, application 118 may identify a current user interface, graphical user interface, or function that application 118 may perform, or an input field of a user interface. Application 118 may provide a declaration of the state of application 118 to digital assistant component 106 via protocol 150 at act 214.

[0104] At action 216, the NLP component 108 may perform a claim-based real-time update of the NLP. Claim-based real-time update of the NLP may refer to updating, modifying, or otherwise adjusting or calibrating the NLP technique 112 or model 110 to improve the accuracy and reliability with which the NLP component 108 may detect input parameters in subsequent input audio signals that are compatible with the functionality claimed by the application 118.

[0105] At act 218, the digital assistant component 106 may receive the second voice query 218. The preprocessor 130 may receive the second voice query and provide the second voice query to the digital assistant component 106. In some cases, the digital assistant component 106 that has been previously called by the preprocessor 130 may remain in an active state and directly receive and parse the second voice query at act 220. At act 220, the NLP component 108 may detect parameters for activating or launching the application 118. At act 222, the digital assistant component 106 may provide the detected parameters to the direct action API 116. At act 224, the direct action API 116 may use a template or other technique to generate an action data structure with input parameters.

[0106] At action 226, the direct action API 116 may provide an action data structure for input into the application 118. At action 228, the application 118 may execute the action data structure. The application 118 may perform an action or otherwise process the received action data structure to change or update the state of the application. The action data structure may cause a change in the state of the application 118. At action 230, the application 118 may update the state and then declare the updated state. The application 118 may re-declare the state of the application to provide the digital assistant component 106 with current state information or context. At action 232, the application may provide a declaration corresponding to the updated state via a protocol. At action 234, the NLP component 108 may receive the declaration from action 232 and perform a declaration-based real-time update of the NLP technology 112 or model 110.

[0107] The application 118 may terminate at action 236. The application 118 may deactivate or enter a standby state at action 236. The user may close or minimize or hide the application 118 at action 236. The application 118 (or the application manager component 114) may provide an indication of deactivation or termination at action 238. The application manager component 114 may detect that the application 118 has been deactivated. At action 240, the NLP component 108 may reset the NLP technique 112 or model 110 to a default state so that input audio signals received after the application 118 is terminated are not processed using the NLP modified for the declared functionality of the application 118.

[0108] Figure 3is an illustration of an exemplary method for dynamically interfacing with an application. Method 300 may be performed by one or more components, systems, or elements of system 100 or system 400 (including, for example, a computing device, a data processing system, a digital assistant component, an NLP component, a direct action API, or an application manager component). Method 300 may include receiving an input audio signal at 302. The input audio signal may be received via an interface of the data processing system. The input audio signal may be detected by a microphone of a client computing device. The input audio signal may include a voice query, instruction, or command from a user of the computing device.

[0109] At action 304, the data processing system can use NLP to process the input audio signal to identify the application. The data processing system can identify application-based terms, keywords or other information in the input audio signal. At decision box 308, the data processing system can determine whether the application is installed on the computing device or otherwise accessible by the computing device. The data processing system can determine whether the computing device (or its account) is authorized to access or use the application. If at decision box 308, the data processing system determines that the application is not authorized, the data processing system can proceed to action 310 to provide an alarm. The alarm can include an indication of downloading and installing the application, an indication that the application is unavailable, or an indication of making another request.

[0110] If at decision box 308, the data processing system determines that the application is installed, the data processing system can proceed to action 306 to activate the application. The data processing system can start, call or otherwise bring the application to the foreground. In some cases, the data processing system can execute the application in a headless state, in which the application may not present a graphical user interface element for display on a display device of the computing device. For example, the data processing system can execute the application in a voice-only mode or be used to dock via audio input and audio output. In some cases, the data processing system can activate the application in a hybrid mode, in which the application can provide output via a graphical user interface and simultaneously receive input via a voice-based interface via a digital assistant component. Activating the application at action 306 can cause the application to declare the current state of the application.

[0111] At action 312, the digital assistant component executed by the data processing system can receive the declaration from the activated application and modify the NLP. The digital assistant component can perform real-time declaration-based updates of the NLP technology or model. The digital assistant component can update the NLP in response to the declaration received from the application.

[0112] At action 314, the data processing system may receive a second input audio signal. The digital assistant component may receive a second input audio signal, such as a voice input, from a user of the computing device. The second input audio signal may be a continuation of the first input audio signal. At action 316, the data processing system may process the second input audio signal using the updated NLP to detect input parameters that are compatible with the application (e.g., the current state of the application or an input field of a function of the application). The digital assistant component may determine that the one or more parameters are compatible for input into an input field of an application activated by the digital assistant on the client computing device based on a comparison between the one or more parameters and the function declared by the application.

[0113] At action 318, the digital assistant component can generate an action data structure based on the detected input parameters or request. The digital assistant component can generate an action data structure in response to determining that the input parameters are compatible with the function or state of the application. The digital assistant component can provide an action data structure for input into the application. The application can execute the action data structure.

[0114] At act 320, the digital assistant component may receive an updated assertion based on a new state of the application in response to execution (or partial execution) of the action data structure. The digital assistant component may update the NLP based on the new assertion.

[0115] At decision block 322, the data processing system may determine whether to select 3P content. 3P content may refer to a digital component object, such as a content item, an image, a video content item, or an audio content item. A digital component object may refer to or include an advertisement. The digital assistant component may determine whether to select 3P content based on a policy, preference, or configuration. For example, an application may provide an indication that a content slot is available in an application and is configured for a 3P content item via a statement. The statement may include the type of content slot (e.g., an audio content slot, a text content slot, an image content slot, a video content slot, a pop-up window content slot, or a banner content slot). In some cases, the computing device may include a profile or preference regarding whether to allow the provision of 3P content. In some cases, the digital assistant component may determine whether to select 3P content based on the current state of the function type or application. For example, if the application function involves navigation, the digital assistant component may determine not to select content. If the function is related to a reservation of a restaurant, the digital assistant component may determine to select 3P content.

[0116] If the digital assistant component determines at decision block 322 that 3P content is selected, the digital assistant may proceed to action 324 to select and provide 3P content. The digital assistant component may transmit a request for content to a remote data processing system having a content selector component. The content selector component may receive the request and information that facilitates the real-time content selection process. The content selector component may provide the selected content item to the digital assistant component. The digital assistant component may receive the content item and present the content item (e.g., play an audio content item), or provide the content item to the application 118 for presentation via the application (e.g., via an audio or visual content slot in the application).

[0117] The digital assistant may determine whether to reset the NLP to a default state at decision block 326 (e.g., after action 324 or if decision block 322 results in a negative, or independently of action 324 or decision block 322). The digital assistant may determine whether to reset the NLP based on the state of the application (such as whether the application is active, deactivated, terminated, on standby, or not installed). If the digital assistant determines not to reset the NLP, the digital assistant component may proceed to action 328 to maintain the NLP. However, if at decision block 326 the digital assistant component determines to reset the NLP, the digital assistant component may proceed to action 330 to undo an update to the NLP to reset the NLP to a default or initial state (e.g., prior to the declarative-based real-time update of action 312).

[0118] Thus, the technical solution can improve NLP processing to detect the value of a parameter in a request received in a voice-based computing environment, and provide the detected value for input into an application. The digital assistant component can receive a declaration from an application to identify the type of the parameter and the value of the parameter, and then use the detected type to perform NLP processing on the voice input. For example, the type of the parameter or value can be "time", "location", or an application-defined parameter type, such as a TableType with associated application-defined values ​​such as "booth", "round table", and "window view". In another example, the parameter type can be a restaurant name, which can facilitate the digital assistant component to recognize the restaurant name in the voice input, rather than the name of a person or geographic area.

[0119] Figure 44 is a block diagram of an exemplary computer system 400. The computer system or computing device 400 may include or be used to implement the system 100 or its components, such as the data processing system 102. The computing system 400 includes a bus 405 or other communication component for transmitting information and a processor 410 or processing circuit coupled to the bus 405 for processing information. The computing system 400 may also include one or more processors 410 or processing circuits coupled to the bus for processing information. The computing system 400 also includes a main memory 415 (such as a random access memory (RAM) or other dynamic storage device) coupled to the bus 405 for storing information and instructions to be executed by the processor 410. The main memory 415 may be or include the data repository 145. The main memory 415 may also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 410. The computing system 400 may further include a read-only memory (ROM) 420 or other static storage device coupled to the bus 405 for storing static information and instructions for the processor 410. A storage device 425 , such as a solid-state device, magnetic disk, or optical disk, may be coupled to bus 405 for persistently storing information and instructions. Storage device 425 may include data repository 145 or may be a part of data repository 145 .

[0120] Computing system 400 may be coupled to a display 435 (such as a liquid crystal display or an active matrix display) via bus 405 for displaying information to a user. An input device 430 (such as a keyboard including alphanumeric and other keys) may be coupled to bus 405 to communicate information and command selections to processor 410. Input device 430 may include a touch screen display 435. Input device 430 may also include a cursor controller (such as a mouse, trackball, or cursor direction keys) for communicating direction information and command selections to processor 410 and for controlling cursor movement on display 435. For example, display 435 may be a computer program product of data processing system 102, client computing device 128, or the like. Figure 1 part of other components.

[0121] The processes, systems, and methods described herein may be implemented by the computing system 400 in response to the processor 410 executing the arrangement of instructions contained in the main memory 415. These instructions may be read into the main memory 415 from another computer-readable medium, such as the storage device 425. Execution of the arrangement of instructions contained in the main memory 415 enables the computing system 400 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement may also be employed to execute the instructions contained in the main memory 415. Hard-wired circuitry may be used in place of software instructions or in combination with software instructions and the systems and methods described herein. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0122] Although already Figure 4 An example computing system is described in the specification, but the subject matter including the operations described in this specification may be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.

[0123] For the situation that the system discussed in this article collects the personal information of the relevant user or can utilize the personal information, the user can be provided with the opportunity to control whether the program or feature can collect personal information (for example, information about the social network, social action or activity, preference of the user, or the location of the user), or control whether and / or how to receive content that may be more relevant to the user from the content server or other data processing system. In addition, before storing or using specific data, the specific data can be anonymized in one or more ways, so that the personally identifiable information is removed when generating parameters. For example, the identity of the user can be anonymized, so that the personally identifiable information of the user cannot be determined, or the geographical location of the user can be summarized (such as, to the city, zip code, or state and county level) when the position information is obtained, so that the specific location of the user cannot be determined. Thus, the user can control the way that the content server collects and uses the information about him or her.

[0124] The subject matter and operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The subject matter described in this specification may be implemented as one or more computer programs, for example, one or more circuits of computer program instructions encoded on one or more computer storage media to be executed by a data processing device or to control the operation of a data processing device. Alternatively or additionally, program instructions may be encoded on an artificially generated propagation signal, for example, a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be or be included in the following: a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. When the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be or be included in the following: one or more separate components or media (for example, multiple CDs, disks, or other storage devices). The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0125] The terms "data processing system", "computing device", "component", or "data processing apparatus" cover various apparatuses, devices, and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or multiple or combinations of the foregoing. The apparatus may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can implement a variety of different computing model architectures, such as web services, distributed computing, and grid computing architectures. The digital assistant component 106, the direct action API 116, the interface 104, the NLP component 108, and other data processing system 102 components may include or share one or more data processing apparatuses, systems, computing devices, or processors. The digital assistant server 144 and the content selection component 146 may include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0126] Computer programs (also referred to as programs, software, software applications, apps, scripts, or codes) can be written in any form of programming language (including compiled languages ​​or interpreted languages, declarative languages, or procedural languages), and can be deployed in any form (including as stand-alone programs or modules, components, subroutines, objects, or other units suitable for use in a computing environment). Computer programs can correspond to files in a file system. Computer programs can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple collaborative files (e.g., files that store one or more modules, subroutines, or partial codes). Computer programs can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0127] The processes and logic flows described in this specification may be performed by one or more programmable processors that execute one or more computer programs (e.g., components of the data processing system 102) to perform actions by operating input data and generating outputs. The processes and logic flows may also be performed by dedicated logic circuits (e.g., FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits)), and the apparatus may also be implemented as dedicated logic circuits (e.g., FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits)). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROMs, EEPROMs, and flash memory devices), disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROM disks, and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit.

[0128] The subject matter described herein can be implemented in a computing system including a back-end component (e.g., as a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or a computing system including a combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: local area networks ("LANs") and wide area networks ("WANs"), internets (e.g., the Internet), and peer-to-peer networks (e.g., point-to-point peer-to-peer networks).

[0129] A computing system such as system 100 or system 400 may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network (e.g., network 105). The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., data packets representing digital components) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., the result of a user interaction) can be received from the client device at the server (e.g., received by a digital assistant server 144 from a digital assistant component 106 of a computing device 128 or a 3P content provider device 140).

[0130] Although operations are depicted in the drawings in a particular order, such operations need not be performed in the particular order shown or in sequential order, and not all illustrated operations need be performed. The actions described herein may be performed in a different order.

[0131] The separation of various system components need not be separated in all embodiments, and the described program components may be included in a single hardware or software product. For example, the NLP component 108 or the direct action API 116 may be a single component, app or program, or a logic device having one or more processing circuits, or executed by one or more processors of the data processing system 102.

[0132] Now that some illustrative embodiments have been described, it is apparent that the foregoing is illustrative rather than restrictive, and is presented by way of example. Specifically, although many of the examples presented herein relate to specific combinations of method actions or system elements, these actions and these elements may be combined in other ways to achieve the same purpose. The actions, elements, and features discussed in conjunction with one embodiment are not intended to be excluded from similar effects in other embodiments or embodiments.

[0133] The words and terms used herein are for description and should not be considered as limiting. The use of "comprising," "consisting of," "having," "including," "involving," "characterized by," and variations thereof herein is intended to encompass the items listed thereafter, their equivalents, and additional items, as well as alternative embodiments consisting of the items listed thereafter exclusively. In one embodiment, the systems and methods described herein consist of one of the described elements, actions, or components, each combination of more than one of the described elements, actions, or components, or all of the described elements, actions, or components.

[0134] Any reference to the embodiments or elements or actions of the systems and methods mentioned herein in the singular may also include embodiments with a plurality of these elements, and any reference to any embodiments or elements or actions herein in the plural may also encompass embodiments including only a single element. Reference in the singular or plural form is not intended to limit currently disclosed systems or methods, their components, actions, or elements to singular or plural configurations. Reference to any action or element based on any information, action, or element may include an embodiment where the action or element is at least partially based on any information, action, or element.

[0135] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment," "some embodiments," "an embodiment," etc. are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment or example. Such terms as used herein do not necessarily all refer to the same embodiment. Any embodiment may be combined with any other embodiment, inclusively or exclusively, in any manner consistent with the aspects and embodiments disclosed herein.

[0136] References to "or" may be interpreted as inclusive, such that any term described using "or" may indicate any of a single, more than one, and all of the terms described. References to at least one of a joint list of terms may be interpreted as inclusive or to indicate any of a single, more than one, and all of the terms described. For example, reference to "at least one of 'A' and 'B'" may include only 'A', only 'B', and both 'A' and 'B'. Such references used in conjunction with "includes" or other open-ended terms may include additional items.

[0137] Where a technical feature in the drawings, detailed description or any claim is followed by a reference numeral, the reference numeral is included to increase the intelligibility of the drawings, detailed description and claims. Therefore, neither the reference numeral nor its absence has any limiting effect on the scope of any claim element.

[0138] The systems and methods described herein may be embodied in other specific forms without departing from their characteristics. For example, a device, product, or service described as 3P or third party (such as 3P digital content provider device 140) may be or include in part or in whole a first party device, product, or service and may be jointly owned by an entity associated with data processing system 102, digital assistant server 144, or other components. The foregoing embodiments are illustrative rather than limiting of the described systems and methods. Accordingly, the scope of the systems and methods described herein is indicated by the appended claims rather than the foregoing description, and changes within the meaning and range of equivalents of the claims are included therein.

Claims

1. A method implemented by one or more processors, the method comprising: receiving a declaration corresponding to the application being activated for execution on the client computing device based on the application being activated for execution on the client computing device; after the application is activated for execution on the client computing device and based on the declaration corresponding to the application being activated for execution on the client computing device, modifying the model in response to the declaration corresponding to the application being activated for execution on the client computing device; Receive input audio; processing the input audio or text recognized from the input audio to detect one or more parameters using the model modified in response to a declaration corresponding to the application being activated to execute on the client computing device; based on the declaration, determining that the one or more parameters are compatible for input to the application; as well as In response to determining that the one or more parameters are compatible for input to the application, the one or more parameters are provided as the input to the application.

2. The method according to claim 1, wherein: The declaration corresponding to the application being activated for execution on the client computing device identifies a function that the application is configured to perform.

3. The method according to claim 1, wherein: The declaration corresponding to the application being activated for execution on the client computing device identifies a state of a user interface of the application.

4. The method according to claim 1, wherein: Processing the input audio or text recognized from the input audio to detect the one or more parameters includes using a dictionary selected based on a declaration corresponding to the application being activated for execution on the client computing device.

5. The method according to claim 1, wherein: Providing the one or more parameters as the input to the application comprises: generating an action data structure for the application based on the one or more parameters; and The action data structure is input into the application to cause the application to execute the action data structure and update an element of a user interface of the application.

6. The method according to claim 5, further comprising: generating a request for a content item based on the action data structure; transmitting said request for said content item to a remote data processing system; as well as In response to receiving the content item from the remote data processing system, the content item is presented.

7. The method according to claim 6, wherein: Presenting the content item includes the application presenting the content item via the user interface of the application.

8. The method according to claim 6, wherein: Presenting the content item includes presenting the content item via an audio output.

9. A computer program product comprising one or more computer-readable storage media having program instructions collectively stored on the one or more computer-readable storage media, the program instructions being executable to: receiving a declaration corresponding to the application being activated for execution on the client computing device based on the application being activated for execution on the client computing device; after the application is activated for execution on the client computing device and based on the declaration corresponding to the application being activated for execution on the client computing device, modifying the model in response to the declaration corresponding to the application being activated for execution on the client computing device; Receive input audio; processing the input audio or text recognized from the input audio to detect one or more parameters using the model modified in response to a declaration corresponding to the application being activated to execute on the client computing device; based on the declaration, determining that the one or more parameters are compatible for input to the application; as well as In response to determining that the one or more parameters are compatible for input to the application, the one or more parameters are provided as the input to the application.

10. The computer program product according to claim 9, wherein: The declaration corresponding to the application being activated for execution on the client computing device identifies a function that the application is configured to perform.

11. The computer program product of claim 9, wherein: The declaration corresponding to the application being activated for execution on the client computing device identifies a state of a user interface of the application.

12. The computer program product of claim 9, wherein: Processing the input audio or text recognized from the input audio to detect the one or more parameters includes using a dictionary selected based on a declaration corresponding to the application being activated for execution on the client computing device.

13. The computer program product of claim 9, wherein: Providing the one or more parameters as the input to the application comprises: generating an action data structure for the application based on the one or more parameters; and The action data structure is input into the application to cause the application to execute the action data structure and update an element of a user interface of the application.

14. The computer program product of claim 13, wherein: The program instructions are further operable to: generating a request for a content item based on the action data structure; transmitting the request for the content item to a remote data processing system; and In response to receiving the content item from the remote data processing system, the content item is presented.

15. The computer program product of claim 14, wherein: Presenting the content item includes the application presenting the content item via the user interface of the application.

16. The computer program product of claim 14, wherein: Presenting the content item includes presenting the content item via an audio output.

17. A system comprising: A processor, a computer readable memory, one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions being executable to: receiving a declaration corresponding to the application being activated for execution on the client computing device based on the application being activated for execution on the client computing device; after the application is activated for execution on the client computing device and based on the declaration corresponding to the application being activated for execution on the client computing device, modifying the model in response to the declaration corresponding to the application being activated for execution on the client computing device; Receive input audio; processing the input audio or text recognized from the input audio to detect one or more parameters using the model modified in response to a declaration corresponding to the application being activated to execute on the client computing device; based on the declaration, determining that the one or more parameters are compatible for input to the application; as well as In response to determining that the one or more parameters are compatible for input to the application, the one or more parameters are provided as the input to the application.

18. The system of claim 17, wherein: The declaration corresponding to the application being activated for execution on the client computing device identifies a function that the application is configured to perform.

19. The system of claim 17, wherein: The declaration corresponding to the application being activated for execution on the client computing device identifies a state of a user interface of the application.

20. The system of claim 17, wherein: Processing the input audio or text recognized from the input audio to detect the one or more parameters includes using a dictionary selected based on a declaration corresponding to the application being activated for execution on the client computing device.

21. A system for interacting with an application, comprising: one or more processors and memory; A digital assistant component executed by the one or more processors and memory for: activating an application for execution on the client computing device in response to first input audio, and causing the application to declare to the digital assistant component via a protocol the functionality that the application is configured to perform; after activating the application and based on the functionality declared by the application, modifying a natural language processing technique in response to the functionality declared by the application; receiving a second input audio; processing the second input audio via the modified natural language processing technique to detect one or more parameters; determining, based on a comparison between the one or more parameters and the functionality declared by the application, that the one or more parameters are compatible for input into an input field of the application activated by the digital assistant on the client computing device; responsive to the determination of compatibility and based on the one or more parameters, generating an action data structure for the application; inputting the action data structure into the application via the protocol so that the application executes the action data structure and updates a user interface element of the application; transmitting a request for a content item to a remote data processing system; as well as In response to receiving the content item from the remote data processing system, presenting a content item selected by the remote data processing system.

22. The system according to claim 21, comprising: The digital assistant component is used to generate the request based on the action data structure.

23. The system of claim 21, comprising: The digital assistant component is used to generate the request based on the functionality declared by the application.

24. The system of claim 21, comprising: The digital assistant component is used to generate the request based on the modified natural language processing technology.

25. The system of claim 21, comprising: The digital assistant component is used to generate the request based on the updated user interface element of the application.

26. The system of claim 21, comprising: The digital assistant component is used to provide the content item to the application for presentation via the user interface of the application.

27. The system of claim 21, comprising: The digital assistant component is used to present the content item via an audio output.

28. The system of claim 21, comprising: The digital assistant component is used to present the content item via an output interface different from the user interface of the application.

29. The system of claim 21, comprising: The digital assistant component is configured to present the content item via an audio output separate from the user interface of the application.

30. The system of claim 21, comprising: The remote data processing system is configured to select the content item during a real-time content selection process.

31. The system of claim 21, comprising: The remote data processing system is configured to select the content item based on the functionality declared by the application.

32. The system of claim 21, comprising: The remote data processing system is configured to select the content item based on the one or more parameters detected in the second input audio.

33. The system of claim 21, comprising: The remote data processing system is configured to select the content item based on the application.

34. The system of claim 21, comprising: The remote data processing system is configured to select the content item based on the action data structure input into the application.

35. The system of claim 21, wherein: The modified natural language processing technique includes converting the second input audio into text.

36. A method of interacting with an application, comprising: In response to the first input audio, the digital assistant component executed by the one or more processors activates an application for execution on the client computing device and causes the application to declare to the digital assistant component, via a protocol, a function that the application is configured to perform; upon activation of the application and based on the functionality declared by the application, modifying, by the digital assistant component, a natural language processing technique in response to the functionality declared by the application; receiving, by the digital assistant component, a second input audio; processing, by the digital assistant component, the second input audio via the modified natural language processing technique to detect one or more parameters; determining, by the digital assistant component based on a comparison between the one or more parameters and the functionality declared by the application, that the one or more parameters are compatible for input into an input field of the application activated by the digital assistant on the client computing device; generating, by the digital assistant component, an action data structure for the application in response to the determination of compatibility and based on the one or more parameters; inputting, by the digital assistant component, the action data structure into the application via the protocol so that the application executes the action data structure and updates a user interface element of the application; transmitting, by the digital assistant component, a request for a content item to a remote data processing system; as well as Presenting, by the digital assistant component in response to receiving the content item from the remote data processing system, a content item selected by the remote data processing system.

37. The method of claim 36, comprising: The request is generated by the digital assistant component based on the action data structure.

38. The method of claim 36, comprising: The content item is provided by the digital assistant component to the application for presentation via the user interface of the application.

39. The method of claim 36, comprising: The content item is presented by the digital assistant component via an audio output.

40. The method of claim 36, wherein: The modified natural language processing technique includes converting the second input audio into text.

Images