Bit Vector Based Content Matching for Third Party Digital Assistant Actions

Abstract

Matching content to actions of a third-party digital assistant using bit vectors is provided. The system receives an application having voice-assistant-enabled actions. The system identifies actions in the application. The system identifies content items provided by the third-party computing device. The system generates bit vectors or actions using machine learning models and performance data of the content items. In response to a request for content from a client device that performs an action of the application, the system selects a content item based on the bit vector corresponding to the action.

Description

[Background technology]

[0001] A computing device can execute applications. The computing device can receive queries from users, process the queries, and provide responses to the queries. However, due to limited availability or capabilities in the input or output interfaces of the computing device, it may be difficult to provide a useful response to a query with limited information via the available interfaces efficiently and without excessive input / output processes. Summary of the Invention [Means for solving the problem]

[0002] The present disclosure is generally directed to content matching based on bit vectors for actions of a digital assistant. In particular, applications running on a computing device can be configured with voice actions that can be invoked by a user. A data processing system of this technical solution can identify the invocation of a voice action in the application, retrieve a bit vector generated for an action in the application, and select a content item to provide in response to or in conjunction with the execution of the voice action.

[0003] For example, efficient and accurate matching of content items (digital component objects or supplemental content items) to third-party developed assistant actions can be difficult due to the large number of assistant actions developed by third-party developers of applications and the various dialogue states that these actions may be in during human interaction. Efficient and accurate matching of content items can be difficult because the action's response may not be revealed until the user begins a dialogue with the action, and there are many different paths that the action may take based on the user's utterances. Furthermore, efficient and accurate matching of content items can be difficult because it is difficult to generate relevant signals from the action to match with the content item.

[0004] The system and method of the technical solution provide content matching based on bit vectors for digital assistant actions. For example, the data processing system of the technical solution may include an action crawler that can simulate a conversation with a digital assistant-based application. The action crawler can crawl various conversation paths in third-party actions. The data processing system can obtain the output of the action crawler and feed the output to a machine learning model to extract embeddings. The data processing system can use the embeddings to retrieve the closest matching content items. The data processing system can score the content items against the output of the action crawler. The data processing system can generate a bit vector for the action, including the associated content items. The data processing system can then retrieve the bit vector to perform real-time content selection in response to a request for content or an invocation of a corresponding action.

[0005] At least one aspect is directed to a system including a data processing system having a memory and one or more processors. The data processing system can receive an application configured with a voice assistant-enabled action from an application developer. The data processing system can identify an action that the application is configured to perform in response to a voice input. The data processing system can identify a content item provided by a third party of the computing device. The data processing system can generate a bit vector corresponding to the action by a machine learning model and performance data of the content item. The bit vector can indicate a candidate content item for each action of the actions. The data processing system can select a content item based on a bit vector of the bit vectors corresponding to the action in response to a request for content for a computing device that performs an action of the application.

[0006] At least one aspect is directed to a method executed by a data processing system having a memory and one or more processors. The method may include the data processing system receiving an application configured with a voice assistant-enabled action from an application developer. The method may include the data processing system identifying an action that the application is configured to perform in response to a voice input. The method may include the data processing system identifying a content item provided by a third party of the computing device. The method may include the data processing system generating a bit vector corresponding to the action by a machine learning model and performance data of the content item. The bit vector may indicate a candidate content item for each action of the actions. The method may include the data processing system selecting a content item based on a bit vector of the bit vectors corresponding to the action in response to a request for content for a computing device that performs an action of the application.

[0007] These and other aspects and implementations are discussed in detail below. The above information and the detailed description below include illustrative examples of the various aspects and implementations and provide an overview or framework for understanding the nature and features of the claimed aspects and implementations. The drawings illustrate and provide a further understanding of the various aspects and implementations and are incorporated in and constitute a part of this specification.

[0008] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and reference designations in the various drawings indicate like elements. For purposes of clarity, every component may not be labeled in every drawing. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 illustrates an example system for vector-based content matching for third-party digital assistant actions, according to an implementation. [Diagram 2] FIG. 13 is a diagram of an example method for generating bit vectors for content matching for a third-party digital assistant action, according to an implementation. [Diagram 3] FIG. 1 illustrates an example method for selecting content using a bit vector in response to invoking a third-party digital assistant action, according to an implementation. [Figure 4] FIG. 4 is a block diagram showing an architecture for a computer system that can be used to implement elements of the systems and methods described and illustrated in this specification, including, for example, the system shown in FIG. 1 and the methods shown in FIG. 2 and FIG. 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Below is a more detailed description of various concepts related to methods, apparatus, and systems for vector-based content matching for third-party digital assistant actions, as well as implementations of those methods, apparatus, and systems. The various concepts introduced above and discussed in more detail below may be implemented in any of a number of ways.

[0011] The technical solution is generally directed to bit vector-based content matching for third-party digital assistant actions. An application developer may develop an application for installation and execution on a computing device. The application developer may be referred to as a third-party application developer because the application developer is an entity independent or separate from the provider, owner, or manufacturer of the computing device. The application developer may provide the application to a data processing system or a cloud computing system or environment. For example, the application developer may provide the application to an online marketplace or application store. The computing device may download the application from the online application store or marketplace.

[0012] The application may include digital assistant actions. The application may include voice actions. The application may interface with a digital assistant running on a computing device. The application may be configured with voice actions to provide a conversational user interface. For example, a user of a computing device may interface with an application through a conversational interface. A conversational interface may refer to or include conducting a voice-based interaction with the application. During the interaction, the user may invoke one or more voice actions in the application.

[0013] In an illustrative example, an application may be for a theme park. The application may include references to various characters associated with the theme park, such as animated characters. The application may provide various content related to the characters, such as multimedia content, videos, movies, shows, music, songs, stories, poems, or other content. The application may provide entertainment-related content in response to an invocation of audio content.

[0014] However, when a voice-based action is invoked, it may be difficult to identify or select supplemental content items provided by supplemental content providers to provide in parallel or along with the execution of the voice-based action. For example, the content selector component may not be aware of the dialog path or conversational path that led to the invocation of a particular voice action. The content selector may not be aware of the conversational branch that led to the utterance that invoked the voice action. This may be because voice actions differ from web pages in that a response to a voice action from an application may take into account one or more previous branches related to the conversation or dialogue flow. The content selector may not have access to those previous branches or information related to the previous branches. The content selector may have minimal or insufficient information. Because the content selector may not have access to previous branches in the conversational flow, it may attempt to select a content item based on the voice action itself. Thus, the content selector may select irrelevant content items, or may perform a resource-intensive selection process that attempts to select a content item but with low accuracy or a low relevancy score. This may result in wasted resource consumption, or selection of an incorrect or inaccurate content item. Providing an inaccurate or irrelevant content item may cause excessive remote procedure calls or network bandwidth utilization, as multiple additional content items may be selected and sent over the network to the computing device. Additionally, providing an inaccurate or irrelevant content item to a computing device may cause a poor user experience. A content item may be selected when a user does not interact with or engage with the content item. In some cases, a user may skip or fast-forward through the incorrect content item, thereby causing the user to provide additional input or interaction with the computing device as compared to the presentation of the correct or related content item.

[0015] To address these and other technical challenges and problems, the technical solution can generate and use bit vectors to match content items to third-party voice actions that are invoked in a third-party application running on a computing device. To do so, a data processing system of the technical solution can receive an application from an application developer. The data processing system can crawl the application for actions. The data processing system can simulate a conversation with the application to identify various branches in a conversation or dialogue flow and corresponding actions that can be invoked in each branch. The same voice action theme in an application can be invoked by multiple dialogue flows in the application, which can result in different responses to the voice action. For example, a dialogue flow can include "talk to Character_A," which can branch into the following three branches: 1) play a game, 2) read a story, or 3) sing a song. Each of these branches can correspond to a different voice action.

[0016] The data processing system can crawl the application to identify various dialogue flows and corresponding voice actions and generate a transcript. The transcript can include the voice actions and the dialogue flows that led to the voice actions or that caused the voice actions to be invoked. The transcript can include web reference ("webref") entities, internal classification information, or vertical classifications. The data processing system can receive supplemental content items provided by supplemental content providers such as advertisers. The data processing system can input the transcript along with the content items into a deep learning model to generate a bit vector. A bit vector can be generated for each voice action. For each voice action, the bit vector can indicate candidate content items that are considered relevant to the voice action by the deep learning model. The deep learning model can include a multi-layer cross attention model. The training data can include the performance of the content items when presented with the voice actions. The performance may include whether the user skipped a content item when it was presented with a particular voice action, or whether the user perceived, viewed, or heard the entire content item when it was played for a duration. The bit vector may provide a concise representation of all inputs to the deep learning model. The data processing system may match and score the content items with respect to the voice actions.

[0017] The data processing system can provide the bit vector to the content selector. Storing the bit vector can be more efficient than separately storing a transcript of the interaction for the voice action and separately storing the content item. For example, the bit vector may consume less memory or storage in the data repository of the data processing system. In another example, it may be more efficient to perform a search within the bit vector to identify candidate content items for a voice action during a real-time content selection process as opposed to scoring a larger collection of content items for the transcript. The content selector can use the bit vector during content provision time to perform real-time content selection in response to a request for content or in response to an invocation of a voice action.

[0018] In some cases, the data processing system may generate content item bit vectors separately for each group of content items to match the most relevant content items with the voice actions. To do so, the data processing system may use a k-dimensional tree or a combination of data structures.

[0019] When the third party application is executed on the computing device, a user can invoke a voice action in the application through a dialogue flow or conversation. A data processing system can receive an indication of the voice action. The data processing system can perform a search using the voice action in a bit vector data repository to identify a corresponding set of candidate content items. The data processing system can retrieve top-ranked candidate content items from the bit vector. The data processing system can score the top-ranked candidate content items using additional signals associated with the computing device or its electronic account to select the highest-scoring content items. The data processing system can then provide the highest-scoring content items to the computing device for presentation. By scoring the top-ranked candidate content items rather than all available content items, the data processing system can reduce utilization of computational resources without sacrificing accuracy or relevance of the selected content items.

[0020] Upon presenting the content item, the data processing system may receive performance information. The performance information may include, for example, whether the user skipped the content item, how long the user listened to the content item, or whether the user listened to the entire duration of the content item. For example, if the content item includes 15 seconds of audio output and the user skips the audio output after 3 seconds, the data processing system may determine that the content item did not perform well. If the user listened to the entire 15 second duration of the content item, the data processing system may determine that the content item performed well with respect to the voice action. The data processing system may update the deep learning model used to generate the bit vector based on the performance of the content item. The data processing system may decide to retrain or otherwise update the model. The data processing system may decide to regenerate or update the bit vector based on the received performance feedback.

[0021] FIG. 1 illustrates an exemplary system 100 for vector-based content matching for third-party digital assistant actions, according to an implementation. The system 100 may include a content selection infrastructure. The system 100 may include a data processing system 102. The data processing system 102 may communicate with one or more of the client computing devices 136 or the supplemental digital content provider devices 132 via a network 105. The network 105 may include computer networks such as the Internet, local area networks, wide area networks, metro area networks, or other area networks, intranets, satellite networks, and other communication networks such as voice or data mobile telephone networks. The network 105 may be used to access information resources such as web pages, websites, domain names, or uniform resource locators that may be provided, output, rendered, or displayed on the client computing device 136.

[0022] The network 105 may include or constitute a display network, e.g., a subset of information resources available on the Internet that are associated with a content placement or search engine results system, or suitable for including a third-party digital component as part of a digital component placement campaign. The network 105 may be used by the data processing system 102 to access information resources, such as web pages, websites, domain names, or uniform resource locators, that may be provided, output, rendered, or displayed by the client computing device 136. For example, via the network 105, a user of the client computing device 136 may access information or data provided by the supplemental digital content provider device 132.

[0023] Network 105 may be any type or form of network, including any of the following: point-to-point network, broadcast network, wide area network, local area network, telecommunications network, data communication network, computer network, Asynchronous Transfer Mode (ATM) network, Synchronous Optical Network (SONET) network, Synchronous Digital Hierarchy (SDH) network, wireless network, and wired network. Network 105 may include wireless links such as infrared channels or satellite bands. The topology of network 105 may include a bus, star, or ring network topology. The network may include a mobile telephone network using any protocol or protocols used to communicate between mobile devices, including advanced mobile phone protocol ("AMPS"), time division multiple access ("TDMA"), code division multiple access ("CDMA"), global system for mobile communication ("GSM"), general packet radio services ("GPRS"), or universal mobile telecommunications system ("UMTS"). Different types of data may be transmitted over different protocols, or the same type of data may be transmitted over different protocols.

[0024] The client computing device 136 may include, for example, a laptop, a desktop, a tablet, a digital assistant device, a smartphone, a mobile telecommunications device, a portable computer, a smart watch, a wearable device, a headset, a speaker, a television, a smart display, or an automobile unit. For example, via the network 105, a user of the client computing device 136 may access information or data provided by the supplemental digital content provider device 132. In some cases, the client computing device 136 may or may not include a display, e.g., the computing device may include a limited type of user interface, such as a microphone and a speaker. In some cases, the primary user interface of the client computing device 136 may be a microphone and a speaker, or an audio interface. In some cases, the client computing device 136 may include a display device coupled to the client computing device 136, and the primary user interface of the client computing device 136 may utilize the display device.

[0025] The client computing device 136 may include at least one input device 138. The input device 138 may include at least one microphone. The input device 138 may include a transducer or other hardware configured to detect sound waves, such as voice input from a user, and convert the sound waves into another format that can be processed by the client computing device 136. For example, the input device 138 may detect sound waves and convert the sound waves into an analog or digital signal. The client computing device 136 may use hardware or software to convert the analog or digital signal into a data packet corresponding to the voice input or other detected audio input. The client computing device 136 may transmit the data packet with the voice input to the data processing system 102 for further processing. The input device 138 may include a touch input device, a keyboard, a mouse, a gesture input, a sensor input, or other types of input interfaces.

[0026] The client computing device 136 may include at least one output device 140. The output device 140 may include a speaker. The output device 140 may output audio or sound. The output device 140 may be driven by an audio driver to generate audio output. The output device 140 may output speech or other audio generated by the data processing system 102 and provided to the client computing device 136 for output. For example, a user may have a conversation with the digital assistant 108 via the input device 138 and the output device 140 of the client computing device 136.

[0027] In some cases, the client computing device 136 may include one or more components or features of the data processing system 102, such as the NLP 106, the digital assistant 108, the interface 104, or the data repository 120. For example, the client computing device 136 may include a local digital assistant 144 or a digital assistant agent with one or more components or features of the server digital assistant 108 or the NLP 106. The client computing device 136 may include a data repository, memory, or storage device that stores one or more of an application, content data, account information, or profile information. The client computing device 136 may include one or more components or features of the computing system 400 shown in FIG. 4.

[0028] The system 100 may include at least one data processing system 102. The data processing system 102 may include at least one logical device, such as, for example, a computing device having a processor for communicating with the computing device 136, the supplemental digital content provider device 132 (or a third-party content provider device, content provider device), or the application developer device 134 over the network 105. The data processing system 102 may include at least one computing resource, server, processor, or memory. For example, the data processing system 102 may include multiple computing resources or servers located in at least one data center. The data processing system 102 may include multiple logically grouped servers to facilitate distributed computing techniques. A logical group of servers may be referred to as a data center, a server farm, or a machine farm. The servers may also be geographically scattered. A data center or machine farm may be managed as a single entity, or a machine farm may include multiple machine farms. The servers in each machine farm may be heterogeneous--one or more of the servers or machines may operate with one or more types of operating system platforms.

[0029] The servers in the machine farm may be housed in high density rack systems along with associated storage systems and located in an enterprise data center. For example, consolidating the servers in this manner may improve system manageability, data security, system physical security, and system performance by placing the servers and high performance storage systems on a localized high performance network. Centralization of all or part of the components of the data processing system 102, including the servers and storage systems and coupling them to advanced system management tools, allows for more efficient use of server resources, lowering power and processing requirements and reducing bandwidth usage.

[0030] The system 100 may include, access, or otherwise interact with at least one third party device, such as a supplemental digital content provider device 132, a service provider device, or an application developer device 134. The supplemental digital content provider device 132 may include at least one logical device, such as, for example, a client computing device 136 or a computing device having a processor for communicating with the data processing system 102 over the network 105.

[0031] The supplemental digital content provider device 132 can provide content items to the data processing system 102 for the data processing system 102 to provide to the client computing device 136 for presentation. The content items can include visual content, audio content, text-based content, multimedia content, or other types of content. The content items can include audio-based digital components for presentation by the client computing device 136 as audio output digital components. The digital components, such as advertisements provided by advertisers, can be referred to as sponsored digital components because they are provided by a third-party sponsor. The digital components can include offers for goods or services, such as a voice-based message stating, "Can I call you a cab?" For example, the supplemental digital content provider device 132 can include memory for storing a series of audio digital components that can be provided in response to a voice-based inquiry. The supplemental digital content provider device 132 can also provide the audio-based digital components (or other digital components) to the data processing system 102, where they can be stored in a data repository of the data processing system 102. The data processing system 102 can select the audio digital component and provide (or instruct the supplemental digital content provider device 132 to provide) the audio digital component to the client computing device 136. The audio-based digital component can be audio only or can be combined with text, image, or video data. A content item can have a duration, such as an audio or video length.When presented via a client computing device 136, a content item may be presented with user interface elements that enable interaction or engagement with the content item. For example, the user interface elements may enable input or commands that cause the client computing device 136 to pause the presentation or playback of the content item, skip playback of the content item, fast-forward the content item, rewind the content item, close or hide the content item, engage with the content item, or otherwise interact with the content item. The type of interaction may indicate a performance associated with the content item. For example, rewinding or playing a content item may indicate a positive performance, while skipping or fast-forwarding the content item may indicate a negative performance.

[0032] The data processing system 102 may include a content placement system having at least one computational resource or server. The data processing system 102 may include an interface 104 or may otherwise communicate with at least one interface 104. The data processing system 102 may include, interface with, or otherwise communicate with at least one natural language processor 106 (or natural language processor component). The interface 104 or natural language processor 106 may form or be referred to as a server digital assistant 108. The data processing system 102 may include, interface with, or otherwise communicate with at least one server digital assistant 108 (or server digital assistant component). The server digital assistant 108 can communicate or interface with one or more voice-based interfaces or various digital assistant devices or surfaces to provide or receive data or perform other functions. The data processing system 102 can include at least one action crawler 110 (action crawler component). The data processing system 102 can include at least one embedding extractor 112 (embedding extractor component). The data processing system 102 can include at least one content selector 114 (or content selector component).The content selector 114 may include at least one content retriever 116 (or content retriever component). The content selector 114 may include at least one content scorer 118 (or content scorer component). The data processing system 102 may include at least one data repository 120. The data repository 120 may store applications 122. The applications 122 may include information about the applications provided by the application developer device 134, metadata of the applications, or executable or source code of the applications, or application package files. The data repository 120 may store transcripts 124 related to the applications 122 and generated by the action crawler 110 regarding voice actions and interactions. The data repository 120 may store content data 126 such as content items provided by the supplemental digital content provider device 132 or generated by the content selector 114, performance of the content items, information about the content items, metadata of the content items, or other information about the content items. The data repository 120 may include models 128, such as deep learning models or other machine learning models, generated or used by the embedding extractor 112 to generate bit vectors. The data repository 120 may include bit vectors 130 generated by the embedding extractor 112 for voice actions of an application.

[0033] The data processing system 102, the interface 104, the NLP 106, the server digital assistant component 108, the action crawler 110, the embed extractor 112, the content selector 114, the content retriever 116, and the content scorer 118 may each include at least one processing unit configured to communicate with a data repository or database of the data processing system 102, or other logic device, such as a programmable logic array engine, or module. The interface 104, the NLP 106, the server digital assistant component 108, the action crawler 110, the embed extractor 112, the content selector 114, the content retriever 116, and the content scorer 118 may be separate components, a single component, or part of the data processing system 102. The system 100 and its components, such as the data processing system 102, may include hardware elements, such as one or more processors, logic devices, or circuits.

[0034] The data processing system 102 can obtain anonymous computer network activity information related to multiple computing devices 136 (or computing devices or digital assistant devices). A user of a client computing device 136 or a mobile computing device can affirmatively authorize the data processing system 102 to obtain network activity information corresponding to the client computing device 136 or the mobile computing device. For example, the data processing system 102 can prompt the user of the client computing device 136 to consent to obtaining one or more types of network activity information. The identity of the user of the client computing device 136 can remain anonymous, and the client computing device 136 can be associated with a unique identifier (e.g., a user or computing device unique identifier provided by the user of the data processing system or computing device). The data processing system 102 can associate each observation with a corresponding unique identifier.

[0035] The data processing system 102 may include an interface 104 (or interface component) that is designed, configured, constructed, or operable, for example, to receive and transmit information using data packets. The interface 104 may receive and transmit information using one or more protocols, such as network protocols. The interface 104 may include a hardware interface, a software interface, a wired interface, or a wireless interface. The interface 104 may facilitate converting or formatting data from one format to another. For example, the interface 104 may include an application programming interface that includes definitions for communicating between various components, such as software components. The interface 104 may communicate with one or more of the client computing devices 136, the supplemental digital content provider devices 132, or the application developer devices 134 via the network 105.

[0036] Data processing system 102 may interface with applications, scripts, or programs installed on client computing device 136, such as application 142 or local digital assistant 144, to communicate input audio signals to interface 104 of data processing system 102 and drive components of client computing device 136 to render, present, or otherwise output visual or audio signals. Data processing system 102 may receive data packets or other signals from application 142 or local digital assistant 144 that include or identify the audio input signal.

[0037] The data processing system 102 may include a natural language processor ("NLP") 106. For example, the data processing system 102 may execute or run the NLP 106 to analyze a received input audio signal or query. For example, the NLP 106 may provide interaction between humans and computers. The NLP 106 may be configured with techniques to understand natural language and enable the data processing system 102 to derive meaning from human or natural language input. The NLP 106 may include or be configured with machine learning based techniques, such as statistical machine learning. The NLP 106 may utilize decision trees, statistical models, or probabilistic models to analyze the input audio signal. NLP 106 may perform functions such as, for example, named entity recognition (e.g., given a stream of text, determining which items in the text map to appropriate names, such as people or places, and whether the type of each such name is a person, a place, or an organization, etc.), natural language generation (e.g., translating information or semantic intent from a computer database into an understandable human language), natural language understanding (e.g., converting text into a more organized representation, such as a first-order logical structure, that a computer module can manipulate), machine translation (e.g., automatically translating text from one human language to another), morpheme segmentation (e.g., breaking words into individual morphemes and identifying classes of morphemes, which may be difficult based on the complexity of the linguistic or structural features of the language under consideration), question answering (e.g., determining answers to questions in a human language, which may be specific or open-ended), and semantic processing (e.g., processing that may be performed after identifying words and encoding the meaning of the words, to associate the identified words with other words having similar meanings).

[0038] The NLP 106 can convert an audio input signal into recognized text by comparing the input signal to a stored, representative set of audio waveforms and selecting the closest match. The set of audio waveforms can be stored in a data repository or other database accessible to the data processing system 102. Representative waveforms may be generated across a large set of users and then augmented with speech samples from the users. After the audio signal is converted into recognized text, the NLP 106 matches the text to words associated with actions that the data processing system 102 can provide, for example, by training across users or by manually specifying them. Aspects or functions of the NLP 106 can be performed by the data processing system 102 or the client computing device 136. For example, an NLP component can be executed on the client computing device 136 to perform aspects of converting an input audio signal into text and transmitting the text by data packets to the data processing system 102 for further natural language processing.

[0039] The audio input signal may be detected by an input device 138 (e.g., a microphone, a sensor, a transducer) of the client computing device 136. Through a transducer, audio driver, or other component, the client computing device 136 may provide the audio input signal (e.g., over the network 105) to the data processing system 102, where it may be received (e.g., by the interface 104) and provided to the NLP 106 or stored in a data repository.

[0040] The data processing system 102 can receive a data packet including an input audio signal detected by a microphone of the client computing device 136 via the interface 104. The data processing system 102 can receive a data packet generated based on the input audio signal detected by the microphone. The data packet can be filtered or unfiltered. The data packet can be a digitized version of the detected input audio signal. The data packet can include text generated by the client computing device 136 based on the detected input audio signal. For example, the local digital assistant of the client computing device 136 can process the detected input audio signal and send a data packet based on the processed input audio signal to the server digital assistant 108 for further processing or to perform an action.

[0041] The data processing system 102 may include a server digital assistant 108. The server digital assistant 108 and the NLP 106 may be a single component, or the server digital assistant 108 may include one or more components or functions of the NLP 106. The server digital assistant 108 may interface with the NLP 106. The data processing system 102 (e.g., the server digital assistant 108) may process data packets to perform actions or otherwise respond to voice input. In some cases, the data processing system 102 may identify an acoustic signature from an input audio signal. The data processing system 102 may identify an electronic account corresponding to the acoustic signature based on searching a data repository (e.g., querying a database). The data processing system 102 may establish a session and an account for use in the session in response to identifying the electronic account. The account may include a profile having one or more policies. The data processing system 102 may analyze the input audio signal to identify a request and a trigger keyword corresponding to the request.

[0042] The NLP 106 can obtain an input audio signal. The NLP 106 of the data processing system 102 can receive a data packet having a voice input or an input audio signal in response to detection of a trigger keyword by a local digital assistant on the client computing device 136. The trigger keyword can be a wake-up signal or a hot word that indicates to the client computing device 136 to convert the subsequent audio input into text and send the text to the data processing system 102 for further processing.

[0043] Upon receiving the input audio signal, the NLP 106 may identify at least one inquiry or request, or at least one keyword corresponding to the request. The request may indicate the intent or subject of the input audio signal. The keyword may indicate the type of action likely to be taken. For example, the NLP 106 may analyze the input audio signal to identify at least one request to join a dinner party and leave the house to watch a movie in the evening. The trigger keyword may include at least one word, phrase, root or partial word, or derived word indicating the action to be taken. For example, the trigger keyword "go" or "to go to" from the input audio signal may indicate the need for transportation. In this example, the input audio signal (or the identified request) does not directly indicate the intent of transportation, but the trigger keyword indicates that transportation is a secondary action to at least one other action indicated by the request. In another example, the voice input may include an inquiry or request to perform an action in an application.

[0044] The NLP 106 may analyze the input audio signal to identify, determine, retrieve, or otherwise obtain a request and one or more keywords associated with the request. For example, the NLP 106 may apply semantic processing techniques to the input audio signal to identify a keyword or a request. The NLP 106 may apply semantic processing techniques to the input audio signal to identify a keyword or a phrase including one or more keywords, such as a first keyword and a second keyword. For example, the input audio signal may include the sentence "I would like to purchase an audiobook." The NLP 106 may apply semantic processing techniques or other natural language processing techniques to a data packet including the sentence to identify the keyword or phrase "I would like to purchase" and "audiobook." The NLP 106 may further identify multiple keywords, such as purchase and audiobook. For example, the NLP 106 may determine that the phrase includes a first and a second keyword.

[0045] The NLP 106 can filter the input audio signal to identify trigger keywords. For example, a data packet carrying the input audio signal can include "It would be great if I could get someone that could help me go to the airport," in which case the NLP 106 can filter out one or more of the following words: "it," "would," "be," "great," "if," "I," "could," "get," "someone," "that," "could," or "help." By filtering out these words, the NLP 106 can more accurately and reliably identify a trigger keyword, such as "go to the airport," and determine that it is a request for a taxi or ride-sharing service.

[0046] The server digital assistant 108 or the local digital assistant 144 can interface with an application 142 running on the client computing device 136. The application 142 can include voice-based actions or a voice interface. The application can receive voice input from a user of the client computing device 136. The client computing device 136 can forward the voice input to the data processing system 102 (e.g., the NLP 106 or the server digital assistant 108) for further processing. In some cases, the application 142 can forward the voice action to the application developer device 134 for processing. The application 142 can forward the voice action to the data processing system 102, which can preprocess the voice action and then forward the voice action to the application developer device 134. In some cases, the application 142 running on the client computing device 136 can be configured to process the voice action and perform a corresponding action. In some cases, the application 142 may forward the voice input to the data processing system 102 for processing, and the data processing system 102 may identify an action or intent associated with the voice input and forward a command to the application 142 to perform the requested action or intent.

[0047] The data processing system 102 may determine to select supplemental content items for a voice action of the application 142. However, when a voice-based action is invoked, it may be difficult to identify or select supplemental content items provided by supplemental content providers to provide in parallel or with the execution of the voice-based action. For example, the content selector 114 may not be aware of the dialogue or conversation path that led to the invocation of a particular voice action. The content selector 114 may not be aware of the conversation branch that led to the utterance that invoked the voice action. This may be because voice actions are different from web pages in that a response to a voice action from an application may take into account one or more previous branches related to the conversation or dialogue flow. The content selector may not have access to those previous branches or information related to the previous branches. The content selector may have minimal or insufficient information. Because the content selector may not have access to previous branches in the conversation flow, it may attempt to select a content item based on the voice action itself. Thus, the content selector may select irrelevant content items, or may perform a resource-intensive selection process that attempts to select a content item but with low accuracy or a low relevance score. This may result in wasted resource consumption, or selection of an incorrect or inaccurate content item. Providing an inaccurate or irrelevant content item may cause excessive remote procedure calls or network bandwidth utilization, as multiple additional content items may be selected and sent over the network to the computing device. Furthermore, providing an inaccurate or irrelevant content item to a computing device may cause a poor user experience.The content items may not perform well on the computing device in that the user may not interact or engage with the content items. In some cases, the user may skip or fast-forward through the incorrect content items, thereby causing the user to provide additional input or interaction with the computing device as compared to the presentation of the correct or related content items.

[0048] To address these and other technical challenges, the data processing system 102 of the technical solution may include an action crawler 110 designed, constructed, and operable to generate transcripts of actions that may be performed by an application. The action crawler 110 may receive an application configured with voice assistant-enabled actions from an application developer 134. The action crawler 110 may identify actions that the application is configured to perform in response to voice input. The action crawler 110 may crawl the application to identify the actions. The action crawler 110 may simulate a conversation with the application to identify various interaction paths that a user may take with the application. Different interaction paths may lead to the performance of the same voice action or different voice actions depending on the intent associated with the interaction path.

[0049] For example, the NLP 106 or the server digital assistant 108 of the data processing system 102 can simulate a conversation with the application to identify multiple actions. The server digital assistant 108 can simulate one or more conversations with the application. In some cases, the data processing system 102 can receive an application package file from the application developer device 134 that includes example dialogue or transcript information. In some cases, the data processing system 102 can automatically determine possible conversations that the application may engage in and identify voice actions that the application is configured to perform.

[0050] For example, the application may relate to providing entertainment content related to animated characters at a theme park. The action crawler 110 may simulate an interaction with the application. The action crawler 110 may launch or invoke the application. The application may begin with a query or prompt such as "Which character would you like to talk to?" The action crawler 110 may select a character from a list of options by generating a simulated voice input including the sentence "Talk to Character_A". The application may respond to the action crawler 110 with a response and a prompt or query such as "Do you want to play a game with Character_A, read a story with Character_A, or sing a song with Character_A?" The action crawler 110 may select one of these voice actions that corresponds to a voice assistant-enabled action. For each of the voice actions, the action crawler 110 may store a transcript including the dialogue flow that led to the invocation or selection of the voice action. The action crawler 110 may repeat this flow for each character in the application.

[0051] In another example, the action crawler 110 can invoke an application and then input a prompt or query that asks the application what tasks the application can perform. The action crawler 110 can then select a task and initiate one or more dialogue flows that result in the invocation of a voice action. Thus, the action crawler 110 can crawl the application to generate a transcript that includes actions that the application is configured to perform according to the one or more dialogues. The data processing system 102 can generate the transcript to store information about the action and the one or more dialogues used to invoke the action. The information can include dialogues, keywords, webref entities, vertical classifications, intent, voice actions, or other information that facilitates content selection, etc. The webref entities can correspond to entities classified in a taxonomy. Web reference entities aid in the understanding of text and augment the repository of knowledge. An entity can be a single person, place, or thing, and the repository can include millions of entities, each with a unique identifier to distinguish between multiple entities with similar names (e.g., Jaguar cars vs. Jaguar animals). The data processing system can access reference entities and scan any text (e.g., the text of a web page, the text of keywords, the text of content, the text of advertisements) to identify entities from a variety of sources. One such source can be, for example, a manually created taxonomy of entities, such as an entity graph of people, places, and things built by a community of users.

[0052] The data processing system 102 can receive a content item from the supplemental digital content provider device 132. The data processing system 102 can identify a content item provided by a third party of a computing device, such as the supplemental digital content provider device 132. For example, the content selector 114 of the data processing system 102 can receive a content item from the supplemental content provider device 132 over the network 105. The data processing system 102 can receive the content item (or sponsored content item) through an interface of the data processing system, such as a graphical user interface. The data processing system 102 can associate the content item with an account identifier associated with the supplemental content provider device 132. The content item can be part of a content placement campaign. The content item can be part of a content group. The content item can be associated with content selection criteria, such as keywords, location, or other signals used to perform content selection in real time by the content selector 114. The data processing system 102 can identify a concept, topic, entity, or vertical related to the content item or content group. The data processing system 102 can further identify or generate performance data for the content item. The performance data may include or indicate how well a content item performed when presented or offered for presentation on one or more client computing devices 136. Performance may refer to, include, or be based on the level or type of interaction with the content item by a user of a client computing device 136. For example, performance information may include or be based on selection of the content item, clicks on the content item, conversions, or completion of a prompt or survey related to the content item. The information may include, for example, whether a user closed the window in which the content item was presented, skipped playing the content item, fast-forwarded playing the content item, repeated playing the content item, or allowed the content item to play for the entire duration of the content item. Thus, the performance data for the content items may indicate a type of interaction with each of the content items in response to a previous presentation of each of the content items via one or more computing devices 136. The type of interaction may include at least one of skipping, fast-forwarding, or duration of presentation. The performance data may be stored in a data structure for the content data 126 in the data repository 120.

[0053] The performance data may be associated or correlated with keywords or other signals related to the content item or signals related to the location where the content item was presented. The signals may include the type of client computing device 136, the location of the client computing device 136 when the content item was presented, information related to the primary content being presented on the client computing device 136 alongside or in conjunction with the content item, or other account or profile information related to the client computing device 136.

[0054] However, due to the large number of content items received from the large number of supplemental digital content provider devices 132, as well as the large number of voice-based assistant actions and the large number of applications, it may be difficult to efficiently and accurately perform real-time content selection of content items. The data processing system 102 of this technical solution may include an embedding extractor 112 that is designed, constructed, and operable to generate, for each voice action, a bit vector indicating an associated candidate content item. The bit vector may be used by the content selector 114 to perform real-time content selection in response to the invocation of the voice action in the application running on the client computing device 136.

[0055] The embed extractor 112 can generate bit vectors corresponding to actions through the machine learning model and performance data of the content items. The bit vectors can indicate candidate content items for the actions. The embed extractor 112 can take as input a transcript generated by the action crawler 110 along with content item information or performance data, and generate bit vectors indicating which content items are associated with each action in the transcript. For example, the transcript can include multiple voice-based actions in a particular application. Each action in the transcript can be associated with a dialogue flow that includes keywords, intents, concepts, topics, or other information related to the simulated conversation that led to the voice action. The embed extractor 112 can use the deep learning model 128 to identify which content items are associated with each action in the transcript. The embed extractor 112 can generate one or more bit vectors for an application that identify multiple candidate content items for each action in the transcript generated by the action crawler 110 for the application.

[0056] A bit vector may refer to or include a data structure that can be efficiently processed by the data processing system 102 or one or more processors and memories thereof. A bit vector may effectively enable the data processing system 102 to perform bit-level parallelism in hardware to perform operations quickly. A bit vector may be an array of bits. A bit vector may include multiple bits. A bit vector may include a mapping of values ​​0 to 1 to indicate which content items are relevant for a particular action. For example, a bit vector for a voice action "read a book with Character_A" may include an indication of which candidate content items from a set of candidate content items are relevant by indicating a 1 or 0 at a location in the bit vector, where the location in the bit vector corresponds to a particular candidate content item. Table 1 shows example bit vectors for different actions in an application and a data structure that lists the candidate content items.

[0057] [Table 1]

[0058] In the illustrative example of Table 1, a candidate content item data structure for a particular application may list five different content items in an ordered list. The order may be based on any order, such as numerical order, alphabetical order, or order of relevance to the application. The embedding extractor 112 may generate a bit vector for each action in the transcript generated by the action crawler 110. For example, for action 1, the bit vector may be [0 0 1 1 1], for action 2, the bit vector may be [0 1 0 1 0], and for action 3, the bit vector may be [1 1 0 0 1]. The bit vector may indicate which of the content items are related to or candidates for a particular voice action. For example, for voice action 1, the related candidate content items may be content items 3, 4, and 5. For voice action 2, the candidate content items may be 2 and 4. For voice action 3, the candidate content items may be 1, 2, and 5.

[0059] The different voice actions shown in Table 1 may refer to different actions such as playing a song with Character A, reading a book, or playing music. In another example, the different actions may refer to actions of the same type but with different characters, such as playing a song, such that voice action 1 can be playing a song with Character_A, voice action 2 can be playing a song with Character_B, and voice action 3 can be playing a song with Character_C. The action crawler 110 can generate different voice action identifiers for the same type of action (e.g., playing a song) based on the dialogue flow that led to the invocation of the application by simulating a conversation with the application to generate a transcript file. Thus, Table 1 includes different bit vectors for each action based on the dialogue flow that led to the invocation of the voice action, which may result in improved accuracy of content selection while maintaining efficiency by using the bit vector format.

[0060] To generate the bit vector for each action, the data processing system 102 can train a deep learning model or machine learning model using the training data. The training data can include historical data related to previous presentations of the content item on numerous client computing devices 136. The historical training data can be based on the performance of the content item and signals related to the performance. The historical training data can include information about the content item (e.g., keywords, topics, entities), signals related to the presentation of the content item (e.g., the type of computing device, the location of the computing device, concepts, keywords, or topics related to the primary content with which the sponsored content is presented, or profile information), performance data (e.g., whether a user interacted with or skipped the content item).

[0061] The data processing system 102 can train the model 128 with the training data using machine learning techniques such as deep learning techniques. Deep learning techniques can refer to deep structured learning. Deep learning techniques can include or be based on artificial neural networks with representation learning. Deep learning techniques can be supervised, semi-supervised, or unsupervised. Deep learning techniques can include recurrent neural networks, convolutional neural networks, deep neural networks, or deep reinforcement learning. Deep learning techniques can use multiple layers to progressively extract high-level features from the input. For example, using content data (e.g., content item information and performance data) and transcripts (e.g., information related to voice actions and interactions that led to the invocation of the voice action), deep learning techniques can determine which content items are predicted to be relevant or perform well when presented alongside, in close spatial or temporal proximity to, or otherwise with or for a particular voice action. The embedding extractor 112 can use deep learning models to predict which content items are likely to perform well (e.g., have a likelihood of interaction that exceeds a threshold of 1%, 2%, 3%, 5%, 10%, 15%, 20% or more, etc., relative to the number of presentations of the content item). The embedding extractor 112 can use deep learning models to predict which content items are the best performing content with respect to a set of available content items received from one or more supplemental digital content provider devices 132. Thus, the embedding extractor 112 can take as input a transcript and performance data for a content item, and output a bit vector for each action in the transcript. The output bit vector represents the performance of the application by the client computing device 136 for a particular action. In response to an invocation of an action at option 142, candidate content items that are candidates for presentation may be indicated.

[0062] The data processing system 102 may include a content selector 114 that is designed, constructed, or operable to select supplemental content items (or sponsored content items or digital component objects). The content selector 114 may select a content item based on a bit vector corresponding to an action in response to a request for content for a computing device 136 that executes an action of an application 142. To avoid excessive computational resource utilization resulting from selecting from a large number of available content items, the content selector 114 may include a content retriever 116 that is designed, constructed, and operable to retrieve candidate content items for an action invoked in an application using the bit vector generated by the embedding extractor 112. The bit vector may store an indication of a subset of all available content items provided by the content items of the various supplemental digital content provider devices 132. The subset of all available content items may be the highest ranked content items or content items corresponding to the most relevant content items for the action of the application. The most relevant content items may be determined by the embedding extractor 112 using the deep learning model 128. This subset of content items may be listed or otherwise identified in the bit vector as candidates for presentation in response to invoking an action in the application 142. However, the content selector 114 may further narrow the subset of content items to select the highest scoring content items from the subset of content items based on signals or other information associated with the particular client computing device 136 that invokes the action in the application 142.

[0063] The content retriever 116 can identify the action and the application 142 on which the action was invoked. For example, the application may be associated with a unique identifier, such as the name of the application, an alphanumeric identifier, or other unique identifier. The content retriever 116 can further identify the action that was invoked in the application 142. The action may be associated with a unique identifier or an identifier unique to the application 142, such as a name, an alphanumeric identifier, or other valid identifier.

[0064] The content retriever 116 can perform a search in the bit vector 130 data structure using the application's identifier and the action's identifier to access, retrieve, obtain, or otherwise identify a bit vector established for the application's action. The content retriever 116 can identify one or more candidate content items indicated in the bit vector for the action. The content retriever 116 can identify a unique identifier for the content item, a uniform resource locator for the content item, a pointer to the content item, or other indicator of the content item that can be used to retrieve or provide the content item to the computing device 136.

[0065] The bit vector can indicate multiple candidate content items for a particular action invoked in the application 142. The candidate content items associated with the bit vector for the action can include content data 126, such as content selection criteria, performance data, or other information that can facilitate content selection. The content retriever 116 can provide the candidate content items indicated in the bit vector to the content scorer 118 to select a content item from the candidate content items to provide to the computing device 136.

[0066] The content selector 114 may include a content scorer 118 that is designed, constructed, and operable to select a content item from the candidate content items represented in the bit vector. The content selector 114 may score the candidate content items to identify the highest scoring or highest ranked content items. The content selector 114 may score the candidate content items using any content scoring technique that may facilitate identifying content items that are most likely to work well on the computing device 136. The content scorer 118 may score the candidate content items using signals associated with the particular client computing device 136 that invoked the action. The signals may include, for example, the type of computing device, the geographic location of the computing device, the language used by the client computing device (e.g., English, French, or Spanish), or profile information associated with an electronic account linked to the client computing device 136. For example, the content scorer 118 may predict which of the candidate content items are most likely to be relevant to an action invoked in the application 142 based on signals associated with the client computing device 136. The content scorer 118 can predict which of the candidate content items are most likely to result in a favorable interaction by a user of the computing device 136. The content scorer 118 can predict which of the candidate content items are most likely to be played for the entire duration of the content item without being skipped or closed by a user of the computing device 136.

[0067] To select sponsored content items or digital components, the content scorer 118 can use content selection criteria to select matching sponsored content items based on partial match, exact match, or phrase match. For example, the content selector 114 can analyze, parse, or otherwise process the subject matter of the candidate sponsored content items to determine whether the subject matter of the candidate sponsored content items corresponds to the subject matter of an intent, keyword, or phrase associated with an action invoked in the application 142. The content scorer 118 can identify, analyze, or recognize voices, audio, words, characters, text, symbols, or images of the candidate digital components using image processing techniques, character recognition techniques, natural language processing techniques, or database searches. The candidate sponsored content items can include metadata indicative of the subject matter of the candidate digital components, in which case the content scorer 118 can process the metadata to determine whether the subject matter of the candidate digital components corresponds to the action invoked in the application 142. The content campaigns provided by the supplemental digital content provider device 132 may include content selection criteria that the content scorer 118 can match to criteria expressed in the second profile layer or the first profile layer.

[0068] The content selector 114 can perform a real-time content selection process upon request. Real-time content selection can refer to or include performing content selection upon request. Real-time can refer to or include selecting content within 0.2, 0.3, 0.4, 0.5, 0.6, or 1 second of receiving a request. Real-time can refer to selecting content upon receiving an input audio signal from the client computing device 136. The real-time content selection process can be expedited by identifying a bit vector of an action invoked in the application 142 and inputting the candidate content items identified in the bit vector to the content scorer 118 to output the highest ranked or scored content item. The content scorer 118 can determine a score or rank for each of a plurality of candidate supplemental content items to select the highest ranked supplemental content item to provide to the client computing device 136.

[0069] In some cases, the content scorer 118 may be configured with a machine learning model that can determine the highest scoring or most relevant content items. The machine learning model may be trained based on the historical performance of content items associated with various signals. The model may output a prediction or score that indicates the likelihood that a content item will perform well in the presence of a particular signal. Thus, the content selector 114 may use machine learning techniques to score or select content items from the candidate content items retrieved by the content retriever 116.

[0070] Thus, the data processing system can receive an indication of the invocation of the action from the computing device 136. The data processing system 102 can perform a search using the action in the data repository 120 that stores the bit vectors 130 to retrieve a bit vector corresponding to the action. The data processing system 102 can identify top-ranked candidate content items from the bit vector for the action. The top-ranked candidate content items can be identified by the embedding extractor 112 using the deep learning model 128 and stored in the bit vector for the action of the application. The data processing system 102 can select the highest-scoring content item from the top-ranked candidate content items identified from the bit vector for the action. The data processing system 102 can use one or more signals associated with the computing device 136 performing the action to select a content item from the top-ranked candidate content items via a real-time content selection process.

[0071] The data processing system 102 can provide the selected content item to the client computing device 136 to have the client computing device 136 present the content item to a user of the computing device 136. The data processing system 102 can provide the content item to the local digital assistant 144 to have the local digital assistant 144 running on the client computing device 136 play the content item via audio output, such as via voice output. The data processing system 102 can, in some cases, provide the content item to the application 142 to have the application 142 present the content item. For example, the application 142 can output the content item by playing the content item using audio or visual output. The application 142 or the local digital assistant 144 can present or output the content item during, before, or after the execution of an action called in the application. In some cases, the computing device 136 can pause the execution of the action to play or output the content item. In some cases, the computing device 136 can wait to play the content item until the action has completed execution in the application 142. In some cases, the computing device 136 may wait a predetermined amount of time, such as 1 second, 2 seconds, 5 seconds, 10 seconds, 15 seconds, 30 seconds, or other time interval, after completing the execution of the action before playing the content item.

[0072] Upon presenting the content item, the data processing system 102 can receive an indication of a level of interaction with the content item. The data processing system 102 can store the level of interaction as performance data in a data structure of the content data 126. The data processing system 102 can retrain the machine learning model using the indication of the level of interaction to update a bit vector corresponding to the action, such as by the embedding extractor 112. For example, if the selected content did not perform well, the embedding extractor 112 can update the bit vector for the action to remove the content item from the bit vector such that the content item is likely not selected in response to a future invocation of the action in the application. In another example, the embedding extractor 112 can update the bit vector to lower the rank or score of the content item to reduce the likelihood that the content item will be selected in response to a future invocation of the action in the application. Thus, the data processing system 102 can improve the deep learning model 128 in response to real-time performance feedback, and the embedding extractor 112 can update the bit vector based on the retrained model.

[0073] FIG. 2 is a diagram of an example method for generating bit vectors for content matching for a third-party digital assistant action, according to an implementation. Method 200 may be performed by one or more systems or components shown in FIG. 1, including, for example, a data processing system. The data processing system may receive an application at 202. The data processing system may receive the application from an application developer. The application developer may provide the application to the data processing system. The application developer may upload the application to the data processing system or otherwise transmit or provide the application to the data processing system. For example, the application may be stored in a binary file, an executable file, an application package file, or another format that facilitates installation or execution of the application on the data processing system or client computing device.

[0074] At 204, the data processing system may identify actions in the application and generate a transcript. The data processing system may crawl the application to identify actions. The data processing system may identify actions by installing and running the application. The data processing system may simulate a conversation with the application to generate one or more dialogue flows that may result in the invocation or execution of actions in the application. The actions may be referred to as voice-assistant enabled actions because they may be invoked via a voice-based conversation with a user of the application. The actions may be invoked by speech from the user. The actions may be identified or selected via a conversation facilitated by a digital assistant running on the data processing system, or on a client computing device, or both.

[0075] The data processing system can generate a transcript of the identified actions. The transcript can be generated for a particular application. The transcript can show various actions identified in the application and information related to the dialogue flow or conversation that led to the invocation of the actions. For example, the transcript can include information about the simulated utterances that led to the invocation of the actions. The transcript can include utterances by the simulated user and the application interacting with the simulated user.

[0076] At 206, the data processing system may receive content data. The content data may refer to supplemental content items. The data processing system may receive the content data from one or more supplemental content providers, such as advertisers. The data processing system may receive the content data from a data repository accessible to the data processing system that stores the content data. The content data may include content items, such as digital component objects, and performance data associated with historical or prior presentations of the content items.

[0077] At 208, the data processing system may generate bit vectors for one or more actions indicated in the transcript. To do so, the data processing system may use a deep learning model trained on historical content performance data. The data processing system may input the transcript and content data related to the application into the deep learning model to generate an embedding or bit vector or bit array for each action that indicates a subset of content items as being candidate content items for presentation in response to invocation of the action in the application. By generating bit vectors that indicate a subset of content items, the data processing system may improve the efficiency and accuracy of the real-time content selection process.

[0078] FIG. 3 is a diagram of an exemplary method for selecting content using a bit vector in response to invocation of a third-party digital assistant action, according to an implementation. Method 300 may be performed by one or more systems or components shown in FIG. 1, including, for example, a data processing system. At 302, the data processing system may receive an indication of invocation of an in-application action. The in-application action may refer to or include a voice assistant-enabled action. For example, the action may be triggered, invoked, or launched in response to voice input from a user of a client computing device. The action may be invoked by a local digital assistant executing on the client computing device. The action may be invoked or analyzed by an NLP component executing on the client computing device or the data processing system.

[0079] In response to the invocation of an action, the client computing device may send an indication to the data processing system. The indication sent to the data processing system may include information about an application, an action, or a signal associated with the computing device. For example, the information sent to the data processing system may include an identifier of the action or an identifier of the application. The signal may include, for example, information related to the geographic location of the computing device, the type of computing device, or a profile or electronic account linked to the computing device. In some cases, the signal may include information about preferences associated with the computing device, such as preferences for types of content.

[0080] The client computing device can send an indication to the data processing system via a local digital assistant, an application, or other component running on the client computing device. For example, an application can send an indication to the data processing system in response to an invocation of an action. The local digital assistant can detect the invocation of the action and package a transmission to the data processing system that includes an identifier for the action or application along with other signals associated with the computing device.

[0081] At 304, the data processing system may perform a search in the data repository to identify a bit vector for an action in the application. The data processing system may perform the search using an identifier of the action and the application. At 306, the data processing system may identify candidate content items from the bit vector. The data processing system may perform a search in the data repository using information received from the client computing device to retrieve or access a bit vector configured for the action. The bit vector may include candidate content items associated with the action.

[0082] At 308, the data processing system may select the highest scoring candidate content item from the bit vector. The data processing system may use any scoring technique to determine the score of each of the content items. The data processing system may use the scores to rank the content items to determine the highest ranked content item. The data processing system may determine the score of each content item based on or using the signal received from the computing device. For example, if a content item is associated with a geographic location that is the same as the geographic location of the client computing device, the data processing system may score the content item higher than another content item that may not be associated with the geographic location of the client computing device. In another example, the client computing device may be configured to use the English language, and a French content item may be scored lower by the data processing system compared to an English content item.

[0083] At 310, the data processing system may send the selected content item to the client computing device to cause the client computing device to present the content item. The data processing system may include instructions regarding how to present the content item. The data processing system may provide instructions regarding when to present the content item, which output device to use to present the content item, or interactive features to allow for the content item. The data processing system may provide the content item to a component of the client computing device that sent an indication of the call to action. For example, the data processing system may return the content item to an application or a local digital assistant running on the client computing device.

[0084] At 312, the data processing system may determine the performance of the content item. The data processing system may determine the performance of the content item based on whether the user interacted with the content item or the type of interaction with the content item. The performance may be based on whether the user allowed the content item to play in its entirety, whether the user skipped playing the content item, whether the user closed the content item, canceled the content item, fast-forwarded through the content item, or otherwise interacted with the content item. For example, if the content item is an audio clip and the computing device played the entire duration of the audio clip, the data processing system may determine that the content item performed successfully. However, if the user skipped playing the content item, the data processing system may determine that the content item did not perform successfully.

[0085] At 314, the data processing system may update the machine learning model based on the performance of the content item. The data processing system may update the machine learning model or deep learning model used to generate the bit vector based on the performance of the content item. For example, if the content item did not perform well, the data processing system may remove the content item from the bit vector. If the content item performed well, the data processing system may keep the content item in the bit vector as a candidate content item or increase the score or weight associated with the candidate content item.

[0086] 4 is a block diagram of an exemplary computer system 400. The computer system or computing device 400 may include or be used to implement the system 100 or components thereof, such as the data processing system 102 or the client computing device 136. The data processing system 102 or the client computing device 136 may include an intelligent personal assistant or a voice-based digital assistant. The computing system 400 includes a bus 405 or other communication component for communicating information and a processor 410 or processing circuitry coupled to the bus 405 for processing information. The computing system 400 may also include one or more processors 410 or processing circuits coupled to the bus for processing information. The computing system 400 also includes a main memory 415, such as a random access memory (RAM) or other dynamic storage device coupled to the bus 405 for storing information and instructions executed by the processor 410. The main memory 415 may be or include the data repository 120. The main memory 415 may also be used for storing location information, temporary variables, or other intermediate information during execution of instructions by the processor 410. The computing system 400 may further include a read-only memory (ROM) 420 or other static storage device coupled to the bus 405 for storing static information and instructions for the processor 410. A storage device 425, such as a solid-state device, magnetic disk, or optical disk, may be coupled to the bus 405 for persistently storing information and instructions. The storage device 425 may include or be part of the data repository 120.

[0087] The computing system 400 may be coupled via bus 405 to a display 435, such as a liquid crystal display or active matrix display, for displaying information to a user. An input device 430, such as a keyboard including alphanumeric and other keys, may be coupled to the bus 405 for communicating information and command selections to the processor 410. The input device 430 may include a touch screen display 435. The input device 430 may also include a cursor control, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the processor 410 and for controlling cursor movement on the display 435. The display 435 may be part of, for example, the data processing system 102 of FIG. 1, or the client computing device 136, or other component.

[0088] The processes, systems, and methods described herein may be implemented by computing system 400 in response to execution by processor 410 of an array of instructions contained in main memory 415. Such instructions may be read into main memory 415 from another computer-readable medium, such as storage device 425. Execution of an array of instructions contained in main memory 415 causes computing system 400 to perform the exemplary processes described herein. One or more processors of a multi-processing arrangement may also be used to execute instructions contained in main memory 415. Hardwired circuitry may be used in place of or in combination with software instructions with the systems and methods described herein. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0089] Although an exemplary computing system is illustrated in FIG. 4, the subject matter including the operations described herein may be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, or in combinations of one or more of them, including the structures disclosed herein and structural equivalents thereof.

[0090] With regard to situations in which the systems discussed herein may collect or utilize personal information about the user, the user may be given the opportunity to control whether a program or feature may collect personal information (e.g., information about the user's social network, social actions or activities, the user's preferences, or the user's location) or whether or how content that may be more relevant to the user should be received from a content server or other data processing system. Additionally, certain data may be anonymized in one or more ways before it is stored or used, such that personally identifiable information is removed when generating parameters. For example, the user's identity may be anonymized such that personally identifiable information cannot be determined about the user, or if location information is obtained (such as to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific location cannot be determined. Thus, the user may control how information is collected about the user and used by the content server.

[0091] The objects and operations described herein may be implemented in digital electronic circuitry, or computer software, firmware, or hardware, or a combination of one or more of them, including the structures disclosed herein and their structural equivalents. The objects described herein may be implemented as one or more circuits of one or more computer programs, e.g., computer program instructions encoded on one or more computer storage media for execution by or to control the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be or may be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. The computer storage medium is not a propagated signal, but may be a source or destination of computer program instructions encoded on an artificially generated propagated signal. Also, a computer storage medium may be or may be included in one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0092] The terms "data processing system", "computing device", "component", or "data processing apparatus" encompass a variety of apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system-on-chip, or a plurality or combination thereof. An apparatus may include dedicated logic circuitry, e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program in question, e.g., code constituting a processor's firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as a web services infrastructure, a distributed computing infrastructure, and a grid computing infrastructure. For example, the action crawler 110, the embedded extractor 112, or other components may include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0093] A computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple associated files (e.g., a file that stores one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one location or distributed across multiple locations and connected together by a communication network.

[0094] The processes and logic flows described herein may be executed by one or more programmable processors (e.g., components of the data processing system 102) executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be executed by, and apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0095] The subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or includes a combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0096] A computing system such as system 100 or system 400 may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communications network (e.g., network 105). The relationship of clients and servers arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server sends data (e.g., data packets representing digital components) to a client device (e.g., for the purpose of displaying the data to a user interacting with the client device and receiving user input from such a user). Data generated at a client device (e.g., a result of a user interaction) may be received at the server from the client device (e.g., received by data processing system 102 from client computing device 136 or supplemental digital content provider device 132).

[0097] Although operations are shown in the figures in a particular order, such operations do not have to be performed in the particular order shown, or in a sequential order, and not all shown operations need be performed. Actions described herein may be performed in different orders.

[0098] The division of various system components need not be required in all implementations, and the described program components may be included in a single hardware or software product. For example, the natural language processor 106 and the interface 104 can be part of a single component, app, or program, or a logical device having one or more processing circuits, or one or more servers of the data processing system 102.

[0099] Having now described several exemplary implementations, it should be apparent that the foregoing is provided by way of illustrative and not limiting examples. In particular, while many of the examples presented herein include specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to achieve the same purpose. Acts, elements, and features discussed in connection with one implementation are not intended to be excluded from similar roles in other implementations or implementations.

[0100] The language and terminology used herein are for purposes of description and should not be regarded as limiting. The use herein of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized in that," and variations thereof, is intended to encompass the preceding recited items, equivalents of those items, and additional items, as well as alternative implementations consisting only of the preceding recited items. In one implementation, the systems and methods described herein consist of one, a combination of two or more of each, or all of the described elements, acts, or components.

[0101] All references herein to system and method implementations or elements or acts in the singular may also encompass implementations that include a plurality of those elements, and all references herein to any implementation or element or act in the plural may also encompass implementations that include only a single element. The singular or plural references are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to a single or multiple configuration. References to any act or element that is based on any information, act, or element may include implementations in which the act or element is based at least in part on the any information, act, or element.

[0102] Any implementation disclosed herein may be combined with any other implementations or embodiments, and references to "implementations," "several implementations," "an implementation," and the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with an implementation may be included in at least one implementation or embodiment. Such terms as used herein do not necessarily all refer to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0103] References to "or" may be considered inclusive such that any term listed using "or" may refer to any one, more than one, and all of the listed terms. Reference to at least one of a conjunctive list of terms may be considered an inclusive OR, referring to any one, more than one, and all of the listed terms. For example, a reference to "at least one of 'A' and 'B'" may include "A" only, "B" only, and both "A" and "B." Such references used in conjunction with "comprising" or other open-ended terms may include additional terms.

[0104] Where technical features of the drawings, detailed description, or any claims are subsequently labeled with a reference sign, the reference sign has been included to facilitate understanding of the drawings, detailed description, and claims, and thus the presence or absence of the reference sign shall not have any limiting effect on the scope of any claim element.

[0105] The systems and methods described herein may be embodied in other specific forms without departing from their characteristics. The above-described implementations are illustrative rather than limiting of the described systems and methods. Accordingly, the scope of the systems and methods described herein is indicated by the appended claims rather than the above description, and all changes that come within the meaning and range of equivalence of the claims are intended to be embraced therein. [Explanation of symbols]

[0106] 100 Systems 102 Data Processing System 104 Interface 105 Network 106 NLP, Natural Language Processor 108 Server Digital Assistant 110 Action Crawler 112 Embedded Extractors 114 Content Selector 116 Content Retriever 118 Content Scorer 120 Data Repositories 122 Applications 124 Transcript 126 Content Data 128 Models 130 Bit Vector 132 Supplemental Digital Content Provider Devices 134 Application Developer Devices 136 Client Computing Devices 138 Input Devices 140 Output Devices 142 Applications 144 Local Digital Assistant 200 ways 300 ways 400 Computing system, computer system, computing device 405 Bus 410 Processor 415 Main Memory 420 Read-Only Memory (ROM) 425 Storage Devices 430 Input Devices 435 Display

Claims

1. receiving, from an application developer, an application configured with voice assistant enabled actions, wherein one or more actions can be invoked in the application; identifying a plurality of actions that the application is configured to perform in response to voice input; Identifying a plurality of content items provided by a plurality of third party computing devices; generating a plurality of bit vectors corresponding to the plurality of actions by inputting into a machine learning model performance data indicating how well each of the plurality of content items performed when provided and a transcript including a dialogue flow that led to the invocation of the action, the plurality of bit vectors indicating a plurality of candidate content items for each action of the plurality of actions and indicating which of the candidate content items are associated with each action in the transcript; In response to a request for content for a computing device that is to execute an action of the application, selecting, from among the plurality of candidate content items, a candidate content item that is predicted to be most likely to be relevant to the action invoked in the application based on the plurality of bit vectors corresponding to the action. A system including a data processing system including a memory and one or more processors for performing the above.

2. 2. The system of claim 1, further comprising the data processing system for crawling the application to identify the plurality of actions the application is configured to perform in response to voice input.

3. A system according to claim 1 or 2, further comprising said data processing system for simulating one or more interactions with said application to identify said plurality of actions.

4. 1. The data processing system for crawling the application to generate a transcript including the plurality of actions that the application is configured to perform according to one or more interactions, 4. The system of claim 1, further comprising a data processing system, wherein the transcript stores information about the plurality of actions and the one or more interactions used to invoke the plurality of actions.

5. 5. The system of claim 1, wherein the performance data for the plurality of content items indicates a type of interaction with each of the plurality of content items in response to previous presentation of each of the plurality of content items via one or more computing devices.

6. The system of claim 5 , wherein the type of interaction includes at least one of skipping, fast-forwarding, or duration of presentation.

7. receiving an indication of the invocation of the action from the computing device; performing a lookup using the action in a data repository that stores the plurality of bit vectors to retrieve the bit vector corresponding to the action; identifying a number of top-ranked candidate content items from the bit vector for the action; Selecting the content item from the plurality of top-ranked candidate content items identified from the bit vector for the action.

7. The system according to claim 1, comprising the data processing system for:

8. The system of claim 7, further comprising: the data processing system for using one or more signals associated with the computing device performing the action to select the content item from the plurality of top-ranked candidate content items by a real-time content selection process.

9. receiving an indication of a level of interaction with the content item selected in response to the action and presented via the computing device; retraining the machine learning model with the indication of the level of interaction to update the bit vector corresponding to the action.

9. The system according to claim 1, comprising the data processing system for:

10. 10. The system of claim 1, wherein each of the plurality of bit vectors stores a subset of the plurality of content items corresponding to a highest ranked content item.

11. Receiving, by a data processing system including a memory and one or more processors, from an application developer, an application configured with voice assistant enabled actions, the application capable of invoking one or more actions; identifying, by the data processing system, a number of actions that the application is configured to perform in response to voice input; identifying, by the data processing system, a plurality of content items provided by a plurality of third party computing devices; generating, by the data processing system, a plurality of bit vectors corresponding to the plurality of actions by inputting into the machine learning model performance data indicating how well the plurality of content items performed when provided and a transcript including a dialogue flow that led to the invocation of the action, the plurality of bit vectors indicating a plurality of candidate content items for each action of the plurality of actions and indicating which candidate content items are associated with each action in the transcript; selecting, by the data processing system in response to a request for content for a computing device that is to execute an action of the application, from among the plurality of candidate content items, a candidate content item that is predicted to be most likely to be relevant to the action invoked in the application based on the plurality of bit vectors corresponding to the action; The method includes:

12. 12. The method of claim 11, further comprising crawling the application by the data processing system to identify the actions the application is configured to perform in response to voice input.

13. 13. A method according to claim 11 or 12, comprising simulating, by said data processing system, one or more interactions with said application to identify said plurality of actions.

14. 14. A method according to any one of claims 11 to 13, comprising crawling the application by the data processing system to generate a transcript including the plurality of actions that the application is configured to perform according to one or more interactions, the transcript storing information about the plurality of actions and the one or more interactions used to invoke the plurality of actions.

15. 15. A method according to any one of claims 11 to 14, wherein the performance data for the plurality of content items indicates a type of interaction with each of the plurality of content items in response to a previous presentation of each of the plurality of content items via one or more computing devices.

16. The method of claim 15 , wherein the type of interaction includes at least one of skipping, fast-forwarding, or duration of presentation.

17. receiving, by the data processing system, an indication from the computing device of the invocation of the action; performing, by the data processing system, a search using the action in a data repository that stores the plurality of bit vectors to retrieve the bit vector corresponding to the action; identifying, by the data processing system, a number of top-ranked candidate content items from the bit vector for the action; selecting, by the data processing system, the content item from the plurality of top-ranked candidate content items identified from the bit vector for the action; 17. The method according to any one of claims 11 to 16, comprising:

18. 20. The method of claim 17, comprising using, by the data processing system, one or more signals associated with the computing device performing the action to select the content item from the plurality of top-ranked candidate content items through a real-time content selection process.

19. receiving, by the data processing system, an indication of a level of interaction with the content item selected in response to the action and presented via the computing device; retraining, by the data processing system, the machine learning model with the indication of the level of interaction to update the bit vector corresponding to the action; 19. The method according to any one of claims 11 to 18, comprising:

20. 20. The method of claim 11, wherein each of the plurality of bit vectors stores a subset of the plurality of content items corresponding to a highest ranked content item.

Images