System and method for processing dependent sequence operations

Abstract

A system and method for processing operations involving dependent sequences. A natural language processor component is capable of parsing an input audio signal to identify a request and a triggering keyword. A prediction component is capable of determining a thread based on the triggering keyword and the request, which includes a first action, a second action following the first action, and a third action following the second action. A content selector component is capable of selecting a content item based on the third action and the triggering keyword. An audio signal generator component is capable of generating an output signal including the content item. Before at least one of the first and second actions occurs, the interface is capable of transmitting the output signal to cause a client computing device to drive a speaker to generate a sound wave corresponding to the output signal.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201780001427.5, filed on August 31, 2017.

[0003] Cross-references to related applications

[0004] This application claims the benefit and priority of U.S. Patent Application No. 15 / 395,682, filed December 30, 2016, entitled “SEQUENCE DEPENDENT OPERATIONPROCESSING OF PACKET BASED DATA MESSAGE TRANSMISSIONS”, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0005] This invention relates to the operation processing of dependent sequences in packet-based data message transmission. Background Technology

[0006] Excessive packet-based or other network traffic transmissions between computing devices can prevent them from properly processing network traffic, completing operations related to it, or responding promptly. If the responding computing device is at or above its processing capacity, excessive network traffic transmissions can also complicate data routing or degrade response quality, potentially leading to inefficient bandwidth utilization. Controlling network transmissions corresponding to content item objects can become complex due to the large number of content item objects that can initiate network traffic transmissions between computing devices. Summary of the Invention

[0007] At least one aspect relates to a system for optimizing the processing of dependent sequence operations in a voice-activated, packet-based computer network environment. A natural language processor component, executed by a data processing system, is capable of receiving data packets. The data packets may include input audio signals detected by sensors of a client computing device. The natural language processor component is capable of parsing the input audio signals to identify a request and a trigger keyword corresponding to the request. A prediction component is capable of determining a thread based on the trigger keyword and the request. The thread may include a first action, a second action following the first action, and a third action following the second action. The prediction component is capable of providing an indication of the third action to a content selector component of the data processing system before at least one of the first and second actions occurs. Based on the third action and the trigger keyword identified by the natural language processor, the content selector component is capable of selecting content items via a real-time content selection process. An audio signal generator component, executed by the data processing system, is capable of generating an output signal including the content items. The interface of the data processing system is capable of transmitting data packets including the output signal generated by the audio signal generator component before at least one of the first and second actions occurs, so that the audio driver component executed by the client computing device drives the speaker of the client computing device to generate sound waves corresponding to the output signal.

[0008] At least one aspect relates to a method for invoking a sequence-dependent operation in a voice-activated, data-packet-based computer network environment. The method includes a natural language processor component executed by a data processing system receiving data packets comprising input audio signals detected by sensors of a client computing device; the method includes the natural language processor component identifying a request and a trigger keyword corresponding to the request based on the input audio signals. The method includes a prediction component determining a thread based on the trigger keyword and the request. The thread may include a first action, a second action following the first action, and a third action following the second action. The method may include the prediction component providing an indication of the third action to a content selector component of the data processing system prior to the occurrence of at least one of the first and second actions. The method may include the content selector component selecting a content item via a real-time content selection process based on the third action and the trigger keyword identified by the natural language processor. The method may include an audio signal generator component executed by the data processing system obtaining an output signal comprising the content item. The method can include transmitting, via the interface of the data processing system, a data packet comprising an output signal obtained by the audio signal generator component before at least one of the first and second actions occurs, such that an audio driver component executed by at least one of the client computing device and the second client computing device drives the speaker of the client computing device to generate a sound wave corresponding to the output signal.

[0009] At least one aspect relates to a computer-readable storage medium storing instructions that, when executed by one or more data processors, cause the one or more data processors to perform operations to select a dependent sequence of operations in a voice-activated, packet-based computer network environment. The operations are capable of receiving data packets comprising input audio signals detected by sensors of a client computing device, via a natural language processor component executed by a data processing system. The operations are capable of identifying a request and a triggering keyword corresponding to the request based on the input audio signals, via the natural language processor component. The operations are capable of selecting a thread based on the triggering keyword and the request, via a prediction component. The thread is capable of including a first action, a second action following the first action, and a third action following the second action. The operations are capable of providing an indication of the third action to a content selector component of the data processing system prior to the occurrence of at least one of the first and second actions, via the prediction component. The operations are capable of selecting content items via a real-time content selection process, via the content selector component, based on the third action and the triggering keyword identified by the natural language processor. The operations are capable of generating an output signal including the content items, via an audio signal generator component executed by the data processing system. The operation is capable of transmitting, via the interface of the data processing system, a data packet including an output signal generated by the audio signal generator component before at least one of the first and second actions occurs, so that an audio driver component executed by the client computing device drives the speaker of the client computing device to generate sound waves corresponding to the output signal.

[0010] These and other aspects, as well as implementation methods, are discussed in detail below. The information above and the detailed description below include illustrative examples of various aspects and implementation methods, and provide an overview or framework for understanding the nature and characteristics of the claimed aspects and implementation methods. The accompanying drawings provide illustration and further understanding of the various aspects and implementation methods, and are incorporated in and constitute a part of this specification. Attached Figure Description

[0011] The accompanying drawings are not intended to be drawn to scale. The same reference numerals and labels in each drawing indicate the same elements. For clarity, each component may not be labeled in every drawing. In the drawings:

[0012] Figure 1 A system is described that optimizes the processing of operations dependent on a given sequence in a voice-activated computer network environment;

[0013] Figure 2 Describe the function diagram of a thread that includes actions that depend on a sequence;

[0014] Figure 3 This describes a method for invoking sequence-dependent operations in a speech-activated, packet-based computer network environment; and

[0015] Figure 4 This is a block diagram illustrating the overall architecture of a computer system that can be used to implement the systems and methods described and shown herein. Detailed Implementation

[0016] The following describes in more detail various concepts and implementations of methods, apparatuses, and systems for optimizing the processing of operations on dependent sequences in a voice-activated computer network environment. The various concepts introduced above and discussed in more detail below can be implemented in any of several ways.

[0017] The systems and methods disclosed herein generally relate to a data processing system that optimizes or dynamically adjusts the order of operations in a processing-dependent sequence via packet-based transmission in a voice-activated computer network environment. The data processing system can improve the efficiency and effectiveness of packet transmission over one or more computer networks, for example, by asynchronously, out of order, or dynamically selecting operations from a plurality of dependent sequence operations. Packet-based or other protocol-based signals corresponding to the selected operations can be routed among multiple computing devices via the computer network. The data processing system can temporarily or permanently skip the initial operation of a set of dependent sequence operations and instead proceed to the operation corresponding to a later or final operation. By bypassing packet-based or other protocol-based data transmission corresponding to earlier operations in a plurality of dependent sequence operations and instead directly proceeding to packet-based data transmission for later operations in the sequence, the data processing system can reduce, delay, or eliminate data processing associated with earlier operations. This saves processing power and other computing resources such as memory, reduces the power consumption of the data processing system, and the reduced data transmission via the computer network reduces the bandwidth requirements and usage of the data processing system.

[0018] The systems and methods described herein can include a data processing system that receives input audio queries. From the input audio query, the data processing system can identify a request and a trigger keyword corresponding to that request. Based on the trigger keyword or request, the data processing system can determine a thread that indicates operations in a multiple dependent sequence. For example, the thread can include a first action, a second action following the first action, and a third action following the second action. In this example, the third action, which is the last action in the thread of dependent sequence operations, can be used by the data processing system to select a content item before the first action and before the second action occur. This can be done before or after performing at least some of the data processing operations based on the first or second action. By bypassing these processing operations, the content item for the third action can be selected using less data processing and less power consumption compared to the case without bypassing them.

[0019] The content item can include an output signal. The data processing system can provide the content item, including the output signal, to the client computing device via packet-based or other protocol-based data message transmission over a computer network. The output signal enables the audio driver component of the client computing device to generate sound waves that can be output from the client computing device, such as audio output. The audio output corresponds to a third (e.g., the latest or most recent in time) action and can be output from the client computing device before the first action or before the second action occurs. Bypassing the first and second actions (or by bypassing the selection and provision of the content item corresponding to these actions) and selecting and sending the content item corresponding to the third action before the actions in the first and second dependent sequences occur results in fewer data processing operations, less memory usage, and less network bandwidth utilization than without bypassing the first and second actions.

[0020] Figure 1An example system 100 is described, illustrating the processing of operations involving dependent sequences in a voice-activated, packet-based (or other protocol-based) computer network environment. System 100 may include at least one data processing system 105. Data processing system 105 may include at least one server having at least one processor. For example, data processing system 105 may include multiple servers located in at least one data center or server farm. Data processing system 105 is able to determine a request and a triggering keyword associated with that request from an audio input signal. Based on the request and triggering keyword, data processing system 105 is able to determine or select a thread comprising multiple dependent sequences of operations, and is able to select content items (and initiate other actions as described herein) in a sequence that does not match the associated operations, for example, as part of a voice-activated communication or planning system. Content items may include one or more audio files that provide audio output or sound waves when presented. In addition to or in lieu of audio content, content items may also include other content (e.g., text, video, or image content). For example, content items may include text or image files or combinations thereof that do not include audio files and do not present audio output.

[0021] Data processing system 105 can include multiple logically grouped servers and facilitate distributed computing technologies. The logical group of servers can be referred to as a data center, server cluster, or machine farm. Servers can be geographically distributed. A data center or machine farm can be managed as a single entity, or the machine farm can comprise multiple machine farms. Servers within each machine farm can be heterogeneous – one or more servers or machines can operate according to one or more types of operating system platforms. Data processing system 105 can include servers stored in one or more high-density rack systems within a data center, as well as associated storage systems, such as those located in an enterprise data center. In this way, data processing system 105 with unified servers can improve system manageability, data security, system physical security, and system performance by positioning servers and high-performance storage systems on a localized high-performance network. Centralizing all or some of the components of data processing system 105, including servers and storage systems, and coupling them with advanced system management tools allows for more efficient use of server resources, saving power and processing demands and reducing bandwidth consumption.

[0022] The data processing system 105 may include at least one Natural Language Processor (NLP) component 110, at least one interface 115, at least one prediction component 120, at least one content selector component 125, at least one audio signal generator component 130, at least one Direct Action Application Programming Interface (API) 135, at least one session handler component 140, and at least one data repository 145. The NLP component 110, interface 115, prediction component 120, content selector component 125, audio signal generator component 130, direct action API 135, and session handler component 140 may each include at least one processing unit, server, virtual server, circuitry, engine, agent, electrical appliance, or other logical device, such as a programmable logic array configured to communicate with the data repository 145 and other computing devices (e.g., client computing device 150, content provider computing device 155, or service provider computing device 160) via at least one computer network 165. Network 165 can include computer networks such as the Internet, local area networks, wide area networks, metropolitan area networks or other domain networks, intranets, satellite networks, other computer networks such as voice or data mobile phone communication networks, and combinations thereof.

[0023] Network 165 can include or constitute a display network, such as a subset of available information resources on the Internet that are associated with content placement or search engine results systems or are eligible to include third-party content items as part of content placement activities. Network 165 can be used by data processing system 105 to access information resources that can be presented, output, displayed, or rendered by client computing device 150, such as web pages, websites, domain names, or Uniform Resource Locators. For example, via network 165, users of client computing device 150 can access information or data provided by content provider computing device 155 or service provider computing device 160.

[0024] Network 165 can include, for example, a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, an SDH (Synchronous Digital Hierarchy) network, a wireless network, or a wired network, and combinations thereof. Network 165 can include wireless links, such as infrared channels or satellite bands. The topology of network 165 can include bus, star, or ring network topologies. Network 165 can include a mobile phone network using any or all protocols for communication between mobile devices, including Advanced Mobile Phone Protocol (“AMPS”), Time Division Multiple Access (“TDMA”), Code Division Multiple Access (“CDMA”), Global System for Mobile Communications (“GSM”), General Packet Radio Service (“GPRS”), or Universal Mobile Telecommunications System (“UMTS”). Different types of data can be transmitted via different protocols, or the same type of data can be transmitted via different protocols.

[0025] Client computing device 150, content provider computing device 155, and service provider computing device 160 may each include at least one logical device, such as a computing device with a processor, for communicating with each other or with the data processing system 105 via network 165. Client computing device 150, content provider computing device 155, and service provider computing device 160 may each include at least one server, processor, or memory, or multiple computing resources or servers located in at least one data center. Client computing device 150, content provider computing device 155, and service provider computing device 160 may each include at least one computing device, such as a desktop computer, laptop computer, tablet computer, personal digital assistant, smartphone, portable computer, server, thin client computer, virtual server, or other computing device.

[0026] The client computing device 150 may include at least one sensor 151, at least one converter 152, at least one audio driver 153, and at least one speaker 154. The sensor 151 may include a microphone or an audio input sensor. The converter 152 may convert audio input into an electronic signal. The audio driver 153 may include a script or program executed by one or more processors of the client computing device 150 to control the sensor 151, converter 152, or audio driver 153, and other components of the client computing device 150 to process audio input or provide audio output. The speaker 154 may emit an audio output signal.

[0027] The client computing device 150 can be associated with an end user who inputs a voice query as audio input to the client computing device 150 (via sensor 151) and receives computer-generated speech output from a speaker 154, which can be provided to the client computing device 150 from the data processing system 105 (or content provider computing device 155 or service provider computing device 160). The computer-generated speech can include recordings from real people or computer-generated language.

[0028] Content provider computing device 155 can provide audio-based content items for display as audio output content items by client computing device 150. Content items can include quotations for goods or services, such as voice-based messages stating, "Would you like me to order you a taxi?" For example, content provider computing device 155 can include memory to store a series of audio content items that can be provided in response to voice-based queries. Content provider computing device 155 can also provide audio-based content items (or other content items) to data processing system 105, where they can be stored in data repository 145. Data processing system 105 can select audio content items and provide them to client computing device 150 (or instruct content provider computing device 155 to provide them). Audio-based content items can be audio only or can be combined with text, image, or video data.

[0029] Service provider computing device 160 may include at least one service provider natural language processor (NLP) component 161 and at least one service provider interface 162. Service provider NLP component 161 (or other components such as the direct action API of service provider computing device 160) may engage with client computing device 150 (via or around data processing system 105) to create real-time, voice- or audio-based dialogues (e.g., sessions) between client computing device 150 and service provider computing device 160. For example, service provider interface 162 may receive or provide data messages to direct action API 135 of data processing system 105. Service provider computing device 160 and content provider computing device 155 may be associated with the same entity. For example, content provider computing device 155 may create, store, or produce available content items for a car-sharing service, and service provider computing device 160 may establish a session with client computing device 150 to arrange for a taxi or car delivering the car-sharing service to pick up the end user of client computing device 150. Through the Direct Action API 135, NLP component 110, or other components, the data processing system 105 can also establish a session with the client computing device (including or bypassing the service provider computing device 160) to, for example, arrange the delivery of taxis or cars for a car-sharing service.

[0030] Data repository 145 may include one or more local or distributed databases and may include a database management system. Data repository 145 may include computer data storage areas or storage and may store one or more parameters 146, one or more policies 147, content data 148 or templates 149, and other data. Parameters 146, policies 147, and templates 149 may include information such as rules relating to voice-based conversations between client computing device 150 and data processing system 105 (or service provider computing device 160). Content data 148 may include content items for audio output or associated metadata, and input audio messages that may be part of one or more communication conversations with client computing device 150.

[0031] System 100 can optimize the processing of operations dependent on a data packet (or other protocol) environment activated by voice. For example, data processing system 105 can include voice-activated assistance services, voice command devices, intelligent personal assistants, knowledge navigators, event planners, or other assistance programs as part of a process. Data processing system 105 can provide one or more instances of audio output displayed from client computing device 150 to perform tasks related to the input audio signal. These tasks can include, for example, communicating with service provider computing device 160 or other third-party computing devices to make a dinner reservation or purchase movie tickets. For example, an end user can input an audio signal to client computing device 150: “OK, I would like to go to go dinner and then a movie tonight.”

[0032] Operations in a dependent sequence can include several actions that occur in a known, suggested, required, or specific order. For example, going from home to the movie theater and then home can include three operations or actions in a dependent sequence. In this example, the first action includes going from home to the movie theater. The second action includes watching a movie in the theater. The third action includes going home from the theater. In this example, the second action (watching a movie) in the sequence follows and depends on the first action (going out for dinner), and the third action (going home) follows and depends on both the first and second actions. The third action can be considered to follow and depend on the second action because it has been expressed that this is intentional; the data processing system 105 predicts that the end user will watch a movie in the theater before going home.

[0033] The data processing system 105 may include an application, script, or program, such as an app, installed on the client computing device 150, for communicating input audio signals to the interface 115 of the data processing system 105 and driving components of the client computing device to present output audio signals. The data processing system 105 may receive data packets or other signals that include or identify audio input signals. For example, the data processing system 105 may execute or run NLP component 110 to receive audio input signals.

[0034] NLP component 110 can convert an audio input signal into recognizable text by comparing the input signal with a stored set of representative audio waveforms (e.g., in a data repository 145) and selecting the closest match. The representative waveforms are generated among a large number of users and can be augmented with speech samples. After the audio signal is converted into recognizable text, NLP component 110 can match the text with words associated with actions, for example, through training among users or by manually setting such words.

[0035] The audio input signal can be detected by the sensor 151 (e.g., microphone) of the client computing device. The client computing device 150 can provide the audio input signal to the data processing system 105 (e.g., via network 165) via a converter 152, an audio driver 153, or other components, wherein the audio input signal can be received (e.g., via interface 115) and provided to the NLP component 110 or stored in the data repository 145 as content data 148.

[0036] NLP component 110 can acquire an input audio signal. From the input audio signal, NLP component 110 can identify at least one request or at least one trigger keyword corresponding to the request. The request can indicate the intent or topic of the input audio signal. The trigger keyword can indicate the type of action that may be taken. For example, NLP component 110 can parse the input audio signal to identify at least one request to leave home in the evening to attend dinner and a movie. The trigger keyword can include at least one word, phrase, root word, or part of a word, or a derived word indicating the action to be taken. For example, the keyword "go" or "to go to" in the input audio signal can indicate the need for transportation. In this example, the input audio signal (or the identified request) does not directly express an intent to travel; however, the trigger keyword indicating transportation is an auxiliary action to at least one other action indicated by the request.

[0037] Prediction component 120 (or other mechanism of data processing system 105) is capable of determining at least one thread associated with the input audio signal based on a request or trigger keyword. A thread can indicate a set of operations that depend on a sequence, such as a series of actions. A thread can include any two or more actions, such as a first action, a second action, and a third action. For example, the input audio signal “OK, I would like to go to go dinner and then a movie tonight” can include at least one request indicating interest in attending dinner and the movie, and at least one trigger keyword indicating a need for transportation, such as “go”. Prediction component 120 is capable of identifying threads with at least three actions, such as the dinner action (first action), the movie action (second action), and the transportation home action (third action). In this example, prediction component 120 predicts, evaluates, or otherwise determines three actions from the request or trigger keyword. A thread can include other actions, such as the initial transportation to dinner action.

[0038] Data processing system 105 or its components (such as prediction component 120) can determine that the actions of a thread are sequential operations. For example, the prediction component can determine that the second action of watching a movie follows the first action of eating dinner, and that the third action of going home from the movie theater follows the second action of watching a movie. Prediction component 120 can access parameters 146 or policies 147 in data repository 145 to determine or otherwise evaluate the order of actions that are sequential. For example, parameter 146 or policy 147 can indicate that a transportation action (e.g., taking a taxi home) will occur after an event action (e.g., watching a movie).

[0039] Content selector component 125 can receive indications of any of the actions in the thread. For example, prediction component 120 can provide content selector component 125 with an indication of a third (or any other) action, directly or indirectly (e.g., via data repository 145). Content selector component 125 can obtain this information from data repository 145, where it can be stored as part of content data 148. The indication of the third action can inform content selector component 125 that transportation is needed from the location of the cinema to a location determined or indicated by client computing device 150, such as cycling to a final destination.

[0040] Content selector component 125 is capable of obtaining indications of later actions in a thread before at least one earlier action occurs. For example, content selector component 125 can receive indications of a third action (e.g., needing to take a car from the movie theater) before a movie is playing in a theater (second action) or before a person inputting an input audio signal to client computing device 150 is having dinner in a restaurant (first action). Content selector component 125 is also capable of obtaining indications of at least one action of a thread before at least one action of the thread completes.

[0041] From information received by the content selector component 125, such as an indication of at least one third action prior to the preceding action in a thread of a dependent sequence, the content selector component 125 can identify at least one content item. The content item can be in response to or related to the third action. For example, in response to a third action indicating a need for transportation, the content item can include an audio message providing services from a car-sharing company. The content selector component 125 can select or otherwise identify content items, for example, by querying the data repository 145 from the content data 148. The content selector component 125 can also select content items from the content provider computing device 155. For example, in response to a query received from the data processing system 105, the content provider computing device 155 can provide content items to the data processing system 105 (or its components) for final output by the client computing device 150.

[0042] Audio signal generator component 130 is capable of generating or otherwise obtaining an output signal that includes content items in response to a third action. For example, data processing system 105 can execute the audio signal generator component to generate or create an output signal corresponding to a content item. Interface 115 of data processing system 105 can provide the transmission of one or more data packets, including the output signal, to client computing device 150 via computer network 165. Interface 115 can be designed, configured, constructed, or operated to, for example, use data packets to receive and transmit information. Interface 115 can use one or more protocols, such as network protocols, to receive and transmit information. Interface 115 can include a hardware interface, a software interface, a wired interface, or a wireless interface. Interface 115 can facilitate the conversion or formatting of data from one format to another. For example, interface 115 can include an application programming interface that includes definitions for communication between various components, such as software components of system 100.

[0043] For example, data processing system 105 can provide output signals to client computing device 150 from data storage unit 145 or from audio signal generator component 130. Data processing system 105 can also provide output signals to client computing device 150 via data packet transmission instruction content provider computing device 155 or service provider computing device 160. The output signals can be acquired, generated, converted into, or transmitted from data processing system 105 (or other computing device) to client computing device 150 as one or more data packets (or other communication protocols).

[0044] In this example, the content selector component 125 jumps to a later (e.g., third) action in the set of operations in the dependent sequence to select the content item for the third action before the activity corresponding to the third action occurs (and before immediate need). By jumping to a later-ordered action, the data processing system 105 does not need to process information about the first or second actions to select the content items for those actions. This reduces the processor utilization, power consumption, and bandwidth required in data transmission if the content items associated with the (first action) or (second action) content items were selected before the content items for the third action were selected.

[0045] As part of the real-time content selection process, the content selector component 125 is capable of selecting content items for a (subsequent or later) third action. For example, content items can be provided to a client computing device to be transmitted as audio output in a conversational manner directly in response to an input audio signal. The real-time content selection process for identifying content items and providing them to the client computing device 150 can occur within one minute or less of the input audio signal time and is considered to be real-time.

[0046] For example, an output signal corresponding to a content item, transmitted to the client computing device 150 via interface 115 and computer network 165 and obtained or generated by audio signal generator component 130, can cause the client computing device 150 to execute audio driver 153 to drive speaker 154 to generate sound waves corresponding to the output signal. The sound waves can include words of the content item or corresponding words. The output signal corresponding to the content item can also include non-audio content items presented as text or image messages that can be presented in the absence of audio elements on the client computing device 150 or different client computing devices 150 (e.g., wearable computing devices), which serve as the source of the input audio signal.

[0047] Sound waves can be output from client computing device 150 before the first or second action occurs. For example, the sound waves can include the audio output "Would you like a taxi waiting for you after the movie?". In this example, data processing system 105 receives an input audio signal such as "OK, I would like to go to dinner and then a movie tonight." From this information, NLP component 110 identifies at least one request or at least one trigger keyword, and prediction component 120 uses the request or trigger keyword to identify at least one thread. Threads include a series of actions that depend on a sequence, such as the first action of having dinner, the second action of watching the movie, and the third action of getting home from the movie theater. Prediction component 120 is able to determine (e.g., at least at a threshold confidence level, such as 75% confidence) that these actions will occur sequentially, where the end user of client computing device 150 will first have dinner, then go to the movie theater after dinner, and then go home from the movie theater or leave the movie theater after dinner. Content selector component 125 can, for example, bypass the processing associated with selecting content items for the first and second actions, and instead select the content item for the third action first, such as the content item for a car-sharing service used to arrange traffic at a movie theater pick-up location. Data processing system 105 or its components (such as audio signal generator component 130) can provide the content item for the third action to client computing device 150. Sound waves can be output from client computing device 150 after the first action occurs and before the second action occurs, for example, between preceding actions in a thread.

[0048] The third content item can be presented at least partially as audio output from the client computing device 150 before the first or second action has occurred. For example, the output signal including the third action content item can be presented as human or automated speech, such as "Would you like a ride home from the movie theater?". The data processing system 105 can receive a response to the third action content item in an ordered sequence of actions before one, more, or all of the preceding actions have occurred. For example, the data processing system 105 can provide the content item for presentation and receive a response, for example, via interface 115, before the end user has gone to dinner or watched a movie. The data processing system 105 is capable of providing output signals that include content items that respond directly or in real-time to the input audio signal of “OK, I would like to go to dinner and then a movie tonight,” so that packet-based data transmission via computer network 165, as part of a communication session between the data processing system 105 and the client computing device 150, has a smooth and human-like feel during real-time conversation. This packet-based data transmission communication session can also include either the content provider computing device 155 or the service provider computing device 160.

[0049] Content selector component 125 is capable of selecting, from the action set of the dependent sequence, a content item provided in response to a third (or any non-first) action, before selecting any content item for any prior action in the selection sequence. This avoids the need to process at least some of the data corresponding to prior actions, and allows the third content item to be selected with fewer processing operations compared to selecting content items in an order that matches the order of the action sequence. For example, after selecting a content item for the third action, content items for prior (e.g., first and second) actions may or may not be selected.

[0050] Content selector component 125 is capable of selecting content items (e.g., content items for a third action) based on at least one trigger keyword indicating a topic different from the topic of one or more requests from the same thread. For example, a request in response to a thread saying "OK, I would like to go to go dinner and then a movie tonight" could indicate the topic of a dinner request (first action) and the topic of a movie request (second action). This topic does not indicate any transportation request. However, NLP component 110 or prediction component 120 (or other components of data processing system 105 that perform as part of direct action API 135) can identify the trigger keywords "go," "go to," or "to go to," and can determine the transportation request (third action) based at least in part on the trigger keywords. Therefore, data processing system 105 can infer actions from the input audio signal or the topic of the input audio signal as a secondary request that is not a primary request.

[0051] Data processing system 105 can prevent or delay the transmission of output signals including content items. For example, interface 115 or a script executed via Direct Action API 135 can prevent the transmission of data packets corresponding to the output signals of content items before a triggering event. The triggering event can include the expiration of a predetermined time period, such as two hours, or a time corresponding to the occurrence of an earlier action in a thread, such as the predetermined end time of a movie. The triggering event can also include an authorization instruction received by data processing system 105 from client computing device 150. The authorization instruction can originate from subsequent audio input associated with the thread but received by the data processing system after an input audio signal, text-based signal, or other inaudible signal, or can include an instruction authorizing settings of client computing device 150 received by client computing device 150 of the content item.

[0052] The data processing system 105 can also delay the transmission of content items associated with or following a second action of a thread to optimize processing utilization. For example, the data processing system 105 can delay the transmission of content items before off-peak hours or periods of data center usage, thereby utilizing the data center more effectively by reducing peak bandwidth usage, heat output, or cooling requirements. Based on data center utilization or bandwidth metrics or requirements of network 165, or the data processing system 105, the data processing system 105 can also initiate conversions or other activities associated with content items, such as booking car services.

[0053] Content items can correspond to actions of a thread, and the content selector component 125 can select content items for one, more, or all of the actions of a thread. The prediction component 120 can identify threads with dependent sequences of actions, such as "dinner" (first action), "movie" (second action), and "travel from movie location" (third action). The direct action API 135 can execute programs or scripts, such as those from the NLP component 110, the prediction component 120, or the content selector component 125, to identify content items for one or more of these actions. The direct action API 135 can execute specified actions determined by the data processing system 105 to satisfy the intent of the end user. Based on the actions specified in its input, the direct action API 135 can execute code or dialogue scripts that identify the parameters required to satisfy the user's request. Such code can, for example, look up additional information in the data repository 145, such as the name of a home automation service, or it can provide audio output for presentation at the client computing device 150 to ask the end user questions, such as the expected destination of the requested taxi. The Direct Action API 135 can determine the necessary parameters and encapsulate the information into an action data structure, which can then be sent to another component, such as the content selector component 125, or to the service provider computing device 160 to be satisfied.

[0054] The Direct Action API 135 of the data processing system 105 can generate a data structure of actions for a thread based on a request or trigger keyword. The action data structure can be generated in response to a request. The action data structure can be included in a message sent to or received by the service provider computing device 160. Based on the request parsed by the NLP component 110, the Direct Action API 135 can determine which of the multiple service provider computing devices 160 should send the message to. For example, if the input audio signal includes "order a taxi," the NLP component 110 can identify the trigger word "order" and the request for a taxi. The Direct Action API 135 can encapsulate the request into an action data structure for transmission as a message to the service provider computing device 160 of the taxi service. The message can also be passed to the content selector component 125. The action data structure can include information for completing the request. In this example, the information can include the pick-up location and the destination location. The Direct Action API 135 can retrieve a template 149 from the repository 145 to determine which fields to include in the action data structure. The Direct Action API 135 can retrieve content from the repository 145 to obtain information about the fields of the data structure. The Direct Action API 135 can use this information to populate the fields in the template to generate the data structure. The Direct Action API 135 can also populate the fields with data from the input audio signal. The template 149 can be standardized for a category of service provider or for a specific service provider. For example, a ride-sharing service provider can use the following standardized template 149 to create a data structure: {client_device_identifier;authentication_credentials; pick_up_location; destination_location; no_passengers; service_level}.

[0055] For example, in response to an audio input signal, a third action content item can be provided to present an audio response at the client computing device 150, such as "Would you like a ride home from the movie theater?". Before or after this presentation, the data processing system 105 can select or provide content items, such as "The Italian restaurant downtown has an opening for 7 pm tonight?" for the first action (dinner), and can select or provide another content item, such as "Would you like tickets to the 9 pm movie at the downtown theater?" for the second action (movie). The data processing system 105 can provide (including instructing the content provider computing device 155 to provide) this content in response to an input audio signal so that it can be presented as an audio output content item by the client computing device 150. The data processing system 105 can select these content items or provide them to the client computing device 150 in any order via interface 115. For example, it is possible to select or provide the content item for a third (e.g., last) action before the content items corresponding to other earlier actions of a thread.

[0056] Data processing system 105 can receive a response to the content item “Would you like a ride home from the movie theater?”. This response can include an input audio signal, such as “yes please,” entered by the end user into client computing device 150 and received by data processing system 105. Based on this response, direct action API 135 can communicate with service provider computing device 160 (which can be associated with the content item, such as a car-sharing company) to book a taxi or ride-sharing vehicle at the movie theater location at the end of the movie. Data processing system 105 can obtain this location or time information from data repository 145 or other sources such as service provider computing device 160 or content provider computing device 155 as part of data packet-based (or other protocol) data message communication with client computing device 150. The confirmation of the reservation (or other conversion) can be provided from the data processing system 105 to the client computing device 150 as an audio communication output signal. This output signal drives the client computing device 150 to display audio output, such as "great, you will have a car waiting for you at 11 pm outside the theater." The data processing system 105 can communicate with the service provider computing device 160 via the Direct Action API 135 to confirm the car reservation.

[0057] Data processing system 105 is able to receive a response (e.g., "yes please") to a content item ("Would you like a ride home from the movie theater?") and is able to route the packet-based data message to service provider NLP component 161 (or other components of the service provider computing device). This packet-based data message enables service provider computing device 160 to make a conversion, for example, instructing a car to accept a reservation outside the movie theater. The order of this conversion or confirmation (or any other conversion of any other action of a thread) can occur before one or more actions of the thread are completed, such as before the movie ends, and after one or more actions of the thread are completed, such as after dinner.

[0058] Based on the response to a content item of a subsequent action in a thread, such as the response "yes please" to the content item "Would you like a ride home from the movie theater?" in a dependent sequence of threads, the data processing system 105 can initiate a transformation or action. This can occur before the end user begins any activity associated with the thread, for example, before dinner or before the movie ends. The processor of the data processing system 105 can call the Direct Action API 135 to execute a script that generates a car reservation through a car-sharing service. The Direct Action API 135 can obtain content data 148 (or parameters 146 or policy 147) from the data repository 145 and data that the end user agrees to receive from the client computing device 150 to determine location, time, user account, logic, or other information in order to reserve a car from the car-sharing service. Using the Direct Action API 135, the data processing system 105 can also communicate with the service provider computing device 160 to complete a transformation, in this example, a car-sharing pick-up reservation.

[0059] The Direct Action API 135 can initiate a conversation or activity to complete an action of the thread at any time during the period when the data processing system 105 receives the first input audio signal associated with the thread (e.g., "OK, I would like to go to dinner and then a movie tonight"), until a threshold period of time after one of the actions in the thread is completed (e.g., until 15 minutes after the second action of watching the movie ends). The data processing system 105 can also determine the end of this period of time based on other factors, such as an indication that the end user has completed the thread's action, for example, the end user has gone home or is no longer at the movie theater. The data processing system 105 can also obtain from the client computing device 150 an indication that the thread has been canceled or put to sleep, or that the end user has canceled or completed the thread's action (data messages transmitted via network 165).

[0060] Based on the completion or completion time of other actions within the thread, the Direct Action API 135 can initiate a transition or activity to complete the thread's action. For example, the data processing system 105 can directly or through communication with the service provider's computing device 160 to reserve goods or services (car-sharing pick-up) within a threshold time period after the end of a prior action in the thread. Within 5 minutes (or other time period) before or after the scheduled or actual end of a movie (second action), the data processing system 105 can confirm or reserve a car-sharing service to dispatch a vehicle to pick up the end user from the movie theater (subsequent third action). This process occurs if no input is received from the end user's computing device 150 during this time period, or in response to a prompt received from the end user's computing device 150 during this time period.

[0061] The Direct Action API 135 can initiate transformations or other activities associated with operations in a thread-dependent sequence in any order. For example, the Direct Action API 135 (or other components of the data processing system 105) can initiate an activity corresponding to the final action (e.g., the third action in a three-action thread, such as booking a taxi) before initiating an activity corresponding to an earlier thread in the sequence (e.g., before making a dinner reservation or booking movie tickets). In this example, the data processing system 105 initiates an earlier action (e.g., the second action) after initiating a later (e.g., the third) action.

[0062] Content selector component 125 can identify, select, or obtain multiple content items generated by multiple content selection processes. The content selection process can be real-time, for example, as part of the same dialogue, communication session, or a series of communication sessions between the data processing system 105 and the client computing device 150 involving a thread. The dialogue can include asynchronous communications spaced apart by periods of time, such as hours or days. The dialogue or communication session can continue for a period of time from the receipt of the first input audio signal until the expected or known end of the thread's final action, when the data processing system 105 receives an indication of thread termination. This can be referred to as the thread's active period. For example, for a thread with three actions (e.g., dinner, movie, and commuting home), content selector component 125 can select at least one content item for each action. Content item selector component 125 can run or obtain results from different content selection processes for each action. During the content selection process, content provider computing device 150 can set the content items for content selector component 125 to select. For a thread having at least three actions, the content item selector component can select or otherwise identify a first content item via a first content selection process, select or otherwise identify a second content item with a second action via a second content selection process, and select or otherwise identify a third content item with a third action via a third content selection process. These content items are provided so that the data processing system 105, presented by the client computing device 150 during an active period, can be considered to be operating in real time. In this example, the content selection process and the presentation of content items occur in real time.

[0063] Data processing system 105 can cancel actions associated with content items. For example, after booking a car service, an end user might watch a movie but then decide to walk home or have dessert after the movie instead of taking a car home. As part of data processing system 105, an app or other program executing at client computing device 150 can receive an input audio signal of "cancel my ride home". NLP component 110 can receive this input audio signal, for example, as one or more packet-based data messages, and can determine that the input audio signal is associated with a pre-existing thread (e.g., dinner, movie, commuting home) and is part of the same conversation or communication. For example, NLP component 110 (or other components of data processing system 105) can evaluate time data, location data, and pre-existing thread data, such as the past, present, or scheduled actions of the thread, from content data 148 in data repository 145 to determine that the input audio signal is associated with a pre-existing thread. For example, based on the request "ride" or the trigger keyword "cancel," NLP component 110 can determine that the input audio signal "cancel my ride home" is related to the content item "Would you like a ride home from the movie theater?" corresponding to a pre-existing thread's third action. Direct Action API 135 (or other components) enables data processing system 105 to communicate via interface 115 with service provider computing device 160 to cancel taxi or car-sharing service bookings that would otherwise be waiting for end-users outside the movie theater after the movie ends.

[0064] NLP component 135 can receive messages based on data packets or other protocols to cancel thread actions before or after any action of the thread is completed. NLP component 135 can receive data messages (incoming audio signals) from client computing device 150 (or obtain them from data repository 145) to cancel thread actions within a time interval triggered by an earlier action of the thread. For example, in a thread with a dependent sequence of three actions: dinner, movie, and transportation, data processing system 105 can receive a transportation cancellation data message 5 minutes (or other time interval) after the movie ends. Data processing system 105 can prompt for confirmation of products or services related to the actions of the thread. For example, in the time interval between the first action and the second action (e.g., after dinner and before the movie), data processing system 105 can send a data message to client computing device 150 that, when presented at client computing device 150, outputs an audio or text message stating, "just confirming that you still want a car waiting for you after the movie." The data processing system 105 can receive responses such as "yes, confirmed" or "no, cancel the car". The direct action API 135 can execute scripts to process this information and can communicate with the service provider computing device 160 to indicate confirmation or cancellation.

[0065] Figure 2 A functional diagram is depicted of thread 200, which has a dependent sequence of actions including first action 205, second action 210, and third action 215. Thread 200 can be determined by components of data processing system 105, such as prediction component 120. Data processing system 105 is able to receive input audio signals, for example as data packets, via interface 115 at time T-0 (T-zero). Data processing system 105 is able to determine that time T-0 is the starting point of the active period 220 of thread 200.

[0066] Based on a request or triggering keyword obtained from an input audio signal (e.g., “OK, I would like to go to dinner and then watch a movie tonight”), the prediction component is able to identify multiple dependent sequences of actions, such as a first action 205 (e.g., dinner), a second action 210 (e.g., movie), and a third action 215 (e.g., taking a car home after watching a movie), which are predicted to occur in that order, and at least one action depends on at least one preceding action occurring. For example, the third action 215 (taking a car home from the movie theater) depends on the occurrence of the second action (the end user arriving at the movie theater to watch a movie).

[0067] Data processing system 105 and components such as content selector component 125 and interface 115 are capable of providing content items for presentation as audio output by client computing device 150 during the active time period 220 of thread 200. For example, during time period T-1, i.e., the period from the start of active time period 220 to the start of first action 205 (e.g., dinner reservation time), for third action 215, data processing system 105 is capable of selecting and providing content items (e.g., “Would you like a ride home from the movie theater?”). Data processing system 105 is capable of selecting or providing any content item associated with any action at any time during active time period 220. For example, during time period T-1 or any other time period of active time period 220, it is capable of providing the first content item (of third action 215) or different content items (e.g., movie recommendations or movie ticket offers) for third action 215. Data processing system 105 is also capable of determining that content items have timed out and will not provide any further presentation for them during a portion of active time period T1. For example, after the data processing system 105 determines that the second action 210 (e.g., a movie at time T-4) has begun, the data processing system 105 is able to determine that the content items of the first action 205 (e.g., dinner at time T-2) will not be provided. In this example, the content items of the first action 205 can be provided to the client computing device between time T-1 (before the first action 205) and time T-3 between the first action 205 and the second action 210.

[0068] During time periods T-1 to T-6, and even during time period T-7, for example, after the data processing system 105 determines that the movie has ended but the end user has chosen to instruct the data processing system 105 to stay out all night, at least one content item for the third action 215 (e.g., taking a car home) can still be provided. During the time period T-5 between the second action 210 and the third action 215, or during any other time period of the active time period 220, such as T-6 (during the third action 215) or T-7 (after the third action 215), the data processing system 105 can provide content items related to the third action 215 to repeat sales or prompt confirmation actions, such as confirming a car reservation for a car-sharing service.

[0069] Figure 3 Method 300 describes actions of invoking sequence-dependent operations in a voice-activated, packet-based computer network environment such as system 100. The method is capable of receiving data packets (ACT 305). For example, data processing system 105 can execute, initiate, or invoke NLP component 110 to receive packet-based or other protocol-based transmissions from client computing device 150 via a network. Data packets can include or correspond to input audio signals detected by sensor 151, such as an end user saying to a smartphone, "OK, I would like to go to dinner and then a movie tonight."

[0070] Method 300 is capable of identifying at least one request or at least one trigger keyword from the input audio signal (ACT 310). For example, NLP component 110 is capable of parsing the input audio signal to identify a request (“dinner” or “movie”) and a trigger keyword corresponding to or related to that request, such as “go”, “go to”, or “togo to”. Based on the request or trigger keyword, method 300 is capable of determining at least one thread (ACT 315). A thread can include a series of dependent sequence actions that the data processing system 100 determines will occur in a set order, such as thread 200, where a first action occurs at time T-2, a second action 210 occurs at time T-4, and a third action 215 occurs at time T-6 after the second action 210 and after the first action 205.

[0071] Method 300 can provide an indication of a third action (ACT 320) to the content selector component 125. For example, the content selector component 125 can receive the indication of the third action 215 before the first action 205 occurs (before time T-2) or before the second action 210 occurs (before time T-4). In response to the indication of the third action, or based on a request and trigger keyword, the content selector component can select at least one content item corresponding to the third action (ACT 325). Method 300 can generate at least one output signal corresponding to the content item (ACT 330) and can provide the output signal to the client computing device 150 (ACT 335). For example, interface 115 can transmit data packets including the output signal from data processing system 105 to client computing device 150. Data processing system 105 can delay or prevent the transmission of data packets before data processing system 105 detects the occurrence of a condition such as the expiration of a time period T1 to T7 or before receiving authorization from client computing device 150.

[0072] The output signal enables the client computing device 150 (e.g., an app or other component of the data processing system 105 residing and executing on the client computing device 150) to drive the speaker 154 to generate sound waves corresponding to the output signal (and content item). The audio output from the content item of the client computing device 150 can occur before the first action 205 occurs (e.g., before time T2) or before the second action 210 occurs (e.g., before time T-4). Method 300 can receive a response (ACT 340) to the content item from the client computing device 150. For example, the data processing system 105 can receive a response signal (ACT 340) generated from an audio input to the client computing device requesting the initiation or taking of an action, such as booking a taxi or purchasing event tickets. The Direct Action API 135 can invoke a program to take or initiate the requested action. The action can be initiated by the Direct Action API based on the received response signal (ACT 340) and other factors such as the action completion time, for example, the end time of the second action 210 at the boundary between time T-4 and time T-5. The action can also be cancelled. For example, after receiving the response signal (ACT 340), the data processing system 105 can cancel the action due to the cancellation or termination of the thread's action, a thread change, or by receiving a cancellation request from the client computing device 150.

[0073] Figure 4This is a block diagram of an example computer system 400. The computer system or computing device 400 may include or be used to implement system 100 or its components, such as data processing system 105. The computing system 400 includes a bus 405 or other communication components for communicating information and a processor 410 or processing circuitry coupled to the bus 405 for processing information. The computing system 400 may also include one or more processors 410 or processing circuitry coupled to the bus for processing information. The computing system 400 also includes a main memory 415 coupled to the bus 405 for storing information, such as random access memory (RAM) or other dynamic storage devices, and instructions executed by the processor 410. The main memory 415 may be or include a data storage repository 145. The main memory 415 may also be used to store location information, temporary variables, or other intermediate information during instruction execution by the processor 410. The computing system 400 may further include a read-only memory (ROM) 420 or other static storage device coupled to the bus 405 for storing static information and instructions for the processor 410. Storage device 425, such as a solid-state device, disk, or optical disk, can be coupled to bus 405 to persistently store information and instructions. Storage device 425 can include or be part of a data storage repository 145.

[0074] The computing system 400 can be coupled to a display 435, such as a liquid crystal display or an active matrix display, via a bus 405 for displaying information to a user. An input device 430, such as a keyboard including alphanumeric and other keys, can be coupled to the bus 405 for communicating information and command selection to the processor 410. The input device 430 can include a touchscreen display 435. The input device 430 can also include a cursor controller, such as a mouse, trackball, or cursor arrow keys, for communicating directional information and command selection to the processor 410 and for controlling cursor movement on the display 435. For example, the display 435 can function as a data processing system 105, a client computing device 150, or... Figure 1 It is part of the other components.

[0075] The processes, systems, and methods described herein can be implemented by a computing system 400 in response to an arrangement of processor 410 executing instructions contained in main memory 415. Such instructions can be read into main memory 415 from another computer-readable medium, such as storage device 425. The arrangement of executing the instructions contained in main memory 415 causes the computing system 400 to perform the illustrative processes described herein. One or more processors in a multiprocessor arrangement may also be used to execute the instructions contained in main memory 415. Hardwired circuitry can be used in conjunction with the systems and methods described herein, either in place of or in combination with software instructions. The systems and methods described herein are not limited to any particular combination of hardware circuitry and software.

[0076] Although Figure 4 Example computing systems have been described, but the subjects described in this specification, including those related to operation, can be implemented in other types of digital circuits, or computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof.

[0077] In situations where the systems discussed in this article collect or can use personal information about users, users may have the opportunity to control whether programs or functions can collect personal information (e.g., information about the user's social networks, social actions or activities, user preferences, or user location), or to control whether or how they receive content from content servers or other data processing systems that may be more relevant to the user. Furthermore, certain data can be anonymized in one or more ways before being stored or used, so that personally identifiable information is removed when generating parameters. For example, a user's identity can be anonymized so that personally identifiable information cannot be determined against the user, or the user's geographic location can be generalized to the locations where location information is obtained (such as city, zip code, or state level) so that the user's specific location cannot be determined. Therefore, users can control how information about themselves is collected and used by content servers.

[0078] The subject matter and operations described in this specification can be implemented as digital electronic circuits, or computer software, firmware, or hardware, including the structures and their equivalents disclosed in this specification, or combinations thereof. The subject matter described in this specification can be implemented as one or more computer programs, such as one or more circuits of computer program instructions encoded on one or more computer storage media, for execution by or control of their operation by a data processing device. Alternatively or additionally, the program instructions can be encoded on artificially generated propagating signals, such as machine-generated electronic, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof, or included therein. Although the computer storage medium is not a propagating signal, it can be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium can also be one or more separate components or media (e.g., multiple CDs, disks, or other storage devices), or included therein. The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0079] The terms "data processing system," "computing device," "component," or "data processing apparatus" encompass a variety of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination thereof. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may include code that creates an execution environment for a computer program under consideration, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement infrastructure for various computing models, such as web services, distributed computing, and grid computing infrastructures. Direct Action API 135, content selector component 125, prediction component 120, or NLP component 110, and other data processing system components 105 may include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0080] Computer programs (also known as programs, software, software applications, apps, scripts, or code) can be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored as markup language documents), as a single file dedicated to the program under consideration, or as multiple co-located files (e.g., multiple parts storing one or more modules, subroutines, or code). A computer program can be deployed to execute on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected via a communication network.

[0081] The processes and logic flows described in this specification can be executed by one or more programmable processors (e.g., components of data processing system 105) that execute one or more computer programs to perform actions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuits, and the apparatus can also be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices including, for example, EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memories can be combined with or integrated with dedicated logic circuits.

[0082] The subject matter described herein can be implemented in computing systems that include, for example, backend components as data servers, middleware components such as application servers, frontend components such as client computers having graphical user interfaces or web browsers that users can interact with embodiments of the subject matter described herein, or combinations of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0083] Computing systems such as System 100 or System 400 can include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network (e.g., network 165). The client-server relationship is implemented using computer programs running on respective computing devices and having a client-server relationship with each other. In some implementations, the server transmits data (e.g., data packets representing content items) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from that user). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server (e.g., by data processing system 105 from computing device 150, content provider computing device 155, or service provider computing device 160).

[0084] Although the operations are depicted in a specific order in the diagrams, it is not necessary to perform them in that specific order or sequentially, nor is it necessary to perform all of the illustrated operations. The actions described herein can be performed in different orders.

[0085] Separation of various system components is not necessary in all implementations, and the program components can be included in a single hardware or software product. For example, NLP component 110, content selector component 125, or prediction component 120 can be a single component, app, or program, or a logic device with one or more processing circuits, or part of one or more servers of data processing system 105.

[0086] Some illustrative embodiments have been described, but it is clear that the above is presented illustratively by way of example only and is not intended to be restrictive. Specifically, while many of the examples presented herein involve specific combinations of method actions or system elements, those actions and elements can be combined in other ways to achieve the same objective. The actions, elements, and features discussed in connection with one embodiment are not intended to exclude similar effects in other embodiments.

[0087] The terms and terminology used herein are for descriptive purposes and should not be considered as limiting. The use of “comprising,” “including,” “having,” “containing,” “involving,” “characterized as,” “characterized in,” and variations thereof is intended to cover the items listed thereafter, their equivalents, and additional items, as well as alternative implementations consisting only of the items listed thereafter. In one implementation, the system and method described herein consist of one or more of the elements, actions, or components, or all of each combination or all of them.

[0088] Any reference to an embodiment, element, or action of a system or method mentioned herein in the singular may also cover an embodiment that includes multiple such elements, and any reference to any embodiment, element, or action herein in the plural may also cover an embodiment that includes only a single element. References in the singular or plural form are not intended to limit the currently disclosed systems or methods, their components, actions, or elements to a single or multiple configurations. References to any action or element based on any information, action, or element may include actions or elements that are at least partially based on an embodiment of that information, action, or element.

[0089] Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to "an implementation," "some implementations," "one implementation," etc., are not necessarily mutually exclusive, but are intended to indicate that a particular feature, structure, or characteristic described in connection with an implementation may be included in at least one implementation or embodiment. These terms used herein do not necessarily all refer to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0090] The use of "or" can be interpreted as inclusive, meaning that any term used with "or" can refer to a single term, more than one term, or all of the terms. For example, referring to at least one of "A" and "B" can include only "A", only "B", or both "A" and "B". Such references, when used in conjunction with "include" or other open-ended terms, can include additional terms.

[0091] Where reference numerals follow technical features in the drawings, detailed description, or any claims, these reference numerals are included to improve the understanding of the drawings, detailed description, and claims. Therefore, the presence or absence of reference numerals has no limiting effect on the scope of any claim element.

[0092] The systems and methods described herein may be embodied in other specific forms without departing from the characteristics of the invention. For example, data processing system 105 may select content items for subsequent actions (e.g., third action 215) in part based on data from a previous action in a sequence of actions of thread 200, such as data from second action 210 indicating that second action 210 has been completed or is about to begin. The foregoing embodiments are illustrative only and are not intended to limit the systems and methods described herein. Therefore, the scope of the systems and methods described herein is indicated by the appended claims rather than the foregoing description, and variations within the equivalent meaning and scope of the claims are included therein.

Claims

1. A system for processing sequence-dependent operations in a voice-activated, packet-based computer network environment, comprising: A data processing system including memory and one or more processors, used for: Receives input data packets including input audio signals detected by sensors of the client computing device; The input audio signal is analyzed to identify keywords; An action sequence is determined based on the keywords, the action sequence including a first action, a second action following the first action, and a third action following the second action; Bypass at least one of the first action and the second action to generate an output signal including the content item; as well as Data packets including the output signal are transmitted to the client computing device to drive the speaker of the client computing device to generate sound waves corresponding to the output signal.

2. The system of claim 1, wherein, The data processing system is configured to prevent the data packet, including the output signal, from being transmitted to the client computing device before either the expiration of a defined time period or before receiving an authorization instruction from the client computing device.

3. The system according to claim 1, wherein, The data processing system receives from the client computing device a packet-based data message, which is recorded as audio input to the client computing device, for canceling the third action.

4. The system according to claim 1, wherein, The data processing system is configured to receive a packet-based data message for canceling the third action within a time interval triggered by either the first action or the second action.

5. The system according to claim 1, wherein, The data processing system is used to transmit the data packet including the output signal after the first action occurs and before the second action occurs, so as to drive the speaker of the client computing device to generate sound waves corresponding to the output signal.

6. The system according to claim 1, wherein, The data processing system is configured to receive a data message, including a response to the content item, from the client computing device prior to the occurrence of at least one of the first action and the second action.

7. The system according to claim 1, wherein, The data processing system is used to initiate the third action based on one of the completion times of the first action and the second action.

8. The system according to claim 1, wherein, The data processing system is used to initiate the third action after at least one of the first action and the second action has been completed.

9. The system according to claim 1, wherein, The data processing system is used to initiate an activity corresponding to the third action based on at least one of the processor utilization of the data processing system and the bandwidth metric of the computer network.

10. The system according to claim 1, wherein, The data processing system is used to initiate the activity corresponding to the third action and then initiate the activity corresponding to the second action.

11. The system according to claim 1, wherein, The content item is a first content item, and the data processing system is used to select a second content item for the first action and a third content item for the second action.

12. The system according to claim 11, wherein, The data processing system is used to select a second content item and a third content item after selecting the first content item, wherein the first content item is associated with the third action, the second content item is associated with the first action, and the third content item is associated with the second action.

13. The system according to claim 1, wherein, The sound wave corresponds to the content item.

14. The system according to claim 13, wherein, The data processing system is configured to obtain a response to the content item and route packet-based data messages to the natural language processor component of the service provider's computing device prior to the completion of at least one of the first and second actions.

15. The system according to claim 13, wherein, The data processing system is configured to obtain a response to the content item and route the packet-based data message to the natural language processor component of the service provider's computing device after at least one of the first and second actions is completed.

16. A method for processing operations dependent on sequences in a speech-activated, packet-based computer network environment, comprising: Receives input data packets including input audio signals detected by sensors of the client computing device; The input audio signal is analyzed to identify keywords; An action sequence is determined based on the keywords, the action sequence including a first action, a second action following the first action, and a third action following the second action; Bypass at least one of the first action and the second action to generate an output signal including the content item; as well as Data packets including the output signal are transmitted to the client computing device to drive the speaker of the client computing device to generate sound waves corresponding to the output signal.

17. The method of claim 16, comprising: The data packet, including the output signal, is prevented from being transmitted to the client computing device before either the expiration of the defined time period or before an authorization instruction is received from the client computing device.

18. The method of claim 16, comprising: Within a time interval triggered by either the first action or the second action, a packet-based data message for canceling the third action is received from the client computing device.

19. The method of claim 16, comprising: A data message including a response to the content item is received from the client computing device prior to the occurrence of at least one of the first action and the second action.

20. The method of claim 16, comprising: The third action is initiated based on either the completion time of the first action or the completion time of the second action.

Images