Selective sensor polling

Abstract

The present invention relates to selective sensor polling. A selective sensor polling system for a computer network environment based on voice-activated data packets is provided. A system may receive an audio signal detected by a microphone of a device. The system may parse the audio signal to identify trigger keywords and requests. The system may select a template for an action data structure having multiple fields. The system may determine to poll a first sensor for data in the first field. The system may determine to obtain data in a memory previously collected by the second sensor. The system may generate and transmit the action data structure using the data from the sensor and the memory, and transmit the action data structure to a third-party device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Patent Application No. 15 / 395,715, filed on December 30, 2016, and entitled “SELECTIVE SENSOR POLLING,” the entire contents of which are incorporated herein by reference for all purposes.

[0003] Description of the case

[0004] This application is a divisional application of Chinese invention patent application No. 201780001587.X, filed on August 31, 2017. Background Art

[0005] Excessive network transmission of network traffic data between computing devices (packet-based or otherwise) may prevent computing devices from properly processing the network traffic data, completing operations related to the network traffic data, or responding to the network traffic data in a timely manner. Excessive network transmission of network traffic data may also complicate data routing or degrade the quality of responses if the responding computing device is at or exceeds its processing capacity, which may result in inefficient bandwidth utilization. Control of network transmissions corresponding to content item objects may be complicated by the large number of content item objects that may initiate network transmissions of network traffic data between computing devices. Summary of the invention

[0006] The present disclosure generally relates to selectively polling sensors over a computer network. For example, a computing system may be able to access multiple sensors configured on multiple computing devices that can detect the same or similar types of information. However, the following steps may be resource intensive: requesting the same or similar types of information from multiple sensors configured on the same computing device or a group of computing devices that are very close to each other so that the detected information is similar. In addition, some sensors or computing devices configured with the sensors may consume more resources (e.g., energy, battery power, processor utilization, or bandwidth) than other sensors or computing devices. Therefore, the systems and methods of the present disclosure may selectively poll one or more sensors to obtain information in a manner that reduces resource consumption.

[0007] The systems and methods of the present disclosure generally relate to a data processing system that selectively polls sensors over a computer network. The data processing system may process voice-based input using a voice model trained on aggregated voice input and augmented with end-user voice input. The data processing system may identify a template for an action data structure based on a request and trigger keywords in the voice input. The data processing system may determine to fill one or more fields in the template to generate the action data structure. In order to fill the fields, the data processing system may interface with a sensor management component to selectively poll sensors of one or more computing devices associated with the end user who provided the voice input. The sensor management component may apply a policy or rule set to identify one or more available sensors and may provide values ​​for filling the fields of the template to generate the action data structure. The data processing system may receive the values ​​from the selected sensors, generate the action data structure, and transmit the data structure to a third-party provider device. The data processing system may then receive an indication from the third-party provider device that an operation corresponding to the action data structure has been initiated.

[0008] At least one aspect relates to a system for selectively polling sensors. The system may include a data processing system. The data processing system may execute a natural language processor ("NLP") component, a direct action application programming interface ("API"), and a sensor management component. The NLP component may receive a data packet including an input audio signal detected by a microphone of a client device via an interface of the data processing system. The NLP may parse the input audio signal to identify a request and a trigger keyword corresponding to the request. The direct action API may select a template for an action data structure responsive to the request based on the trigger keyword. The template may include a first field. The sensor management component may identify a plurality of available sensors configured to obtain information of the first field. The plurality of available sensors may include a first sensor and a second sensor. The sensor management component may determine a state of each of the plurality of sensors. The sensor management component may select the first sensor of the plurality of sensors based on the state. The sensor management component may poll the first sensor for data corresponding to the first field. The direct action API may populate the first field with the data received by the sensor management component in response to the polling of the first sensor. The direct action API may generate the action data structure based on the first field of the template. The direct action API may transmit the action data structure to a third-party provider device to cause the third-party provider device to invoke an operational session between the third-party provider device and the client device. The data processing system may receive from the third-party provider device an indication that the third-party provider device has established the operational session with the client device.

[0009] At least one aspect relates to a method for selectively polling a sensor. The method may include the data processing system receiving, via an interface, a data packet including an input audio signal detected by a microphone of a client device. The method may include the data processing system parsing the input audio signal to identify a request and a trigger keyword corresponding to the request. The method may include the data processing system selecting a template for an action data structure responsive to the request based on the trigger keyword. The template may include a first field. The method may include the data processing system identifying a plurality of available sensors configured to obtain information of the first field. The plurality of available sensors may include a first sensor and a second sensor. The method may include the data processing system determining a state of each of the plurality of sensors. The method may include selecting the first sensor of the plurality of sensors based on the state. The method may include the data processing system polling the first sensor for data corresponding to the first field. The method may include the data processing system populating the first field based on the data received by the sensor management component in response to the polling of the first sensor. The method may include the data processing system generating the action data structure based on the first field of the template via the direct action API. The method may include the data processing system transmitting the action data structure to a third-party provider device. The method may include the data processing system receiving, from the third party provider device, an indication that the third party provider device has established the operational session with the client device.

[0010] These and other aspects and embodiments are discussed in detail below. The above information and the following detailed description include illustrative examples of the various aspects and embodiments, and provide an overview or framework for understanding the nature and character of the claimed aspects and embodiments. The accompanying drawings provide illustration and further understanding of the various aspects and embodiments, and are incorporated into and constitute a part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings are not intended to be drawn to scale. The same reference numbers and names in the various drawings indicate the same elements. For the purpose of clarity, not every component may be labeled in every drawing. In the drawings:

[0012] Figure 1 is an illustration of a system for selectively polling sensors via a computer network.

[0013] Figure 2 is an illustration of the operation of a system to selectively poll sensors via a computer network.

[0014] Figure 3is an illustration of a method for selectively polling sensors over a computer network.

[0015] Figure 4 is a block diagram illustrating the general architecture of a computer system that may be employed to implement elements of the systems and methods described and illustrated herein. DETAILED DESCRIPTION

[0016] The following is a more detailed description of various concepts related to methods, apparatuses, and systems for selectively polling sensors over a computer network and implementations of these methods, apparatuses, and systems.The various concepts introduced above and discussed in more detail below may be implemented in any of numerous ways.

[0017] The present disclosure generally relates to selectively polling sensors over a computer network. For example, a computing system may be able to access multiple sensors configured on multiple computing devices that can detect the same or similar types of information. However, requesting the same or similar types of information from multiple sensors configured on the same computing device or a group of computing devices that are very close to each other so that the detected information is similar may be resource intensive. In addition, some sensors or computing devices configured with the sensors may consume more resources (e.g., energy, battery power, processor utilization, or bandwidth) than other sensors or computing devices. Therefore, the systems and methods of the present disclosure may selectively poll one or more sensors to obtain information in a manner that reduces resource consumption.

[0018] The systems and methods of the present disclosure generally relate to a data processing system that selectively polls sensors through a computer network. The data processing system can process voice-based input using a voice model trained on aggregated voice input and augmented with end-user voice input. The data processing system can identify a template for an action data structure based on a request and trigger keywords in the voice input. The data processing system can determine to fill one or more fields in the template to generate the action data structure. In order to fill the fields, the data processing system can interface with a sensor management component to selectively poll sensors of one or more computing devices associated with the end user who provided the voice input. The sensor management component can apply a policy or rule set to identify one or more available sensors and can provide values ​​for filling the fields of the template to generate the action data structure. The data processing system can receive the values ​​from the selected sensors, generate the action data structure, and transmit the data structure to a third-party provider device. The data processing system can then receive an indication from the third-party provider device that an operation corresponding to the action data structure has been initiated.

[0019] For example, the data processing system may generally improve sensor selection and optimize sensor configuration for collecting data for use by a voice-based system. When a device goes online from an offline state, the data processing system may perform data synchronization or batch upload. The data processing system may enable or disable sensors based on characteristics associated with the data, user, or task being collected. The data processing system may selectively enable sensors based on a strategy that improves data quality while reducing resource utilization. For example, the data processing system may determine that the user is carrying multiple devices with location sensors. The data processing system may determine which devices have the largest power remaining and request location only from that device. In another example, the data processing system may determine that a first device is already obtaining location information (e.g., via a navigation application) and piggybacks off the location data.

[0020] Figure 1 An example system 100 for selectively polling sensors via a computer network is illustrated. System 100 may include a content selection infrastructure. System 100 may include a data processing system 102. Data processing system 102 may communicate with one or more of content provider computing device 106, service provider computing device 108, or client computing device 104 via network 105. Network 105 may include a computer network such as the Internet, a local area network, a wide area network, a metropolitan area network, or other regional network, an intranet, a satellite network, and other communication networks such as voice or data mobile phone networks. Network 105 may be used to access information resources, such as web pages, web sites, domain names, or uniform resource locators that may be presented, output, rendered, or displayed on at least one computing device 104 (such as a laptop, desktop, tablet, personal digital assistant, smart phone, portable computer, or speaker). For example, a user of computing device 104 may access information or data provided by service provider 108 or content provider 106 via network 105.

[0021] Network 105 may include or constitute a display network, for example, a subset of information resources available on the Internet that are associated with content placement or a search engine results system or that are eligible to include third-party content items as part of a content item placement activity. Network 105 may be used by data processing system 102 to access information resources, such as web pages, web sites, domain names, or uniform resource locators that may be presented, output, rendered, or displayed by client computing device 104. For example, a user of client computing device 104 may access information or data provided by content provider computing device 106 or service provider computing device 108 via network 105.

[0022] The network 105 may be any type or form of network and may include any of the following: a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, an SDH (Synchronous Digital Hierarchy) network, a wireless network, and a wireline network. The network 105 may include a wireless link, such as an infrared channel or a satellite band. The topology of the network 105 may include a bus, a star, or a ring network topology. The network may include a mobile phone network using any one or more protocols for communicating between mobile devices, including Advanced Mobile Phone Protocol ("AMPS"), Time Division Multiple Access ("TDMA"), Code Division Multiple Access ("CDMA"), Global System for Mobile Communications ("GSM"), General Packet Radio Service ("GPRS"), or Universal Mobile Telecommunications System ("UMTS"). Different types of data may be transmitted via different protocols, or the same type of data may be transmitted via different protocols.

[0023] System 100 may include at least one data processing system 102. Data processing system 102 may include at least one logical device, such as a computing device having a processor to communicate, for example, with computing device 104, content provider device 106 (content provider 106), or service provider device 108 (or service provider 108) via network 105. Data processing system 102 may include at least one computing resource, server, processor, or memory. For example, data processing system 102 may include multiple computing resources or servers located in at least one data center. Data processing system 102 may include multiple logically grouped servers and facilitate distributed computing techniques. Logical groups of servers may be referred to as data centers, server farms, or machine farms. Servers may also be geographically dispersed. A data center or machine farm may be managed as a single entity, or a machine farm may include multiple machine farms. The servers within each machine farm may be heterogeneous - one or more of these servers or machines may operate according to one or more types of operating system platforms.

[0024] The servers in the machine farm may be stored in a high-density rack system along with associated storage systems and located in an enterprise data center. Consolidating servers in this manner may improve system manageability, data security, physical security of the system, and system performance by locating the servers and high-performance storage systems on a localized high-performance network, for example. Centralization of all or some of the data processing system 102 components (including servers and storage systems) and coupling them with advanced system management tools allows for more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage.

[0025] System 100 may include, access, or otherwise interact with at least one service provider device 108. Service provider device 108 may include at least one logical device, such as a computing device having a processor to communicate, for example, with computing device 104, data processing system 102, or content provider 106 via network 105. Service provider device 108 may include at least one computing resource, server, processor, or memory. For example, service provider device 108 may include multiple computing resources or servers located in at least one data center. Service provider device 108 may include one or more components or functionality of data processing system 102.

[0026] The content provider computing device 106 can provide audio-based content items for display as audio output content items by the client computing device 104. The content items can include an offer of goods or services, such as a voice-based message stating: "Do you want me to order a taxi for you?" For example, the content provider computing device 155 can include a memory for storing a series of audio content items that can be provided in response to voice-based queries. The content provider computing device 106 can also provide audio-based content items (or other content items) to the data processing system 102, where they can be stored in the data repository 124. The data processing system 102 can select audio content items and provide these audio content items (or instruct the content provider computing device 104 to provide these audio content items) to the client computing device 104. The audio-based content items can be exclusively audio or can be combined with text, image or video data.

[0027] The service provider device 108 may include, interface with, or otherwise communicate with at least one service provider natural language processor component 142 and a service provider interface 144. The service provider computing device 108 may include at least one service provider natural language processor (NLP) component 142 and at least one service provider interface 144. The service provider NLP component 142 (or other component such as a direct action API of the service provider computing device 108) may engage with the client computing device 104 (via the data processing system 102 or bypassing the data processing system 102) to create a back-and-forth real-time voice or audio-based dialogue (e.g., a session) between the client computing device 104 and the service provider computing device 108. The service provider NLP 142 may include one or more functions or features as the NLP component 112 of the data processing system 102. For example, the service provider interface 144 may receive and provide data messages to the direct action API 116 of the data processing system 102. The service provider computing device 108 and the content provider computing device 106 may be associated with the same entity. For example, the content provider computing device 106 may create, store, or make available content items for a car sharing service, and the service provider computing device 108 may establish a session with the client computing device 106 to arrange for the delivery of a taxi or car for the car sharing service to pick up an end user of the client computing device 104. The data processing system 102 may also establish a session with the client computing device, including or bypassing the service provider computing device 104, via the direct action API 116, the NLP component 112, or other components, to arrange for the delivery of a taxi or car for the car sharing service, for example.

[0028] The computing device 104 may include at least one sensor 134, a converter 136, an audio driver 138, or a preprocessor 140, interface with at least one sensor 134, a converter 136, an audio driver 138, or a preprocessor 140, or otherwise communicate with at least one sensor 134, a converter 136, an audio driver 138, or a preprocessor 140. The sensor 134 may include, for example, an ambient light sensor, a proximity sensor, a temperature sensor, an accelerometer, a gyroscope, a motion detector, a GPS sensor, a position sensor, a microphone, or a touch sensor. The converter 136 may include a speaker or a microphone. The audio driver 138 may provide a software interface to the hardware converter 136. The audio driver may execute an audio file or other instructions provided by the data processing system 102 to control the converter 136 to generate corresponding sound waves or sound waves. The preprocessor 140 may be configured to detect keywords and perform actions based on the keywords. The preprocessor 140 may filter out one or more terms or modify terms before transmitting the terms to the data processing system 102 for further processing. Preprocessor 140 may convert analog audio signals detected by microphone into digital audio signals and transmit one or more data packets carrying the digital audio signals to data processing system 102 via network 105. In some cases, preprocessor 140 may transmit data packets carrying some or all of the input audio signals in response to detecting instructions for performing such transmission. Instructions may include, for example, trigger keywords or other keywords or approvals for transmitting data packets including the input audio signals to data processing system 102. Computing device 104 may or may not include a display; for example, a computing device may include a limited type of user interface, such as a microphone and a speaker. In some cases, the main user interface of computing device 104 may be a microphone and a speaker.

[0029] The client computing device 104 may be associated with an end user who enters a voice query as audio input into the client computing device 104 (via the sensor 134) and receives audio output from the transducer 136 (e.g., a speaker) in the form of computer-generated speech that may be provided from the data processing system 102 (or the content provider computing device 106 or the service provider computing device 108) to the client computing device 104. The computer-generated speech may include a recording of speech from a real person or generated by a computer.

[0030] The data repository 124 may include one or more local or distributed databases, and may include a database management system. The data repository 124 may include a computer data storage unit or memory and may store one or more parameters 126, one or more policies 128, content data 130, or templates 132 in addition to data. The parameters 126, policies 128, and templates 132 may include information such as rules for voice-based conversations between the client computing device 104 and the data processing system 102 (or service provider computing device 108). The content data 130 may include content items of audio output or related metadata, and input audio messages that may be part of one or more communication sessions with the client computing device 104. A communication session may include or be referred to as an operational session. In some cases, an operational session may refer to a communication session in which one or more operations are performed by a third-party provider device 108, a client computing device, a content provider device 106, a data processing system 102, or other components or entities.

[0031] The data processing system 102 may include a content delivery system having at least one computing resource or server. The data processing system 102 may include at least one interface 110, interface with at least one interface 110, or otherwise communicate with at least one interface 110. The data processing system 102 may include at least one natural language processor component 112, interface with at least one natural language processor component 112, or otherwise communicate with at least one natural language processor component 112. The data processing system 102 may include at least one direct action application programming interface (“API”) 116, interface with at least one direct action application programming interface (“API”) 116, or otherwise communicate with at least one direct action application programming interface (“API”) 116. The data processing system 102 may include at least one session handler 114, interface with at least one session handler 114, or otherwise communicate with at least one session handler 114. The data processing system 102 may include at least one content selector component 118, interface with at least one content selector component 118, or otherwise communicate with at least one content selector component 118. The data processing system 102 may include, interface with, or otherwise communicate with at least one sensor management component 120. The data processing system 102 may include, interface with, or otherwise communicate with at least one audio signal generator 122. The data processing system 102 may include, interface with, or otherwise communicate with at least one audio signal generator 122. The data processing system 102 may include, interface with, or otherwise communicate with at least one data repository 124. The at least one data repository 124 may include or store parameters 126, policies 128, content data 130, or templates 132 in one or more data structures or databases. The parameters 126 may include, for example, thresholds, distances, time intervals, durations, scores, or weights. The content data 130 may include, for example, content activity information, content groups, content selection criteria, content item objects, or other information provided by the content provider 106 or obtained or determined by the data processing system to facilitate content selection. The content data 130 may include, for example, historical performance of content activity.

[0032] The interface 110, the natural language processor component 112, the conversation handler 114, the direct action API 116, the content selector component 118, the sensor management component 120, or the audio signal generator component 122 may each include at least one processing unit or other logic device, such as a programmable logic array engine or a module configured to communicate with a database repository or database 124. The interface 110, the natural language processor component 112, the conversation handler 114, the direct action API 116, the content selector component 118, the sensor management component 120, the audio signal generator component 122, and the data repository 124 may be separate components, a single component, or part of the data processing system 102. The system 100 and its components, such as the data processing system 102, may include hardware elements, such as one or more processors, logic devices, or circuits.

[0033] Data processing system 102 may obtain anonymous computer network activity information associated with multiple computing devices 104. A user of computing device 104 may affirmatively authorize data processing system 102 to obtain network activity information corresponding to the user's computing device 104. For example, data processing system 102 may prompt the user of computing device 104 to consent to obtaining one or more types of network activity information. The identity of the user of computing device 104 may remain anonymous and computing device 104 may be associated with a unique identifier (e.g., a unique identifier of the user or computing device provided by the user of the data processing system or computer). The data processing system may associate each observation with a corresponding unique identifier.

[0034] The content provider 106 can establish an electronic content activity. The electronic content activity can be stored in the data repository 124 as content data 130. The electronic content activity can refer to one or more content groups corresponding to a common theme. The content activity can include a hierarchical data structure, which includes content groups, content item data objects, and content selection criteria. In order to create a content activity, the content provider 106 can specify a value for the activity level parameter of the content activity. The activity level parameter can include, for example, the name of the activity, the preferred content network for delivering the content item object, the value of the resource to be used for the content activity, the start and end date of the content activity, the duration of the content activity, the timetable for the delivery of the content item object, the language, the geographical location, and the type of computing device on which the content item object is provided. In some cases, an impression can refer to when the content item object is pre-fetched from its source (e.g., the data processing system 102 or the content provider 106), and is countable. In some cases, due to the possibility of click fraud, robot activities can be filtered and excluded as impressions. Thus, in some cases, an impression may refer to a measurement of a response from a web server to a page request from a browser, filtered from robot activity and error codes, and recorded at a point as close as possible to the opportunity to render a content item object for display on computing device 104. In some cases, an impression may refer to a visible or audible impression; for example, a content item object is at least partially (e.g., 20%, 30%, 30%, 40%, 50%, 60%, 70%, or more) visible on a display device of client computing device 104, or audible via speaker 136 of computing device 104. A click or selection may refer to a user interaction with a content item object, such as a voice response to an audible impression, a mouse click, a touch interaction, a gesture, a shake, an audio interaction, or a keyboard click. A conversion may refer to a user taking a desired action with respect to a content item object; for example, purchasing a product or service, completing a survey, visiting a physical store corresponding to a content item, or completing an electronic transaction.

[0035] The content provider 106 may further establish one or more content groups for a content campaign. A content group includes one or more content item objects and corresponding content selection criteria, such as keywords, words, terms, phrases, geographic locations, types of computing devices, time of day, interests, themes, or verticals. Content groups under the same content campaign may share the same campaign level parameters, but have customized specifications for specific content group level parameters, such as keywords, negative keywords (e.g., blocking delivery of content items if negative keywords are present on primary content), bids on keywords, or parameters associated with bids or content campaigns.

[0036] In order to create a new content group, the content provider can provide values ​​for the content group level parameters of the content group. The content group level parameters include, for example, the content group name or content group theme and bids for different content delivery opportunities (e.g., automatic delivery or managed delivery) or results (e.g., clicks, impressions, or conversions). The content group name or content group theme can be one or more terms that the content provider 106 can use to capture the topic or theme that the content item object of the content group will be selected for display. For example, a car dealer can create a different content group for each model of the vehicle it transports, and can also create a different content group for each brand of the vehicle it transports. Examples of content group themes that can be used by car dealers include, for example, "Make A Sports Car", "Make B Sports Car", "Make C Sedan", "Make C Truck", "Make C Hybrid" or "Make D Hybrid". For example, an example content campaign theme can be "Hybrid" and include content groups for both "Make C Hybrid" and "Make D Hybrid".

[0037] The content provider 106 may provide one or more keywords and content item objects to each content group. Keywords may include terms related to products or services associated with or identified by the content item object. Keywords may include one or more terms or phrases. For example, a car dealer may include "sports car", "V-6 engine", "four-wheel drive", "fuel efficiency" as keywords for a content group or content campaign. In some cases, the content provider may specify negative keywords to avoid, prevent, block or disable content delivery on specific terms or keywords. The content provider may specify the type of match used to select the content item object, such as exact match, phrase match or broad match.

[0038] The content provider 106 may provide one or more keywords to be used by the data processing system 102 to select content item objects provided by the content provider 106. The content provider 106 may identify one or more keywords to bid on and further provide bid amounts for various keywords. The content provider 106 may provide additional content selection criteria to be used by the data processing system 102 to select content item objects. Multiple content providers 106 may bid on the same or different keywords, and the data processing system 102 may run a content selection process or an advertising auction in response to receiving an indication of the keywords of the electronic message.

[0039] The content provider 106 may provide one or more content item objects for selection by the data processing system 102. When a content placement opportunity that matches the resource allocation, content schedule, maximum bid, keywords, and other selection criteria specified for the content group becomes available, the data processing system 102 (e.g., via the content selector component 118) may select the content item object. Different types of content item objects may be included in the content group, such as voice content items, audio content items, text content items, image content items, video content items, multimedia content items, or content item links. When selecting a content item, the data processing system 102 may transmit the content item object for rendering on the computing device 104 or a display device of the computing device 104. Rendering may include displaying the content item on a display device or playing the content item via a speaker of the computing device 104. The data processing system 102 may provide instructions for rendering the content item object to the computing device 104. The data processing system 102 may instruct the computing device 104 or the audio driver 138 of the computing device 104 to generate an audio signal or sound wave.

[0040] The data processing system 102 may include an interface component 110 that is designed, configured, constructed, or operable to receive and transmit information using, for example, data packets. The interface 110 may receive and transmit information using one or more protocols, such as a network protocol. The interface 110 may include a hardware interface, a software interface, a wired interface, or a wireless interface. The interface 110 may facilitate converting or formatting data from one format to another. For example, the interface 110 may include an application programming interface that includes definitions for communicating between various components, such as software components.

[0041] The data processing system 102 may include an application, script, or program installed at the client computing device 104, such as an app for transmitting an input audio signal to an interface 110 of the data processing system 102 and driving a component of the client computing device to render an output audio signal. The data processing system 102 may receive a data packet or other signal including or identifying an audio input signal. For example, the data processing system 102 may execute or run an NLP component 112 to receive or obtain an audio signal and parse the audio signal. For example, the NLP component 112 may provide interaction between humans and computers. The NLP component 112 may be configured with technology for understanding natural language and allowing the data processing system 102 to derive meaning from human or natural language input. The NLP component 112 may include or be configured with machine learning-based technology, such as statistical machine learning. The NLP component 112 may utilize a decision tree, a statistical model, or a probability model to parse the input audio signal. The NLP component 112 may perform, for example, functions such as named entity recognition (e.g., given a stream of text, determining which items in the text map to appropriate names (such as people or places) and what type each such name is, such as person, place, or organization), natural language generation (e.g., converting information from a computer database or semantic intent into understandable human language), natural language understanding (e.g., converting text into a more formal representation, such as a first-order logic structure that a computer module can manipulate), machine translation (e.g., automatically translating text from one human language to another), lexical segmentation (e.g., breaking words into individual morphemes and identifying categories of morphemes, which may be challenging based on the morphological or structural complexity of the words of the language being considered), question answering (e.g., determining answers to human language questions, which may be specific or open-ended), semantic processing (e.g., processing that may occur after a word is recognized and its meaning is encoded in order to associate the recognized word with other words of similar meaning).

[0042] The NLP component 112 converts the audio input signal into recognized text by comparing the input signal against a stored set of representative audio waveforms (e.g., in a data repository 124) and selecting the closest match. The audio waveform set may be stored in the data repository 124 or other database accessible to the data processing system 102. Representative waveforms are generated across a large set of users and can then be augmented with speech samples from the users. After the audio signal is converted into recognized text, the NLP component 112 matches the text to words associated with actions that the data processing system 102 can service, for example, via training across users or by manual designation.

[0043] The audio input signal may be detected by a sensor 134 or transducer 136 (e.g., a microphone) of the client computing device 104. Via the transducer 136, the audio driver 138, or other components, the client computing device 104 may provide the audio input signal to the data processing system 102 (e.g., via the network 105), where it may be received (e.g., by the interface 110) and provided to the NLP component 112 or stored in the data repository 124.

[0044] The NLP component 112 may obtain an input audio signal. Based on the input audio signal, the NLP component 112 may identify at least one request or at least one trigger keyword corresponding to the request. The request may indicate the intention or theme of the input audio signal. The trigger keyword may indicate the type of action that is likely to be taken. For example, the NLP component 112 may parse the input audio signal to identify at least one request to leave home in the evening to attend dinner and a movie. The trigger keyword may include at least one word, phrase, root or partial word or derivative indicating an action to be taken. For example, the trigger keyword "go" or "to go to" from the input audio signal may indicate the need for transportation. In this example, the input audio signal (or the identified request) does not directly express the intention for transportation, but the trigger keyword indicates that transportation is an auxiliary action to at least one other action indicated by the request.

[0045] The NLP component 112 may parse the input audio signal to identify, determine, retrieve or otherwise obtain the request and trigger keywords. For example, the NLP component 112 may apply semantic processing techniques to the input audio signal to identify trigger keywords or requests. The NLP component 112 may apply semantic processing techniques to the input audio signal to identify trigger phrases including one or more trigger keywords (such as a first trigger keyword and a second trigger keyword). For example, the input audio signal may include the sentence "I need someone to do my laundry and my dry cleaning". The NLP component 112 may apply semantic processing techniques or other natural language processing techniques to the data grouping including the sentence to identify the trigger phrases "do my laundry" and "do my dry cleaning". The NLP component 112 may also identify multiple trigger keywords, such as laundry and dry cleaning. For example, the NLP component 112 may determine that the trigger phrase includes a trigger keyword and a second trigger keyword.

[0046] The NLP component 112 may filter the input audio signal to identify trigger keywords. For example, a data packet carrying the input audio signal may include "It would be great if I could get someone that could help me go to the airport", in which case the NLP component 112 may filter out one or more terms as follows: "it", "would", "be", "great", "if", "I", "could", "get", "someone", "that", "could", or "help". By filtering out these terms, the NLP component 112 can more accurately and reliably identify trigger keywords, such as "go to the airport", and determine that this is a request for a taxi or ride-sharing service.

[0047] In some cases, the NLP component may determine that the data packet carrying the input audio signal includes one or more requests. For example, the input audio signal may include the sentence "I need someone to do my laundry and my drycleaning". The NLP component 112 may determine that this is a request for laundry service and dry cleaning service. The NLP component 112 may determine that this is a single request to a service provider that can provide both laundry service and dry cleaning service. The NLP component 112 may determine that these are two requests; a first request to a service provider that performs laundry service, and a second request to a service provider that provides dry cleaning service. In some cases, the NLP component 112 may combine multiple determined requests into a single request and transmit the single request to the service provider device 108. In some cases, the NLP component 112 may transmit separate requests to the corresponding service provider devices 108, or transmit the two requests to the same service provider device 108 separately.

[0048] The data processing system 102 may include a direct action API 116 designed and constructed to generate an action data structure in response to a request based on a trigger keyword. The processor of the data processing system 102 may call the direct action API 116 to execute a script that generates a data structure to the service provider device 108 to request or order a service or product, such as a car from a car sharing service. The direct action API 116 may obtain data from the data repository 124 and, with the consent of the end user, receive data from the client computing device 104 to determine the location, time, user account, logic or other information to allow the service provider device 108 to perform an operation, such as booking a vehicle from a car sharing service. Using the direct action API 116, the data processing system 102 may also communicate with the service provider device 108 to complete the conversion by making a car sharing pickup reservation in this example.

[0049] The direct action API 116 may perform a specified action to satisfy the end user's intent as determined by the data processing system 102. Depending on the action specified in its input, the direct action API 116 may execute code or a dialog script that identifies the parameters needed to fulfill the user's request. Such code may, for example, look up additional information in the data repository 124, such as the name of a home automation service, or it may provide an audio output for rendering at the client computing device 104 to ask the end user a question, such as the intended destination of the requested taxi. The direct action API 116 may determine the necessary parameters and may package the information into an action data structure, which may then be sent to another component, such as the content selector component 118, or to the service provider computing device 108 to be fulfilled.

[0050] Direct action API 116 can receive instructions or commands from NLP component 112 or other components of data processing system 102 to generate or construct action data structure. Direct action API 116 can determine the type of action in order to select a template from template repository 132 stored in data repository 124. The type of action can include, for example, service, product, reservation or ticket. The type of action can also include the type of service or product. For example, the type of service can include car sharing service, food delivery service, laundry service, maid service, repair service or housekeeping service. The type of product can include, for example, clothes, shoes, toys, electronic equipment, computers, books or jewelry. The type of reservation can include, for example, dinner reservation or hair salon appointment. The type of ticket can include, for example, movie ticket, stadium ticket or airline ticket. In some cases, the type of service, product, reservation or ticket can be classified based on price, location, delivery type, availability or other attributes.

[0051] The direct action API 116, upon identifying the type of request, may access the corresponding template from the template repository 132. The template may include fields in a structured data set that may be populated by the direct action API 116 to further perform the operation requested by the service provider device 108 (such as an operation of dispatching a taxi to pick up the end user at the ride location and transporting the end user to the destination location). The direct action API 116 may perform a search in the template repository 132 to select a template that matches the trigger keyword and one or more characteristics of the request. For example, if the request corresponds to a request for a car or a ride to a destination, the data processing system 102 may select a car sharing service template. The car sharing service template may include one or more of the following fields: device identifier, ride location, destination location, number of passengers, or type of service. The direct action API 116 may populate the fields with values. To populate the fields with values, the direct action API 116 may ping, poll, or otherwise obtain information from one or more sensors 134 of the computing device 104 or a user interface of the device 104.

[0052] Polling may refer to actively sampling information from an external device, such as sensor 134, by data processing system 102 or by computing device 104 in response to an instruction from data processing system 102. Polling may be a synchronous activity. During polling, data processing system 102 may wait for sensor 134 to check its readiness, status, detect environmental conditions, or perform any other function or activity that sensor 134 is configured to perform (e.g., collect and record temperature readings; detect ambient light levels; determine location; determine pressure; determine altitude; determine speed of motion; or determine direction of motion). In some cases, polling may refer to requesting data from computing device 104 that is collected, measured, detected, or otherwise determined at least in part by using one or more sensors 134.

[0053] For example, the direct action API 116 may use a location sensor (such as a GPS sensor) to detect the source location. The direct action API 116 may obtain further information by submitting a survey, prompt, or query to an end user of the computing device 104. The direct action API may submit the survey, prompt, or query via the interface 110 of the data processing system 102 and the user interface (e.g., an audio interface, a voice-based user interface, a display, or a touch screen) of the computing device 104. Thus, the direct action API 116 may select a template for an action data structure based on a trigger keyword or request, populate one or more fields in the template with information detected by one or more sensors 134 or obtained via a user interface, and generate, create, or otherwise construct an action data structure to facilitate the execution of an operation by the service provider device 108.

[0054] The data processing system 102 may select a template 132 based on a template data structure based on various factors, including, for example, one or more of a trigger keyword, a request, a third-party provider device 108, a type of third-party provider device 108, a category into which the third-party provider device 108 falls (e.g., taxi service, laundry service, flower service, or food delivery), location, or other sensor information.

[0055] To select a template based on a trigger keyword, the data processing system 102 (e.g., via the direct action API 116) may use the trigger keyword to perform a lookup or other query operation on the template database 132 to identify a template data structure that maps to or otherwise corresponds to the trigger keyword. For example, each template in the template database 132 may be associated with one or more trigger keywords to indicate that the template is configured to generate an action data structure in response to the trigger keyword that the third-party provider device 108 may process to establish a communication session.

[0056] In some cases, the data processing system 102 may identify the third-party provider device 108 based on the trigger keyword. To identify the third-party provider device 108 based on the trigger keyword, the data processing system 102 may perform a lookup in the data repository 124 to identify the third-party provider device 108 mapped to the trigger keyword. For example, if the trigger keyword includes "ride" or "to got to", the data processing system 102 (e.g., via the direct action API 116) may identify the third-party provider device 108 as corresponding to Taxi Service Company A. The data processing system 102 may use the identified third-party provider device 108 to select a template from the template database 132. For example, the template database 132 may include a mapping or correlation between the third-party provider device 108 or entity to a template configured to generate an action data structure in response to the trigger keyword that the third-party provider device 108 can process to establish a communication session. In some cases, the template may be customized for the third-party provider device 108 or for a category of the third-party provider device 108. The data processing system 102 may generate an action data structure based on the template for the third-party provider 108.

[0057] To construct or generate an action data structure, the data processing system 102 may identify one or more fields in the selected template to be populated with values. Fields may be populated with numeric values, strings, Unicode values, Boolean logic, binary values, hexadecimal values, identifiers, location coordinates, geographic regions, timestamps, or other values. Fields or the data structure itself may be encrypted or masked to maintain data security.

[0058] When determining the fields in the template, data processing system 102 can identify values ​​of the fields to populate the fields of the template to create the action data structure. Data processing system 102 can obtain, retrieve, determine or otherwise identify values ​​of the fields by performing a lookup or other query operation on data repository 124.

[0059] In some cases, the data processing system 102 may determine that the information or value for the field does not exist in the data repository 124. The data processing system 102 may determine that the information or value stored in the data repository 124 is outdated, stale, or otherwise unsuitable for the purpose of constructing an action data structure in response to the trigger keyword and request identified by the NLP component 112 (e.g., the location of the client computing device 104 may be an old location instead of the current location; an account may be expired; a destination restaurant may have moved to a new location; physical activity information; or transportation methods).

[0060] If data processing system 102 determines that it does not currently have access to values ​​or information for fields of the template in the memory of data processing system 102, data processing system 102 may obtain these values ​​or information. Data processing system 102 may obtain or obtain information by querying or polling one or more available sensors of client computing device 104, prompting an end user of client computing device 104 for information, or accessing online web-based resources using HTTP protocol. For example, data processing system 102 may determine that it does not have the current location of client computing device 104, which may be a required field of the template. Data processing system 102 may query client computing device 104 for location information. Data processing system 102 may request client computing device 104 to provide location information using one or more location sensors 134 (such as global positioning system sensors), WIFI triangulation, cell tower triangulation, Bluetooth beacons, IP addresses, or other location sensing technologies.

[0061] In some cases, direct action API 116 may request sensor information from sensor management component 120. Data processing system 102 may include sensor management component 120. Sensor management component 120 may execute on data processing system 102 separate from or independent of client computing device 104. In some cases, sensor management component 120 may include one or more agents, scripts, executable programs configured on computing device 104 to interface with sensor management component 120 executing on data processing system 102.

[0062] The sensor management component 120 may include hardware or software for measuring characteristics of a communication session. For example, the direct action API 116 may select a template of an action data structure corresponding to a request or in response to a request based on a trigger keyword. The template may include one or more fields to be filled, such as a first field and a second field. In order to fill these fields, the direct action API 116 may query the sensor management component 120, request from the sensor management component 120, call the sensor management component 120, or otherwise interface with the sensor management component 120. The sensor management component 120 may obtain information of the field by reducing resource utilization by the sensor 134 or the computing device 104 associated with the sensor 134. For example, the sensor management component 120 may identify a plurality of available sensors 134 configured to obtain information of the first field. The plurality of available sensors may include a first sensor and a second sensor. The sensor management component 120 may determine a state of each of the plurality of sensors. The sensor management component 120 may select a first sensor of the plurality of sensors based on the state. The sensor management component 120 may poll the first sensor for data corresponding to the first field. The direct action API 116 may populate the first field with data received by the sensor management component in response to polling the first sensor.

[0063] The sensor management component 120 may identify a plurality of available sensors 134. The sensor management component 120 may store in the data repository 124 a list of sensors 134 that may be utilized by the end user account associated with the computing device 104. The sensor 134 may have an available status. Available may refer to the sensor being online, active, in standby mode, or in a low power mode. A sensor may be available if it passes a diagnostic test or process. In some cases, if the computing device 104 may be instructed to bring the sensor 134 online, it may be available even if the sensor 134 is offline. If the sensor 134 is offline, does not respond to a check in a timely manner, is malfunctioning, provides an erroneous or inconsistent value, or a diagnostic test or process fails, the status of the sensor 134 may be unavailable.

[0064] In some cases, the client computing device 104 may push sensor state information to the sensor management component 120 when establishing a communication session. In some cases, the sensor management component 120 may send a request for a list of available sensors 134 to the client computing device 104 or one or more client computing devices 104 associated with the account identifier. The sensor management component 120 may identify a set of one or more computing devices 104 based on the proximity of the computing devices to each other or based on network activity associated with the one or more computing devices 104 (e.g., an end user may be actively using one or more computing devices 104 to interact with the data processing system 102 or its components). In some cases, the data processing system 102 may poll all previously known sensors 134 used by the sensor management component based on a time interval.

[0065] Thus, the data processing system 102 may identify a plurality of sensors that are capable of providing and can be used to provide data to fill the first field of the template. The data processing system may then select one (or a subset) of the sensors from which data is to be received. The data processing system 102 may determine to poll sensors that are not coupled to the computing device 104 that provided the input audio signal. For example, the end user may use the first computing device 104 to provide the input audio signal. The first computing device 104 may include a first sensor 134 coupled to the first computing device 104. The data processing system may determine that there is a second computing device 104 that is also associated with the end user account (e.g., both the first and second computing devices 104 have successfully performed a handshake process with the data processing system 102). The second computing device 104 may be close to the first computing device 104, or the data processing system 102 may otherwise determine that the second computing device 104 may also provide data of sufficient quality to fill the first field of the template (e.g., the location information provided by the two sensors is within a tolerance level (such as 25 meters, 50 meters, 75 meters, or 100 meters)). Even though first computing device 104 invoked, initiated, or established communication with data processing system 102, data processing system 102 may determine to use second sensor 134 coupled to second computing device 104 instead of first sensor 134 coupled to first computing device. Data processing system 102 may use a resource reduction policy to determine to use second sensor 134 coupled to the second computing device instead of first sensor 134 coupled to first computing device 104 that provided input audio signals to NLP component 110. Data processing system 102 may determine to use second computing device 104 because it may have more battery remaining, have greater resource availability, or be configured to provide higher quality or more accurate data, which may result in fewer subsequent requests for sensor information.

[0066] In some cases, the sensor management component 120 can reduce resource utilization by having the first sensor detect a first environmental condition (e.g., position, speed, temperature, ambient light, ambient sound, etc.) while retrieving from memory a second environmental condition previously detected by a second sensor. Rather than instructing the second sensor to be online or activated to detect an environmental condition, the sensor management component 120 can retrieve the previously detected value from memory to reduce resource utilization. For example, the sensor management component can poll the first sensor for data corresponding to a first field while obtaining data corresponding to a second field from a memory of the computing device 104. The second data can be stored in the memory of the computing device 104 before the data processing system 102 requests the second data. In response to a request for data from the second sensor received from the direct action API 116, the sensor management component 120 may not poll the second sensor 134. The sensor management component 120 can determine to poll the first sensor for data corresponding to the first field, but not to poll the second sensor for data corresponding to the second field. The sensor management component 120 can use a policy, logic, or rule set to determine whether to poll one or more sensors. For example, the policy, logic, or rule set may include or be based on a conditional rule, an if / then condition, a trigger event, a tolerance, a threshold, a time interval, a location, a geo-fence, or a type of activity. For example, the sensor management component 120 may determine to poll a first sensor for location information because the last location data received by the data processing system 102 may have expired based on a time interval (e.g., 10 seconds, 5 seconds, 20 seconds, 30 seconds, 1 minute, or longer). The data processing system 102 may obtain data from a memory of a second sensor because the second sensor may be a temperature sensor and the data processing system 102 may determine that a timestamp when the last temperature measurement was detected and stored in the memory may satisfy the time interval (e.g., 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes, or longer).

[0067] The sensor management component 120 may adjust the configuration of the sensor to collect data based on the type of data. For example, the sensor management component 120 may adjust the sampling rate, sampling interval, or sampling duration of the sensor. The sensor management component 120 may increase or decrease the sampling rate. The sampling rate may refer to the number of samples taken during a measurement time interval. The sampling rate may include, for example, 0.005 Hz, 0.01 Hz, 0.015 Hz, 0.02 Hz, 0.05 Hz, 0.1 Hz, 0.2 Hz, 0.5 Hz, 0.7 Hz, 1 Hz, 2 Hz, 3 Hz, 5 Hz, 10 Hz, or some other sampling rate that provides data to create an action data structure while optimizing sensor resource utilization (e.g., data within tolerance levels to perform operations without excessive data collection).

[0068] The sensor management component 120 can adjust the sampling interval. The sampling interval can refer to the time period when the sensor is turned on or actively collecting sensor data. The sampling duration can refer to how long the sensor is turned on to actively collect data at the sampling rate. For example, the sensor management component 120 can instruct or command the sensor to turn on every 5 minutes (e.g., the sampling interval) to collect sensor data at 1 Hz (e.g., the sampling rate) for a duration of 30 seconds (e.g., the sampling duration). The policies stored in the policy data structure 128 can include different values ​​for the sampling rate, sampling interval, or duration based on the type of sensor, the type of activity, or other characteristics.

[0069] In some cases, the sensor management component 120 may disable one or more sensors. The sensor management component 120 may disable a sensor based on or in response to characteristics of data collected by the sensor. The sensor management component 120 may temporarily or permanently disable a sensor. Disabling a sensor 134 may prevent the sensor from collecting or detecting sensor data or making other measurements. In some cases, disabling a sensor may refer to disabling or turning off electronic hardware or an API that controls or interfaces with the sensor. For example, the computing device 102 may stop polling the sensor for data and collecting and storing data in response to an instruction from the sensor management component 120 to disable the sensor 134.

[0070] The sensor management component 120 may apply a resource utilization reduction strategy to the characteristics of the data collected by the first sensor to disable the first sensor. The characteristics of the data may be data including or referring to the quality of the data, the amount of data, data indicating the performance of the sensor, or data indicating the availability of computing resources or battery power to continue collecting data. For example, the data may indicate that the sensor is malfunctioning, uncalibrated, or otherwise collecting erroneous data. In order to prevent the collection of further erroneous data and to prevent the waste of resources (e.g., battery power, processing power, or bandwidth), the sensor management component 120 may disable the sensor 134 for a certain period of time. The sensor management component 120 may poll the sensor after the period of time and evaluate the data at that time. If the data is again unsatisfactory (e.g., inconsistent with data collected by other sensors, historical data, the type of data, the format of the data), the sensor management component 120 may again disable the sensor for a certain time interval, which may be longer than the first time interval or permanent until the sensor is repaired.

[0071] In some cases, data collected from a first sensor can be analyzed by sensor management component 120 to disable a second sensor of the same computing device or another computing device. For example, a battery sensor can indicate that the battery is low (e.g., 20% charge remaining), and then disable one or more sensors of device 104. In another example, a temperature sensor can indicate that the device is overheated, in response to which sensor management component 120 can disable a location sensor or other sensors.

[0072] In some cases, the data processing system may identify multiple available sensors 134 that are configured to obtain the same or similar types of information (e.g., location information, activity information, speed, temperature, ambient light, or ambient sound). The sensor management component 120 may determine that the sensor 134 is available if the sensor 134 is functioning properly, may respond to a request for data, or may deliver the detected data to the computing device 104, which is connected to the network 104 to route the data to the data processing system 102. The sensor management component 120 may determine that the sensor 134 is available if it is proximate to an end user of the computing device 104 or is otherwise in a location or configured to collect relevant data for generating an action data structure.

[0073] For example, the sensor management component 120 may determine that there are two sensors configured to collect location information associated with an account of an end user of the computing device 104. The sensor management component 120 may determine that the two sensors are close to the end user based on a comparison of the data collected by the two sensors because they are both collecting similar location information. The sensor management component 120 applying a policy may determine that only one of the sensors is to be polled for location data and the other sensor is disabled so as to reduce resource consumption compared to operating both sensors. For example, the two sensors may be configured on two different computing devices 104 (such as a smart phone and a smart watch) that are both close to the end user. The sensor management component 120 may determine the battery status of the two different computing devices 104 and select one of the two sensors configured on the computing device 104 that has a greater remaining battery power. The remaining battery power may be a percentage of the remaining power, the remaining absolute power, or an estimated amount of time that the battery can power the computing device 104 under the current utilization (e.g., processor or memory utilization, sensor utilization, network interface utilization). Therefore, the sensor management component 120 may disable one of the sensors to save battery consumption and request location information from the active sensor.

[0074] In some cases, the sensor management component 120 may piggyback previously collected information. For example, the sensor management component may determine that there are multiple available sensors close to the terminal user that are configured to obtain location information. The multiple available sensors may include a first sensor, a second sensor, or a third sensor. The sensor management component 120 may determine that the first sensor collects the current location information of the terminal user (for example, within a threshold time such as 1 second, 2 seconds, 3 seconds or longer). This information may have been collected before the direct action API requests sensor information from the sensor management component 120. In addition, the sensor management component 120 may determine that the third sensor is offline and lacks current location information. Therefore, instead of commanding the third sensor to enter an online state and consume resources, the sensor management component 120 may determine to use information from the first sensor or poll the first sensor to collect updated location information while leaving the third sensor offline or in a low-power state.

[0075] In some cases, the sensor management component 120 may instruct the client device 104 to perform a batch upload of the collected sensor data. The batch upload of the collected sensor data may reduce resource consumption compared to individual uploads or streaming the data as it is received by the sensor 134. The client device 104 may perform a batch upload in response to entering an online state from an offline state (e.g., reconnecting to the network 104; being turned on; or being within the line of sight of a GPS satellite). In some cases, the sensor management component 120 may instruct the computing device 104 to perform a batch upload based on location, such as entering or exiting a geo-fence (e.g., a retail store location or other physical location or area). In some cases, the computing device may upload a list of applications installed on the computing device 104 (e.g., by accessing a registry of the computing device or other database storing installed applications).

[0076] The direct action API 116 may populate one or more fields (e.g., a first field) with data received by the sensor management component 120 in response to polling the first sensor. The direct action API 116 may populate a second field with data received from a memory of the client device by the sensor management component 120 to reduce resource consumption. The direct action API 116 may then generate an action data structure based on the first field and the second field of the template.

[0077] The direct action API 116 may transmit the action data structure to a third-party provider device (e.g., the service provider device 108) to cause the third-party provider device 108 to invoke an operational session. An operational session may refer to or include the third-party provider device 108 performing an operation using or based on the action data structure, such as performing a requested service, purchasing a requested product, or invoking a dialog application programming interface (e.g., the service provider NLP component 142) to establish an operational session or a communication session between the third-party provider device 108 and the client computing device 104. In response to establishing the operational session or the communication session between the service provider device 108 and the client computing device 104, the service provider device 108 may transmit data packets directly to the client computing device 104 via the network 105. In some cases, the service provider device 108 may transmit the data packets to the client computing device 104 via the data processing system 102 and the network 105.

[0078] The data processing system 102 may receive an indication from the third party provider device 108 that the third party provider device 108 has established an operational session with the client device 104. For example, the indication may identify the type of operation being performed (e.g., providing a service such as a taxi service; purchasing a product; responding to a query). The data processing system 102 may also receive an indication that an operation was performed during the operational session (e.g., a taxi picked up an end user and transported the end user to a destination location). The indication that the operation was performed may be provided via an operation data structure that includes, for example, an identifier of the end user, a timestamp, a type of operation, an identifier of the third party service provider 108, or a price. The operation data structure may be formed using a template from the template data structure 132. The template may be standard for all operations, or customized based on the type of operation.

[0079] In some cases, the third party provider device 108 may execute at least a portion of the conversation API 142. For example, the third party provider device 108 may process certain aspects of the communication session or certain types of queries. The third party provider device 108 may utilize the NLP component 112 executed by the data processing system 102 to facilitate processing of audio signals associated with the communication session and generate responses to the queries. In some cases, the data processing system 102 may include a conversation API 142 configured for the third party provider 108. In some cases, the data processing system routes data packets between the client computing device and the third party provider device to establish a communication session. The data processing system 102 may receive an indication from the third party provider device 108 that the third party provider device has established a communication session with the client device 104. The indication may include an identifier of the client computing device 104, a timestamp relative to when the communication session is established, or other information associated with the communication session, such as an action data structure associated with the communication session. In some cases, data processing system 102 may include a session handler component 114 for managing an operational session or a communication session and a sensor management component 120 for managing or selecting sensors from which data is collected.

[0080] The data processing system 102 may include, execute, access, or otherwise communicate with the session handler component 114 to establish a communication session between the client device 104 and the data processing system 102. The communication session may refer to one or more data transmissions between the client device 104 and the data processing system 102, including input audio signals detected by the sensor 134 of the client device 104 and output signals transmitted by the data processing system 102 to the client device 104. The data processing system 102 (e.g., via the session handler component 114) may establish a communication session (e.g., an operational session) in response to receiving the input audio signal. The data processing system 102 may set a duration for the communication session. The data processing system 102 may set a timer or counter for the duration set for the communication session. In response to the expiration of the timer, the data processing system 102 may terminate the communication session.

[0081] A communication session may refer to a network-based communication session in which the client device 104 provides authentication information or credentials to establish the session. In some cases, a communication session refers to the subject or context of audio signals carried by data packets during the session. For example, a first communication session may refer to audio signals related to taxi services (e.g., including keywords, action data structures, or content item objects) transmitted between the client device 104 and the data processing system 102; while a second communication session may refer to audio signals related to laundry and dry cleaning services transmitted between the client device 104 and the data processing system 102. In this example, the data processing system 102 may determine that the contexts of the audio signals are different (e.g., via the NLP component 112) and separate the two groups of audio signals into different communication sessions. The session handler 114 may terminate the first session related to the ride service in response to identifying one or more audio signals related to the dry cleaning and laundry services. Therefore, the data processing system 102 may initiate or establish a second session for audio signals related to dry cleaning and laundry services in response to detecting the context of the audio signals.

[0082] The data processing system 102 may include, execute, or otherwise communicate with a content selector component 118 to receive trigger keywords identified by a natural language processor and select content items via a real-time content selection process based on the trigger keywords. In some cases, the direct action API 116 may transmit an action data structure to the content selector component 118 to perform a real-time content selection process and establish a communication session between the content provider device 106 (or third-party provider device 108) and the client computing device 104.

[0083] The content selection process may refer to or include selecting a sponsored content item object provided by a third-party content provider 106. The content selection process may include a service in which content items provided by multiple content providers are parsed, processed, weighted, or matched in order to select one or more content items to provide to the computing device 104. The content selection process may be performed in real time or offline. Performing the content selection process in real time may refer to performing the content selection process in response to a request for content received via the client computing device 104. The real-time content selection process may be performed (e.g., initiated or completed) within a time interval (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, or 20 minutes) of receiving the request. The real-time content selection process may be performed during a communication session with the client computing device 104 or within a certain time interval after the communication session is terminated.

[0084] For example, the data processing system 102 may include a content selector component 118 designed, constructed, configured, or operable to select content item objects. In order to select content items displayed in a speech-based environment, the data processing system 102 (e.g., via the NLP component 112) may parse the input audio signal to identify keywords (e.g., trigger keywords), and use these keywords to select matching content items based on broad matching, exact matching, or phrase matching. For example, the content selector component 118 may analyze, parse, or otherwise process the subject of the candidate content item to determine whether the subject of the candidate content item corresponds to the subject of the keyword or phrase of the input audio signal detected by the microphone of the client computing device 104. The content selector component 118 may use image processing technology, character recognition technology, natural language processing technology, or database search to identify, analyze, or recognize the voice, audio, term, character, text, symbol, or image of the candidate content item. The candidate content item may include metadata indicating the subject of the candidate content item, in which case the content selector component 118 may process the metadata to determine whether the subject of the candidate content item corresponds to the input audio signal.

[0085] The content provider 106 may provide additional indicators when setting a content activity that includes a content item. The content provider 106 may provide information at the content activity or content group level that the content selector component 118 may identify by performing a search using information about candidate content items. For example, a candidate content item may include a unique identifier that may be mapped to a content group, content activity, or content provider. The content selector component 118 may determine information about the content provider 106 based on information in a content activity data structure stored in the data repository 124.

[0086] Data processing system 102 may receive a request for content presented on computing device 104 via a computer network. Data processing system 102 may identify a request by processing an input audio signal detected by a microphone of client computing device 104. The request may include selection criteria for the request, such as device type, location, and keywords associated with the request. The request may include an action data structure or action data structures.

[0087] In response to the request, the data processing system 102 may select a content item object from a data repository 124 or a database associated with the content provider 106, and provide the content item via the network 105 for presentation via the computing device 104. The content item object may be provided by a content provider device 108 different from the service provider device 108. The content item may correspond to a type of service different from the type of service of the action data structure (e.g., the relationship between a taxi service and a food delivery service). The computing device 104 may interact with the content item object. The computing device 104 may receive an audio response to the content item. The computing device 104 may receive an indication for selecting a hyperlink or other button associated with the content item object, which enables or allows the computing device 104 to identify the service provider 108, request a service from the service provider 108, instruct the service provider 108 to perform a service, transmit information to the service provider 108, or otherwise query the service provider device 108.

[0088] The data processing system 102 may include, execute, or communicate with an audio signal generator component 122 to generate an output signal. The output signal may include one or more parts. For example, the output signal may include a first part and a second part. The first part of the output signal may correspond to an action data structure. The second part of the output signal may correspond to a content item selected by the content selector component 118 during the real-time content selection process.

[0089] The audio signal generator component 122 may generate an output signal having a first portion of sound corresponding to the first data structure. For example, the audio signal generator component 122 may generate a first portion of the output signal based on one or more values ​​populated into a field of the action data structure by the direct action API 116. In the taxi service example, the values ​​of the fields may include, for example, a pickup location of 123 Main Street, a destination location of 1234 Main Street, a number of passengers of 2, and an economy of service level. The audio signal generator component 122 may generate a first portion of the output signal to confirm that the end user of the computing device 104 wants to proceed with transmitting the request to the service provider 108. The first portion may include the following output "Would you like to an economy car from taxi service provider A to pick two people up at 123 Main Street and drop off at 1234 Main Street?"

[0090] In some cases, the first part may include information received from the service provider device 108. The information received from the service provider device 108 may be customized or tailored for the action data structure. For example, the data processing system 102 may transmit the action data structure to the service provider 108 before instructing the service provider 108 to perform an operation (e.g., via the direct action API 116). Alternatively, the data processing system 102 may instruct the service provider device 108 to perform initial or preliminary processing on the action data structure to generate preliminary information about the operation. In the example of a taxi service, the preliminary processing of the action data structure may include identifying available taxis that meet the service level requirements located around the ride location, estimating the amount of time it takes for the nearest available taxi to arrive at the ride location, estimating the time to arrive at the destination, and estimating the price of the taxi service. The estimated initial value may include a fixed value, an estimated value or a range of values ​​that are subject to change based on various conditions. The service provider device 108 may return the preliminary information to the data processing system 102 or directly to the client computing device 104 via the network 105. The data processing system 102 may incorporate the preliminary results from the service provider device 108 into an output signal and transmit the output signal to the computing device 104. The output signal may include, for example, "Taxi Service Company A can pick you up at 123 Main Street in 10 minutes, and drop you off at 1234 Main Street by 9 AM for $10. Do you want to order this ride?" This may form the first part of the output signal.

[0091] In some cases, the data processing system 102 may form a second portion of the output signal. The second portion of the output signal may include content items selected by the content selector component 118 during the real-time content selection process. The first portion may be different from the second portion. For example, the first portion may include information corresponding to the action data structure in direct response to the data packet carrying the input audio signal detected by the sensor 134 of the client computing device 104, while the second portion may include content items selected by the content selector component 104 that may be tangentially related to the action data structure, or include sponsored content provided by the content provider device 106. For example, the end user of the computing device 104 may request a taxi from taxi service company A. The data processing system 102 may generate the first portion of the output signal to include information about taxis from taxi service company A. However, the data processing system 102 may generate the second portion of the output signal to include content items selected based on the keyword "taxiservice" and information that the end user may be interested in contained in the action data structure. For example, the second portion may include content items or information provided by different taxi service companies (such as taxi service company B). Although the user may not have specifically requested Taxi Service Company B, the data processing system 102 may still provide content items from Taxi Service Company B because the user may have chosen to perform an operation with Taxi Service Company B.

[0092] Data processing system 102 may transmit information from the action data structure to Taxi Service Company B to determine the ride time, the time to the destination, and the price of the trip. Data processing system 102 may receive this information and generate a second portion of the output signal as follows: "Taxi Service Company B can pick you up at 123 Main Street in 2 minutes, and drop you off at 1234 Main Street by 8:82 AM for $15. Do you want this ride instead?" The end user of computing device 104 may then select a trip provided by Taxi Service Company A or a trip provided by Taxi Service Company B.

[0093] Before providing the sponsored content item corresponding to the service provided by taxi service company B in the second portion of the output signal, the data processing system 102 may notify the end user of the computing device that the second portion corresponds to the content item object selected during the real-time content selection process (e.g., by the content selector component 118). However, the data processing system 102 may have limited access to different types of interfaces to provide notifications to the end user of the computing device 104. For example, the computing device 104 may not include a display device, or the display device may be disabled or turned off. The display device of the computing device 104 may consume more resources than the speaker of the computing device 104, so turning on the display device of the computing device 104 may be less efficient than using the speaker of the computing device 104 to convey the notification. Therefore, in some cases, the data processing system 102 may improve the efficiency and effectiveness of information transmission through one or more interfaces or one or more types of computer networks. For example, the data processing system 102 (e.g., via the audio signal generator component 122) may modularize a portion of the output audio signal that includes the content item to provide an indication or notification to the end user that the portion of the output signal includes the sponsored content item.

[0094] The data processing system 102 (e.g., via the interface 110 and the network 105) may transmit data packets including the output signal generated by the audio signal generator component 122. The output signal may cause the audio driver component 138 of the client device 104 or the audio driver component 138 executed by the client device 104 to drive the speaker (e.g., the transducer 136) of the client device 104 to generate sound waves corresponding to the output signal.

[0095] Figure 2 is an illustration of the operation of a system for selectively polling sensors via a computer network. The system may include Figure 1 1. The NLP component 112 may receive and parse an audio signal detected by a computing device. At act 202, the NLP component 112 may pass information to the direct action API 116. The direct action API 116 may generate an action data structure in response to the audio signal parsed by the NLP component 112 and determining that sensor information is to be collected to populate one or more fields of a template.

[0096] At act 204, the direct action API can request sensor information from the sensor management component 120. The sensor management component 120 can establish a sensor data collection session 206 with multiple computing devices (e.g., the first, second, and third computing devices 104). At act 206, the sensor management component 120 can poll one or more of the computing devices 104. The sensor management component 120 can determine to poll only one of the computing devices 104 for sensor data to reduce aggregate resource consumption between multiple computing devices 104 that are relevant to the end user and provide similar information. The sensor management component 120 can instruct one or more of the computing devices 104 to poll the sensors 134.

[0097] At act 208, one or more of the computing devices 104 may poll, enable, activate, invoke, or otherwise cause the sensor 134 to collect sensor data and provide the sensor data to the computing device 104 for provision to the sensor management component 120 or the data processing system 102. At act 210, the sensor management component 120 may provide the collected sensor data to the direct action API 116. Thus, the sensor management component 120 may optimize sensor utilization to reduce aggregate resource utilization among multiple computing devices 104 associated with an end user (or proximate to an end user that may provide the requested sensor information) by selecting one or more sensors or a subset of such sensors to poll.

[0098] Figure 3 is an illustration of an example method for selectively polling a sensor. Method 300 may be performed by one or more components, systems, or elements of system 100 or system 400. Method 300 may include a data processing system receiving an input audio signal (act 305). The data processing system may receive the input audio signal from a client computing device. For example, a natural language processor component executed by the data processing system may receive the input audio signal from a client computing device via an interface of the data processing system. The data processing system may receive a data packet carrying or including an input audio signal detected by a sensor of a client computing device (or client device).

[0099] At act 310, method 300 may include a data processing system parsing an input audio signal. A natural language processor component may parse the input audio signal to identify a request and a trigger keyword corresponding to the request. For example, an audio signal detected by a client device may include "Okay device, I need a ride from Taxi Service Company A to go to 1234 Main Street". In this audio signal, an initial trigger keyword may include "okay device", which may indicate to the client device that the input audio signal is to be transmitted to the data processing system. A preprocessor of the client device may filter out the term "okay device" before sending the remaining audio signal to the data processing system. In some cases, the client device may filter out additional terms or generate keywords to be transmitted to the data processing system for further processing.

[0100] The data processing system may identify trigger keywords in the input audio signal. The trigger keywords may include, for example, "togo to" or "ride" or variations of these terms. The trigger keywords may indicate the type of service or product. The data processing system may identify a request in the input audio signal. The request may be determined based on the term "I need". Semantic processing techniques or other natural language processing techniques may be used to determine the trigger keywords and requests.

[0101] At behavior 315, the data processing system may select a template for the action data structure. The template may be responsive to a trigger keyword, request, or identified third-party provider. The template may include one or more fields, such as a first field.

[0102] At act 320, the data processing system may poll one or more sensors, or instruct the computing device to poll a sensor. Although the data processing system may identify multiple sensors that can provide data to populate the first field, the data processing system may select a sensor to poll. The data processing system may determine to poll a sensor that is not coupled to the computing device 104 that provided the input audio signal. For example, the end user may use a first computing device 104 to provide the input audio signal. The first computing device 104 may include a first sensor coupled to the first computing device. The data processing system may determine that there is a second computing device 104 that is also associated with the end user account. The second computing device may be proximate to the first computing device, or the data processing system may otherwise determine that the second computing device may also provide data of sufficient quality to populate the first field of the template (e.g., to bring the location information provided by the two sensors within a tolerance level such as 25 meters, 50 meters, 75 meters, or 100 meters). The data processing system may determine to use a second sensor coupled to the second computing device 104 instead of a first sensor coupled to the first computing device, even if the first computing device invoked, initiated, or established communication with the data processing system 102. The data processing system may use a resource reduction policy to determine to use a second sensor coupled to the second computing device. For example, the second computing device may have more battery remaining, have greater resource availability, or be connected to a power source.

[0103] In some cases, the data processing system may determine to retrieve data previously detected by a sensor from memory. For example, the data processing system may determine to poll one sensor without polling a second sensor, even if information from the second sensor is needed to populate a second field. For example, the data processing system may determine to reduce resource consumption by using previously collected information (e.g., prior to a request for sensor data). Therefore, the data processing system may poll the first sensor and use the stored data from the second sensor. The data processing system may determine to do so based on a rule or policy or by analyzing the stored data to determine that it meets a condition or threshold (e.g., collected within a certain time interval).

[0104] At act 325, the data processing system may generate an action data structure with the sensor data. The data processing system may generate the action data structure based on a trigger keyword, a request, a third-party provider device, or other information. The action data structure may be responsive to a request. For example, if an end user of a client computing device requests a taxi from taxi service company A, the action data structure may include information requesting taxi service from taxi service company A. The data processing system may select a template for taxi service company A and fill in the fields in the template with values ​​obtained from one or more sensors or memory to allow taxi service company A to dispatch a taxi to the user of the client computing device to pick up the user and transport the user to the requested destination.

[0105] At act 330, the data processing system may receive an indication from the third-party provider device that the third-party provider device has established an operation session with the client device. The indication may indicate that the operation has been initiated, pending, in progress, or completed. The indication may include a data structure with additional information about the operation.

[0106] Figure 4 4 is a block diagram of an example computer system 400. The computer system or computing device 400 may include or be used to implement the system 100 or its components, such as the data processing system 102. The data processing system 102 may include an intelligent personal assistant or a voice-based digital assistant. The computing system 400 includes a bus 405 or other communication component for transmitting information and a processor 410 or processing circuit coupled to the bus 405 for processing information. The computing system 400 may also include one or more processors 410 or processing circuits coupled to the bus for processing information. The computing system 400 also includes a main memory 415, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 405 for storing information and instructions to be executed by the processor 410. The main memory 415 may be or include the data repository 145. The main memory 415 may also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 410. The computing system 400 may also include a read-only memory (ROM) 420 or other static storage device coupled to the bus 405 for storing static information and instructions for the processor 410. A storage device 425, such as a solid-state device, a magnetic disk, or an optical disk, may be coupled to the bus 405 to persistently store information and instructions. The storage device 425 may include or be part of the data repository 145.

[0107] The computing system 400 may be coupled via the bus 405 to a display 435, such as a liquid crystal display or an active matrix display, for displaying information to a user. An input device 430, such as a keyboard including alphanumeric and other keys, may be coupled to the bus 405 for communicating information and commands to the processor 410. The input device 430 may include a touch screen display 435. The input device 430 may also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 410 and for controlling movement of a cursor on the display 435. For example, the display 435 may be a Figure 1 Part of data processing system 102, client computing device 150, or other component.

[0108] The processes, systems, and methods described herein may be implemented by the computing system 400 in response to the processor 410 executing an arrangement of instructions contained in the main memory 415. Such instructions may be read into the main memory 415 from another computer-readable medium, such as the storage device 425. The execution of the arrangement of instructions contained in the main memory 415 causes the computing system 400 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement may also be employed to execute the instructions contained in the main memory 415. Hard-wired circuitry may be used in place of software instructions or in conjunction with the systems and methods described herein. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0109] Although already Figure 4 An example computing system is described in the specification, however the subject matter including the operations described in this specification may be implemented with other types of digital electronic circuitry, or with computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or with a combination of one or more of them.

[0110] For situations where the systems discussed herein collect personal information about users or can make use of personal information, users may be provided with the opportunity to control whether programs or features can collect personal information (e.g., information about the user's social network, social actions or activities, the user's preferences, or the user's location) or to control whether or how to receive content that may be more relevant to the user from a content server or other data processing system. In addition, certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed when generating parameters. For example, the identity of a user may be anonymized so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (such as to a city, zip code, or state level) in the case where location information is obtained, so that the user's specific location cannot be determined. Therefore, the user can control how information is collected about him or her and used by the content server.

[0111] The subject matter and operations described in this specification may be implemented with digital electronic circuits or with computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or with a combination of one or more of them. The subject matter described in this specification may be implemented as one or more computer programs (e.g., one or more circuits of computer program instructions) encoded on one or more computer storage media for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by a data processing device). A computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Although a computer storage medium is not a propagation signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagation signal. A computer storage medium may also be or be included in one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0112] The terms "data processing system", "computing device", "component" or "data processing apparatus" include various apparatuses, devices and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip or multiple programmable processors, computers, systems on a chip, or a combination of the above. The apparatus may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. For example, the direct action API 116, the content selector component 118, or the NLP component 112 and other data processing system 102 components may include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0113] A computer program (also referred to as a program, software, software application, app, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system. A computer program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple collaborative files (e.g., files storing portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0114] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs (e.g., components of data processing system 102) to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and the apparatus may also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0115] The subject matter described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser that a user can use to interact with an implementation of the subject matter described in this specification), or a combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., peer-to-peer point-to-point networks).

[0116] A computing system such as system 100 or system 400 may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network (e.g., network 165). The relationship of the client and server is generated by computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., data packets representing content items) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user interacting with the client device). Data generated at the client device (e.g., the result of the user interaction) can be received from the client device at the server (e.g., received by the data processing system 102 from the computing device 150 or the content provider computing device 155 or the service provider computing device 160).

[0117] Although operations are depicted in the drawings in a particular order, it is not required that such operations be performed in the particular order shown or in sequential order, and it is not required that all illustrated operations be performed. The actions described herein may be performed in a different order.

[0118] Separation of various system components is not required in all embodiments, and the described program components may be included in a single hardware or software product. For example, NLP component 110 or content selector component 125 or sensor management component 120 may be a single component, app or program, or a logical device having one or more processing circuits, or part of one or more servers of data processing system 102.

[0119] Having now described some illustrative embodiments, it will be apparent that the foregoing is illustrative and not restrictive, having been presented by way of example. In particular, although many of the examples presented herein relate to specific combinations of method behaviors or system elements, those behaviors and those elements may be combined in other ways to achieve the same goals. The behaviors, elements, and features discussed in conjunction with one embodiment are not intended to be excluded from similar roles in other implementations or embodiments.

[0120] The words and terms used herein are for descriptive purposes and should not be considered limiting. The use of "comprising," "including," "having," "involving," "characterized by," "characterized by," and variations thereof are intended to encompass alternative embodiments consisting of the items listed thereafter, their equivalents and additions, as well as the items listed thereafter exclusively. In one embodiment, the systems and methods described herein are comprised of one, more than one, each combination, or all of all of the described elements, acts, or components.

[0121] Any reference to an embodiment or element or behavior of a system and method referenced herein in the singular may also include an embodiment including a plurality of such elements, and any reference to any embodiment or element or behavior referenced herein in the plural may also include an embodiment including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, behaviors, or elements to a single or multiple configurations. References to any behavior or element based on any information, behavior, or element may include an embodiment in which the behavior or element is based, at least in part, on any information, behavior, or element.

[0122] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment," "some embodiments," "an embodiment," etc. are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment or example. Terms as used herein do not necessarily all refer to the same embodiment. Any embodiment may be combined, inclusively or exclusively, with any other embodiment in any manner consistent with the aspects and embodiments disclosed herein.

[0123] References to "or" may be interpreted as inclusive, such that any term described using "or" may indicate any of a single, more than one, and all of the terms described. For example, a reference to "at least one of 'A' and 'B'" may include only 'A'", only 'B', and both 'A' and 'B'. Such references used in conjunction with "including" or other open-ended terms may include additional items.

[0124] Where reference numerals are attached to technical features in the drawings, detailed description or any claims, these reference numerals have been included to increase the intelligibility of the drawings, detailed description and claims. Therefore, neither the reference numerals nor their absence have any impact on the scope of any claim element.

[0125] The system and method described herein can be specifically implemented in other specific forms without departing from its characteristics. For example, the data processing system 102 can select content items for subsequent actions (e.g., for the third action 215) based in part on the data of the previous action in the action sequence from the thread 200 (such as the data indicating that the second action 210 is completed or about to start from the second action 210). The above-mentioned embodiments are illustrative and do not limit the described system and method. The scope of the system and method described herein is therefore indicated by the appended claims rather than the above description, and changes falling within the meaning and scope of the equivalents of the claims are included therein.

Claims

1. A system comprising: A data processing system comprising a memory and one or more processors for: receiving, via an interface of the data processing system, a data packet comprising an input audio signal detected by a microphone of a client device; parsing the input audio signal to identify a request and a keyword corresponding to the request; selecting a template for an action data structure responsive to the request based on the keyword, the action data structure being used to encapsulate one or more parameters used by a third-party provider device to perform a provision-type or product-type operation, the template comprising a first field; identifying a plurality of available sensors configured to obtain information of the first field of the action data structure, the plurality of available sensors being coupled to one or more client devices different from the client device, the one or more client devices i) each being associated with an end-user account associated with the client device, ii) each being within a threshold distance of the client device, and iii) each having successfully performed a handshake procedure with the data processing system using credentials of the end-user account and being currently online; selecting a first sensor of the plurality of available sensors of a second client device of the one or more client devices for data corresponding to the first field of the action data structure used by the third-party provider device to perform an operation of providing the service type or the product type; generating the action data structure using data received from the first sensor to provide the service type or the product type based on the first field of the template; as well as The action data structure is transmitted to the third party provider device to cause the third party provider device to provide one or more actions of the service type or the product type based on the one or more parameters encapsulated in the action data structure.

2. The system according to claim 1, comprising: The data processing system is configured to determine a sensor of the client device that is not used, wherein the first sensor selected for the data is not coupled to the client device.

3. The system according to claim 1, comprising the data processing system being configured to: determining a status of each of the plurality of available sensors; and The first sensor of the second client device is selected based on a state of the first sensor.

4. The system according to claim 1, comprising the data processing system being configured to: The first sensor is polled for data corresponding to the first field of the action data structure.

5. The system according to claim 1, comprising the data processing system being configured to: The action data structure is provided to the third party provider device to invoke an operational session between the third party provider device and the client device.

6. The system according to claim 1, comprising the data processing system being configured to: An indication of performance of the one or more actions is received from the third party provider device.

7. The system according to claim 1, comprising the data processing system being configured to: identifying a second plurality of available sensors configured to obtain location information, the second plurality of available sensors comprising the first sensor and a second sensor; determining a battery status of each of the plurality of available sensors; as well as The first sensor of the plurality of available sensors is selected based on a battery status of each of the plurality of available sensors.

8. The system according to claim 1, comprising the data processing system being configured to: receiving data including position information from the first sensor; selecting a second sensor from the plurality of available sensors; and Additional information is received from the second sensor.

9. The system according to claim 1, comprising the data processing system being configured to: identifying a second plurality of available sensors configured to obtain location information, the second plurality of available sensors comprising the first sensor and a second sensor; determining, prior to a request for location information from the data processing system, that the first sensor detects current location information of the client device; as well as The current position information is obtained from the first sensor.

10. The system according to claim 1, comprising the data processing system being configured to: identifying a second plurality of available sensors configured to obtain location information, the second plurality of available sensors comprising the first sensor and a second sensor; determining that the second sensor is in an offline state and lacks current location information; as well as The current location information is obtained from the first sensor, wherein the second sensor is in an offline state.

11. A method comprising: receiving, by a data processing system including one or more processors and memory, via an interface of the data processing system, a data packet including an input audio signal detected by a microphone of a client device; parsing, by the data processing system, the input audio signal to identify a request and a keyword corresponding to the request; selecting, by the data processing system, a template for an action data structure responsive to the request based on the keyword, the action data structure being used to encapsulate one or more parameters used by a third-party provider device to perform a provision-type or product-type operation, the template comprising a first field; identifying, by the data processing system, a plurality of available sensors configured to obtain information of the first field of the action data structure, the plurality of available sensors being coupled to one or more client devices different from the client device, the one or more client devices i) being associated with an end-user account associated with the client device, ii) being within a threshold distance of the client device, and iii) having successfully performed a handshake procedure with the data processing system using credentials of the end-user account and being currently online; selecting, by the data processing system, a first sensor of the plurality of available sensors of a second client device of the one or more client devices for data corresponding to the first field of the action data structure used by the third-party provider device to perform an operation of providing the service type or the product type; generating, by the data processing system, the action data structure using data received from the first sensor to provide the service type or the product type based on the first field of the template; as well as The action data structure is transmitted by the data processing system to the third-party provider device to cause the third-party provider device to provide one or more actions of the service type or the product type based on the one or more parameters encapsulated in the action data structure.

12. The method according to claim 11, comprising: A determination is made, by the data processing system, not to use a sensor of the client device, wherein the first sensor selected for the data is not coupled to the client device.

13. The method according to claim 11, comprising: determining, by the data processing system, a status of each of the plurality of available sensors; as well as The first sensor of the second client device is selected, by the data processing system, based on a state of the first sensor.

14. The method according to claim 11, comprising: The first sensor is polled by the data processing system for data corresponding to the first field of the action data structure.

15. The method according to claim 11, comprising: The action data structure is provided by the data processing system to the third party provider device to invoke an operational session between the third party provider device and the client device.

16. The method according to claim 11, comprising: An indication of performance of the one or more actions is received by the data processing system from the third party provider device.

17. The method according to claim 11, comprising: identifying, by the data processing system, a second plurality of available sensors configured to obtain location information, the second plurality of available sensors including the first sensor and a second sensor; determining, by the data processing system, a battery status of each of the plurality of available sensors; as well as The first sensor of the plurality of available sensors is selected by the data processing system based on a battery status of each of the plurality of available sensors.

18. The method according to claim 11, comprising: receiving, by the data processing system, data including position information from the first sensor; selecting, by the data processing system, a second sensor from among the plurality of available sensors; as well as Additional information is received by the data processing system from the second sensor.

19. The method according to claim 11, comprising: identifying, by the data processing system, a second plurality of available sensors configured to obtain location information, the second plurality of available sensors including the first sensor and a second sensor; determining, by the data processing system, prior to a request for location information from the data processing system, that the first sensor detects current location information of the client device; as well as The current position information is obtained by the data processing system from the first sensor.

20. The method according to claim 11, comprising: identifying, by the data processing system, a second plurality of available sensors configured to obtain location information, the second plurality of available sensors including the first sensor and a second sensor; determining, by the data processing system, that the second sensor is in an offline state and lacks current location information; as well as The current location information is obtained by the data processing system from the first sensor, wherein the second sensor is in an offline state.

Images