Feedback controller for data transmission

Abstract

The present invention relates to a feedback controller for data transmission. A feedback control system for data transmission in a voice-activated packet-based computer network environment is provided. A system can receive an audio signal detected by a microphone of a device. The system can parse the audio signal to identify trigger keywords and requests. The system can select content items using the trigger keywords or requests. The content items can be configured to establish a communication session between the device and a third-party device. The system can monitor the communication session to measure characteristics of the communication session. The system can generate a quality signal based on the measured characteristics.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201780001629.X, filed on August 31, 2017.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims the benefit of U.S. Patent Application No. 15 / 395,694, filed on December 30, 2016, entitled “FEEDBACK CONTROLLER FOR DATA TRANSMISSIONS,” which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0005] The present invention relates to a feedback controller for data transmission. Background Art

[0006] Excessive network transmission of network traffic data between computing devices, packet-based or otherwise, may prevent computing devices from properly processing the network traffic data, completing operations related to the network traffic data, or responding to the network traffic data in a timely manner. Excessive network transmission of network traffic data may also complicate data routing or reduce response quality if the responding computing device is at or exceeds its processing capacity, which may result in inefficient bandwidth utilization. Control of network transmissions corresponding to content item objects can be complicated by a large number of content item objects that can initiate network transmissions of network traffic data between computing devices. Summary of the invention

[0007] The present disclosure generally relates to a feedback controller for transmitting data through one or more interfaces or one or more types of computer networks. For example, a computing system can have access to a limited number of interfaces, a limited number of types of interfaces, or a limited number of available interfaces at a given time. Because some types of interfaces may consume more computing resources or batteries, it is challenging for the system to effectively transmit information in response to currently available interfaces. Because different computing resources are challenging to efficiently process and consistently and accurately parse audio-based instructions in a voice-based computing environment, it is challenging to effectively, reliably and accurately transmit information on different computing resources. For example, different computing resources may not have access to the same voice model, or may access an outdated or unsynchronized voice model, which may make it challenging to accurately and consistently parse audio-based instructions.

[0008] The system and method of the present disclosure generally relate to a feedback controller for data transmission. A data processing system can process speech-based input using a speech model trained based on aggregated speech to parse speech-based instructions and select content items through a real-time content selection process performed by a content selector component. The data processing system can transmit the selected content items to a client computing device to initiate a communication session between the client computing device and a third-party provider device associated with the selected content items. The data processing system can monitor or otherwise receive information about the communication session to measure the characteristics of the communication session and generate a quality signal. Then, the data processing system can adjust or control the content selector component based on the quality signal to affect the real-time content selection process. For example, compared with allowing or allowing a content item to be selected and establishing a communication session, preventing or preventing a content selector component from selecting a content item object associated with a low-quality communication session can reduce wasteful resource consumption. In addition, for a client device using battery power, a feedback monitor component can save battery use.

[0009] At least one aspect relates to a feedback control system for data transmission over a computer network. The system can include a data processing system that executes a natural language processor and a content selector component. The system can include a feedback monitor component. The natural language processor component can receive a data packet including an input audio signal detected by a sensor of a client device via an interface of the data processing system. The natural language processor component can parse the input audio signal to identify a request and a trigger keyword corresponding to the request. The data processing system can include a content selector component for receiving a trigger keyword identified by the natural language processor, and for selecting a content item via a real-time content selection process based on the trigger keyword. The system can include a feedback monitor component. The feedback monitor component can receive a data packet carrying an auditory signal transmitted between a client device and a conversational application programming interface, which establishes a communication session with the client device in response to an interaction with the content item. The feedback monitor can measure characteristics of the communication session based on the auditory signal. The feedback monitor component can generate a quality signal based on the measured characteristics. The content selector component can adjust the real-time selection process based on the quality signal.

[0010] At least one aspect relates to a method for transmitting data over a computer network using a feedback control system. The method can be performed at least in part by a data processing system executing a natural language processor component and a content selector component. The method can be performed at least in part by a feedback monitor component. The method can include the natural language processor component receiving, via an interface of the data processing system, a data packet including an input audio signal detected by a sensor of a client device. The method can include the data processing system parsing the input audio signal to identify a request and a trigger keyword corresponding to the request. The method can include the content selector component receiving the trigger keyword identified by the natural language processor. The method can include the content selector component selecting a content item via a real-time content selection process based on the trigger keyword. The method can include the feedback monitor component receiving a data packet carrying an auditory signal transmitted between the client device and a conversational application programming interface, the conversational application programming interface establishing a communication session with the client device in response to interaction with the content item. The method can include the feedback monitor component measuring the quality of the communication session based on the auditory signal. The method can include the feedback monitor component generating a quality signal based on the measured characteristics. The method can include the feedback monitor component adjusting the real-time selection process based on the quality signal.

[0011] These and other aspects and embodiments are discussed in detail below. The above information and the following detailed description contain illustrative examples of the various aspects and embodiments, and provide an overview or framework for understanding the nature and characteristics of the claimed aspects and embodiments. The accompanying drawings provide illustration and further understanding of the various aspects and embodiments, and are incorporated into and constitute part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings are not intended to be drawn to scale. Like reference numerals and numbers refer to like elements. For clarity, not every component will be labeled in every drawing. In the drawings:

[0013] Figure 1 is a diagram of a feedback control system for data transmission over a computer network.

[0014] Figure 2 is a diagrammatic illustration of the operation of a feedback control system for data transmission over a computer network.

[0015] Figure 3 is a graphic representation of a method for transmitting data over a computer network using a feedback control system.

[0016] Figure 4 is a block diagram illustrating the general architecture for a computer system that can be employed to implement the system elements and methods described and illustrated herein. DETAILED DESCRIPTION

[0017] The following is a more detailed description of various concepts and embodiments of methods, apparatus, and systems for feedback control systems for data transmission over a computer network.The various concepts introduced above and discussed in more detail below can be implemented in any of numerous ways.

[0018] The present disclosure generally relates to a feedback controller for transmitting data through one or more interfaces or one or more types of computer networks. For example, a computing system may have access to a limited number of interfaces, a limited number of types of interfaces, or a limited number of available interfaces that may exist at a given time. Because certain types of interfaces may consume more computing resources or power, it is challenging for the system to effectively transmit information in response to currently available interfaces. Because it is challenging for different computing resources to efficiently process and consistently and accurately parse audio-based instructions in a voice-based computing environment, it is challenging to effectively, reliably, and accurately transmit information on different computing resources. For example, different computing resources may not have access to the same voice model, or may have access to outdated or unsynchronized voice models, which may make it challenging to accurately and consistently parse audio-based instructions.

[0019] The systems and methods of the present disclosure generally relate to a feedback controller for data transmission. A data processing system can process speech-based input using a speech model trained based on aggregated speech to parse speech-based instructions and select content items via a real-time content selection process performed by a content selector component. The data processing system can transmit the selected content items to a client computing device to initiate a communication session between the client computing device and a third-party provider device associated with the selected content item. The data processing system can monitor or otherwise receive information about the communication session to measure the characteristics of the communication session and generate a quality signal. The data processing system can then adjust or control the content selector component based on the quality signal to affect the real-time content selection process.

[0020] Figure 1An example feedback control system 100 for data transmission over a computer network is shown. System 100 can include a content selection infrastructure. System 100 can include a data processing system 102. Data processing system 102 can communicate with one or more of a content provider computing device 106, a service provider computing device 108, or a client computing device 104 via a network 105. Network 105 can include a computer network such as the Internet, a local network, a wide area network, a metropolitan area network, or other regional networks, an intranet, a satellite network, and other communication networks such as voice or data mobile phone networks. Network 105 can be used to access information resources such as web pages, websites, domain names, or uniform resource locators that can be displayed, output, presented, or displayed on at least one computing device 104 (e.g., a laptop, desktop, tablet, personal digital assistant, smart phone, portable computer, or speaker). For example, via network 105, a user of computing device 104 can access information or data provided by service provider 108 or content provider 106.

[0021] Network 105 can include or constitute a display network, such as a subset of information resources available on the Internet that are associated with a content placement or search engine results system or can include third-party content items as part of a content item placement promotion campaign. Network 105 can be used by data processing system 102 to access information resources such as web pages, websites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed by client computing device 104. For example, a user of client computing device 104 can access information or data provided by content provider computing device 106 or service provider computing device 108 via network 105.

[0022] The network 105 can be any type or form of network, and can include any of the following: a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an asynchronous transfer mode (ATM) network, a synchronous optical network (SONET) network, a synchronous digital hierarchy (SDH) network, a wireless network, and a wired network. The network 105 can include wireless links such as infrared channels or satellite bands. The topology of the network 105 can include a bus, a star, or a ring network topology. The network can include a mobile phone network using any one or more protocols for communicating between mobile devices, including Advanced Mobile Phone Protocol ("AMPS"), Time Division Multiple Access ("TDMA"), Code Division Multiple Access ("CDMA"), Global System for Mobile Communications ("GSM"), General Packet Radio Service ("GPRS"), or Universal Mobile Telecommunications System ("UMTS"). Different types of data can be transmitted via different protocols, or the same type of data can be transmitted via different protocols.

[0023] System 100 can include at least one data processing system 102. Data processing system 102 can include at least one logical device, such as a computing device having a processor to communicate with, for example, computing device 104, content provider device 106 (content provider 106), or service provider device 108 (or service provider 108) via network 105. Data processing system 102 can include at least one computing resource, server, processor, or memory. For example, data processing system 102 can include multiple computing resources or servers located in at least one data center. Data processing system 102 can include multiple logically grouped servers and facilitate distributed computing technology. Logical groups of servers can be referred to as data centers, server farms, or machine farms. Servers can also be geographically dispersed. A data center or machine farm can be managed as a single entity, or a machine farm can include multiple machine farms. The servers in each machine farm can be heterogeneous—one or more of the servers or machines can operate according to one or more types of operating system platforms.

[0024] The servers in the machine farm can be stored in high-density rack systems and associated storage systems and located in an enterprise data center. Consolidating servers in this manner can improve system manageability, data security, physical security of the system, and system performance by locating the servers and high-performance storage systems on a localized high-performance network, for example. Centralizing all or some of the data processing system 102 components (including servers and storage systems) and coupling them with advanced system management tools allows for more efficient use of server resources, thereby saving power and processing requirements and reducing bandwidth usage.

[0025] System 100 can include, access, or otherwise interact with at least one service provider device 108. Service provider device 108 can include at least one logical device, such as a computing device having a processor to communicate with, for example, computing device 104, data processing system 102, or content provider 106 via network 105. Service provider device 108 can include at least one computing resource, server, processor, or memory. For example, service provider device 108 can include multiple computing resources or servers located in at least one data center. Service provider device 108 can include one or more components or functions of data processing system 102.

[0026] The content provider computing device 106 can provide audio-based content items for display as audio output content items by the client computing device 104. The content item can include an offer of goods or services, such as a voice-based message stating: "Do you want me to reserve a taxi for you?" For example, the content provider computing device 155 can include a memory to store audio content items that can be provided in response to voice-based queries. The content provider computing device 106 can also provide audio-based content items (or other content items) to the data processing system 102, which can be stored in the data repository 124. The data processing system 102 can select the audio content item and provide (or instruct the content provider computing device 104 to provide) the audio content item to the client computing device 104. The audio-based content item can be dedicated audio, or can be combined with text, image, or video data.

[0027] The service provider device 108 can include, interface with, or otherwise communicate with at least one service provider natural language processor component 142 and a service provider interface 144. The service provider computing device 108 can include at least one service provider natural language processor (NLP) component 142 and at least one service provider interface 144. The service provider NLP component 142 (or other component such as a direct action API of the service provider computing device 108) can be combined with the client computing device 104 (via the data processing system 102 or bypassing the data processing system 102) to create a back-and-forth real-time voice or audio-based dialogue (e.g., a session) between the client computing device 104 and the service provider computing device 108. The service provider NLP 142 can include one or more functions or features as the NLP component 112 of the data processing system 102. For example, the service provider interface 144 can receive or provide data messages to the direct action API 116 of the data processing system 102. The service provider computing device 108 and the content provider computing device 106 can be associated with the same entity. For example, the content provider computing device 106 can create, store, or make available content items for a car sharing service, and the service provider computing device 108 can establish a session with the client computing device 106 to arrange for the delivery of a taxi or a car for the car sharing service to pick up an end user of the client computing device 104. The data processing system 102 can also establish a session with the client computing device, including or bypassing the service provider computing device 104, via the direct action API 116, the NLP component 112, or other components, to arrange, for example, for the delivery of a taxi or a car for the car sharing service.

[0028] The computing device 104 can include, interface with, or otherwise communicate with at least one sensor 134, a transducer 136, an audio driver 138, or a preprocessor 140. The sensor 134 can include, for example, an ambient light sensor, a proximity sensor, a temperature sensor, an accelerometer, a gyroscope, a motion detector, a GPS sensor, a position sensor, a microphone, or a touch sensor. The transducer 136 can include a speaker or a microphone. The audio driver 138 can provide a software interface to the hardware transducer 136. The audio driver can execute an audio file or other instructions provided by the data processing system 102 to control the transducer 136 to generate corresponding sound waves or sound waves. The preprocessor 140 can be configured to detect keywords and perform actions based on the keywords. The preprocessor 140 can filter out one or more terms or modify terms before transmitting the terms to the data processing system 102 for further processing. Preprocessor 140 can convert analog audio signals detected by microphone into digital audio signals and transmit one or more data packets carrying the digital audio signals to data processing system 102 via network 105. In some cases, preprocessor 140 can transmit data packets carrying some or all of the input audio signals in response to detecting instructions to perform such transmission. The instructions can include, for example, a trigger keyword or other keyword or authorization to transmit a data packet including the input audio signal to data processing system 102.

[0029] The client computing device 104 can be associated with an end user who inputs a voice query as audio input into the client computing device 104 (via the sensor 134) and receives audio output from the transducer 136 (e.g., a speaker) in the form of computer-generated speech that can be provided from the data processing system 102 (or the content provider computing device 106 or the service provider computing device 108) to the client computing device 104. The computer-generated speech can include a recording from a real person or a computer-generated language.

[0030] The data repository 124 can include one or more local or distributed databases, and can include a database management system. The data repository 124 can include a computer data storage device or memory, and can store one or more parameters 126, one or more policies 128, content data 130 or templates 132, and other data. The parameters 126, policies 128, and templates 132 can include information such as rules for voice-based conversations between the client computing device 104 and the data processing system 102 (or service provider computing device 108). The content data 130 can include content items for audio output or related metadata, as well as input audio messages that can be part of one or more communication sessions with the client computing device 104.

[0031] The data processing system 102 can include a content placement system having at least one computing resource or server. The data processing system 102 can include, interface with, or otherwise communicate with at least one interface 110. The data processing system 102 can include, interface with, or otherwise communicate with at least one natural language processor component 112. The data processing system 102 can include, interface with, or otherwise communicate with at least one direct action application programming interface (“API”) 116. The data processing system 102 can include, interface with, or otherwise communicate with at least one session handler 114. The data processing system 102 can include, interface with, or otherwise communicate with at least one content selector component 118. The data processing system 102 can include, interface with, or otherwise communicate with at least one feedback monitor component 120. The data processing system 102 can include, interface to, or otherwise communicate with at least one audio signal generator 122. The data processing system 102 can include, interface to, or otherwise communicate with at least one data repository 124. At least one data repository 124 can include or store parameters 126, policies 128, content data 130, or templates 132 in one or more data structures or databases. The parameters 126 can include, for example, thresholds, distances, time intervals, durations, scores, or weights. The content data 130 can include, for example, content promotion campaign information, content groups, content selection criteria, content item objects, or other information provided by the content provider 106 or obtained or determined by the data processing system to facilitate content selection. The content data 130 can include, for example, historical performance of content promotion campaigns.

[0032] The interface 110, the natural language processor component 112, the conversation handler 114, the direct action API 116, the content selector component 118, the feedback monitor component 120, or the audio signal generator component 122 can each include at least one processing unit or other logic device, such as a programmable logic array engine or a module configured to communicate with a data repository or database 124. The interface 110, the natural language processor component 112, the conversation handler 114, the direct action API 116, the content selector component 118, the feedback monitor component 120, the audio signal generator component 122, and the data repository 124 can be separate components, a single component, or part of the data processing system 102. The system 100 and its components, such as the data processing system 102, can include hardware elements, such as one or more processors, logic devices, or circuits.

[0033] Data processing system 102 can obtain anonymous computer network activity information associated with multiple computing devices 104. Users of computing devices 104 can affirmatively authorize data processing system 102 to obtain network activity information corresponding to the user's computing device. For example, data processing system 102 can prompt the user of computing device 104 to consent to obtaining one or more types of network activity information. The identity of the user of computing device 104 can remain anonymous, and computing device 104 can be associated with a unique identifier (e.g., a unique identifier of the user or computing device provided by the data processing system or the user of the computing device). The data processing system can associate each observation with a corresponding unique identifier.

[0034] The content provider 106 can establish an electronic content campaign. The electronic content campaign can be stored in the data repository 124 as content data 130. The electronic content campaign can refer to one or more content groups associated with a common theme. The content campaign can contain a hierarchical data structure, which contains content groups, content item data objects, and content selection criteria. In order to create a content campaign, the content provider 106 can specify the value of the campaign level parameter for the content campaign. The campaign level parameter can contain, for example, the campaign name, the preferred content network for placing the content item object, the value of the resource to be used for the content campaign, the start and end date of the content campaign, the duration of the content campaign, the arrangement of the content item object placement, the language, the geographical location, and the type of computing device on which the content item object is to be provided. In some cases, an impression can refer to when the content item object is obtained from its source (e.g., the data processing system 102 or the content provider 106) and is countable. In some cases, due to the possibility of click fraud, robot activities can be filtered and excluded as impressions. Thus, in some cases, a flash can refer to a measurement of a response from a web server to a page request from a browser, filtered from robot activity and error codes, and recorded at a point in time as close as possible to the opportunity to render a content item object for display on computing device 104. In some cases, a flash can refer to a viewable or audible flash; for example, a content item object is at least partially (e.g., 20%, 30%, 30%, 40%, 50%, 60%, 70%, or more) viewable on a display device of client computing device 104, or audible via speaker 136 of computing device 104. Clicking or selecting can refer to a user's interaction with a content item object, such as a voice response to an audible flash, a mouse click, a touch interaction, a gesture, a shake, an audio interaction, or a keyboard click. A conversion can refer to a user taking a desired action on a content item object; for example, purchasing a product or service, completing a survey, visiting a physical store corresponding to a content item, or completing an electronic transaction.

[0035] The content provider 106 can further establish one or more content groups for a content campaign. A content group contains one or more content item objects and corresponding content selection criteria, such as keywords, words, terms, phrases, geographic locations, types of computing devices, time of day, interests, themes, or verticals. Content groups under the same content campaign can share the same campaign-level parameters, but may adjust specifications for specific content group-level parameters, such as keywords, negative keywords (e.g., preventing placement of a content item in the presence of negative keywords about the primary content), keyword bids, or parameters associated with bids or content campaigns.

[0036] In order to create a new content group, the content provider can provide the value of the content group level parameter for the content group. The content group level parameter includes, for example, a content group name or a content group theme, and bids for different content placement opportunities (e.g., automatic placement or managed placement) or results (e.g., clicks, flashes, or conversions). The content group name or content group theme can be one or more terms that the content provider 106 can use to capture the topic or theme of the content item object to be selected for display of the content group. For example, a car dealer can create different content groups for each brand of vehicle it supports, and can further create different content groups for each model of vehicle it supports. Examples of content group themes that car dealers can use include, for example, "Make A Sports Car", "Make B Sports Car", "Make C Sedan", "Make C Truck", "Make C Hybrid" or "Make D Hybrid". The theme of the example content promotion campaign can be "Hybrid", and includes content groups for both "Make C Hybrid" and "Make D Hybrid".

[0037] Content provider 106 can provide one or more keywords and content item objects to each content group. Keywords can include terms related to the product or service associated with or identified by the content item object. Keywords can include one or more terms or phrases. For example, a car dealer can use "sports car", "V-6 engine", "four-wheel drive", "fuel efficiency" as keywords for content groups or content promotion activities. In some cases, content providers can specify negative keywords to avoid, prevent, block or disable content placement of specific terms or keywords. Content providers can specify matching types for selecting content item objects, such as exact match, phrase match, or broad match.

[0038] Content providers 106 can provide one or more keywords to be used by data processing system 102 to select content item objects provided by content providers 106. Content providers 106 can identify one or more keywords to bid on, and further provide bid amounts for various keywords. Content providers 106 can provide additional content selection criteria to be used by data processing system 102 to select content item objects. Multiple content providers 106 can bid on the same or different keywords, and data processing system 102 can run a content selection process or an advertising auction in response to receiving an indication of the keywords of the electronic message.

[0039] Content provider 106 can provide one or more content item objects selected by data processing system 102. Data processing system 102 (e.g., via content selector component 118) can select content item objects when content placement opportunities become available, matching resource allocation, content plan, highest bid, keywords, and other selection criteria specified for content groups. Different types of content item objects can be included in content groups such as voice content items, audio content items, text content items, image content items, video content items, multimedia content items, or content item links. After selecting a content item, data processing system 102 can transmit the content item object for rendering on a computing device 104 or a display device of computing device 104. Rendering can include displaying the content item on a display device, or playing the content item via a speaker of computing device 104. Data processing system 102 can provide instructions to computing device 104 to render content item objects. Data processing system 102 can instruct computing device 104 or an audio driver 138 of computing device 104 to generate an audio signal or sound wave.

[0040] The data processing system 102 can include an interface component 110 designed, configured, constructed, or operated to receive and transmit information using, for example, data packets. The interface 110 can receive and transmit information using one or more protocols such as a network protocol. The interface 110 can include a hardware interface, a software interface, a wired interface, or a wireless interface. The interface 110 can facilitate converting or formatting data from one format to another format. For example, the interface 110 can include an application programming interface, which includes definitions for communication between various components such as software components.

[0041] The data processing system 102 can include an application, script, or program installed at the client computing device 104, such as an app that transmits an input audio signal to an interface 110 of the data processing system 102 and drives a component of the client computing device to render an output audio signal. The data processing system 102 can receive a data packet or other signal that contains or recognizes an audio input signal. For example, the data processing system 102 can execute or run an NLP component 112 to receive or obtain an audio signal and parse the audio signal. For example, the NLP component 112 can provide for interaction between a person and a computer. The NLP component 112 can be configured with a technology for understanding natural language and allowing the data processing system 102 to derive meaning from human or natural language input. The NLP component 112 can include or be configured with a machine learning-based technology, such as statistical machine learning. The NLP component 112 can parse the input audio signal using a decision tree, a statistical model, or a probability model. The NLP component 112 is capable of performing functions such as named entity recognition (e.g., given a stream of text, determining which items in the text map to proper names such as people or places, and what the type of each such name is, such as person, place, or organization), natural language generation (e.g., converting information from a computer database or semantic meaning into understandable human language), natural language understanding (e.g., converting text into a more formal representation, such as a first-order logic structure that a computer module can manipulate), machine translation (e.g., automatically translating text from one human language to another human language), morphological segmentation (e.g., separating words into individual morphemes, and identifying categories of morphemes, which can be challenging based on the morphological or structural complexity of the words of the language that should be taken into account), question answering (e.g., determining answers to human language questions, which can be specific or open-ended), and semantic processing (e.g., processing that may occur after a word is recognized and its meaning is encoded in order to associate the recognized word with other words of similar meaning).

[0042] The NLP component 112 converts the audio input signal into recognized text by comparing the input signal to a stored set of representative audio waveforms (e.g., in a data repository 124) and selecting the closest match. The set of audio waveforms can be stored in a data repository 124 or other database accessible to the data processing system 102. Representative waveforms are generated across a large group of users and can then be enhanced with speech samples from the users. After converting the audio signal into recognized text, the NLP component 112 matches the text to words that are associated with actions that the data processing system 102 can service, such as by training across users or by manual specification.

[0043] The audio input signal can be detected by a sensor 134 or transducer 136 (e.g., a microphone) of the client computing device 104. Via the transducer 136, the audio driver 138, or other components, the client computing device 104 can provide the audio input signal to the data processing system 102 (e.g., via the network 105), where it can be received (e.g., by the interface 110) and provided to the NLP component 112 or stored in the data repository 124.

[0044] The NLP component 112 can obtain an input audio signal. Based on the input audio signal, the NLP component 112 can identify at least one request or at least one trigger keyword corresponding to the request. The request can indicate the meaning or subject of the input audio signal. The trigger keyword can indicate the type of action that may be taken. For example, the NLP component 112 can parse the input audio signal to identify at least one request to leave home at night to attend dinner and a movie. The trigger keyword can contain at least one word, phrase, root or partial word, derivative that indicates the action to be taken. For example, the trigger keyword "go" or "to go" from the input audio signal can indicate that a means of transportation is needed. In this example, the input audio signal (or the identified request) does not directly indicate a transportation purpose, but the trigger keyword indicates that transportation is an auxiliary action to at least one other action indicated by the request.

[0045] The NLP component 112 can parse the input audio signal to identify, determine, retrieve or otherwise obtain the request and trigger keywords. For example, the NLP component 112 can apply semantic processing technology to the input audio signal to identify the trigger keyword or request. The NLP component 112 can apply semantic processing technology to the input audio signal to identify the trigger phrase containing one or more trigger keywords (such as, the first trigger keyword and the second trigger keyword). For example, the input audio signal can contain the sentence "I need someone to help me with laundry and dry cleaning". The NLP component 112 can apply semantic processing technology or other natural language processing technology to the data packet including the sentence to determine the trigger phrases "help me with laundry" and "help me with dry cleaning". The NLP component 112 can further identify multiple trigger keywords, such as laundry and dry cleaning. For example, the NLP component 112 can determine that the trigger phrase contains a trigger keyword and a second trigger keyword.

[0046] The NLP component 112 can filter the input audio signal to identify trigger keywords. For example, a data packet carrying the input audio signal can contain "It would be great if I could get someone that could help me go to the airport", in which case the NLP component 112 can filter out one or more terms as follows: "it", "would", "be", "great", "if", "I", "could", "get", "someone", "that", "could", or "help". By filtering out these terms, the NLP component 112 can more accurately and reliably identify trigger keywords, such as "go to the airport", and determine that this is a request for a taxi or ride-sharing service.

[0047] In some cases, the NLP component can determine that the data packet carrying the input audio signal contains one or more requests. For example, the input audio signal can contain the sentence "I need someone to help me with my laundry and dry cleaning." The NLP component 112 can determine that this is a request for laundry services and dry cleaning services. The NLP component 112 can determine that this is a single request for a service provider that can provide laundry services and dry cleaning services. The NLP component 112 can determine that these are two requests; a first request to a service provider that performs laundry services and a second request to a service provider that provides dry cleaning services. In some cases, the NLP component 112 can combine multiple determined requests into a single request and transmit the single request to the service provider device 108. In some cases, the NLP component 112 can transmit each request to the corresponding service provider device 108, or transmit the two requests to the same service provider device 108 separately.

[0048] The data processing system 102 can include a direct action API 116 that is designed and constructed to generate an action data structure responsive to the request based on the trigger keyword. The processor of the data processing system 102 can call the direct action API 116 to execute a script that generates a data structure to the service provider device 108 to request or order a service or product, such as a car from a car sharing service. The direct action API 116 can obtain data from the data repository 124, as well as receive data agreed to by the end user from the client computing device 104 to determine the location, time, user account, logistics or other information to allow the service provider device 108 to perform an operation, such as reserving a car from a car sharing service. Using the direct action API 116, the data processing system 102 can also communicate with the service provider device 108 to complete the conversion by reserving a car sharing ride in this example.

[0049] The direct action API 116 can perform a specified action to satisfy an end user's intent as determined by the data processing system 102. Based on the action specified in its input, the direct action API 116 can execute code or a dialog script that identifies the parameters necessary to satisfy the user's request. Such code can, for example, look up additional information such as the name of a home automation service in the data repository 124, or it can provide an audio output for rendering at the client computing device 104 to ask the end user a question such as the intended destination of the requested taxi. The direct action API 116 can determine the necessary parameters and can package that information into an action data structure, which can then be sent to another component (such as the content selector component 118) or to the service provider computing device 108 to be fulfilled.

[0050] Direct action API 116 can receive instructions or commands from NLP component 112 or other components of data processing system 102 to generate or construct action data structures. Direct action API 116 can determine the type of action so as to select a template from template repository 132 stored in data repository 124. The type of action can include, for example, services, products, reservations, or tickets. The type of operation can further include the type of service or product. For example, the type of service can include car sharing services, meal delivery services, laundry services, employment services, maintenance services, or home services. The type of product can include, for example, clothes, shoes, toys, electronic products, computers, books, or jewelry. The type of reservation can include, for example, dinner reservations or hair salon reservations. The type of ticketing can include, for example, movie tickets, sports venue tickets, or air tickets. In some cases, the type of service, product, reservation, or ticketing can be classified according to price, location, transportation type, availability, or other attributes.

[0051] The direct action API 116 can access the corresponding template from the template repository 132 after identifying the type of the request. The template can contain fields in a structured data set that can be filled by the direct action API 116 to facilitate the operation requested by the service provider device 108 (such as sending a taxi to pick up the end user at the pickup location and send the end user to the destination location). The direct action API 116 can perform a search in the template repository 132 to select a template that matches the trigger keyword and one or more features of the request. For example, if the request corresponds to a request for a car or ride to a destination, the data processing system 102 can select a car sharing service template. The car sharing service template can contain one or more of the following fields: device identifier, pickup location, destination location, number of passengers, or service type. The direct action API 116 can fill the field with a value. In order to fill the field with a value, the direct action API 116 can ping, poll, or otherwise obtain information from one or more sensors 134 of the computing device 104 or a user interface of the device 104. For example, the direct action API 116 can detect the source location using a location sensor such as a GPS sensor. The direct action API 116 can obtain further information by submitting an inquiry, prompt, or query to a terminal of a user of the computing device 104. The direct action API can submit the inquiry, prompt, or query via the interface 110 of the data processing system 102 and a user interface (e.g., an audio interface, a voice-based user interface, a display, or a touch screen) of the computing device 104. Thus, the direct action API 116 can select a template for an action data structure based on a trigger keyword or request, populate one or more fields in the template with information detected by one or more sensors 134 or obtained via a user interface, and generate, create, or otherwise construct an action data structure to facilitate execution of an operation by the service provider device 108.

[0052] The data processing system 102 is capable of selecting a template from the template data structure 132 based on various factors including, for example, one or more of a trigger keyword, a request, a third-party provider device 108, a type of third-party provider device 108, a category to which the third-party provider device 108 belongs (e.g., taxi service, laundry service, flower service, or food delivery), a location, or other sensor information.

[0053] To select a template based on a trigger keyword, the data processing system 102 (e.g., via the direct action API 116) can use the trigger keyword to perform a lookup or other query operation on the template database 132 to identify a template data structure that is mapped or otherwise corresponds to the trigger keyword. For example, each template in the template database 132 can be associated with one or more trigger keywords to indicate that the template is configured to generate an action data structure in response to the trigger keyword that the third-party provider device 108 can process to establish a communication session.

[0054] In some cases, the data processing system 102 can identify the third-party provider device 108 based on the trigger keyword. To identify the third-party provider device 108 based on the trigger keyword, the data processing system 102 can perform a lookup in the data repository 124 to identify the third-party provider device 108 mapped to the trigger keyword. For example, if the trigger keyword contains "ride" or "to go to", the data processing system 102 (e.g., via the direct action API 116) can identify the third-party provider device 108 as corresponding to Taxi Service Company A. The data processing system 102 can select a template from the template database 132 using the identified third-party provider device 108. For example, the template database 132 can contain a mapping or association between a third-party provider device 108 or entity and a template configured to generate an action data structure in response to a trigger keyword that the third-party provider device 108 can process to establish a communication session. In some cases, the template can be customized for the third-party provider device 108 or a category of the third-party provider device 108. The data processing system 102 can generate an action data structure based on a template for the third party provider 108 .

[0055] To construct or generate an action data structure, the data processing system 102 can identify one or more fields in the selected template to fill with values. The fields can be filled with numerical values, strings, Unicode values, Boolean logic, binary values, hexadecimal values, identifiers, location coordinates, geographic regions, timestamps, or other values. The fields or data structures themselves can be encrypted or shielded to keep the data secure.

[0056] After determining the fields in the template, the data processing system 102 can identify the values ​​of the fields to populate the fields of the template to create the action data structure. The data processing system 102 can obtain, retrieve, determine or otherwise identify the values ​​of the fields by performing a lookup or other query operation on the data repository 124.

[0057] In some cases, the data processing system 102 can determine that information or values ​​for fields do not exist in the data repository 124. The data processing system 102 can determine that the information or values ​​stored in the data repository 124 are outdated, stale, or otherwise unsuitable for the purpose of building an action data structure in response to the trigger keyword and request identified by the NLP component 112 (e.g., the location of the client computing device 104 may be an old location instead of the current location; an account may have expired; a destination restaurant may have moved to a new location; physical activity information; or transportation methods).

[0058] If data processing system 102 determines that it does not currently have a value or information to access a template field in the memory of data processing system 102, data processing system 102 can obtain the value or information. Data processing system 102 can obtain or obtain information by querying or polling one or more available sensors of client computing device 104, prompting an end user of client computing device 104 for the information, or accessing an online web-based resource using HTTP protocol. For example, data processing system 102 can determine that it does not have the current location of client computing device 104, which may be a required field of the template. Data processing system 102 can query client computing device 104 for location information. Data processing system 102 can request client computing device 104 to provide location information using one or more location sensors 134, such as global positioning system sensors, WIFI triangulation, cellular tower triangulation, Bluetooth beacons, IP addresses, or other location sensing technologies.

[0059] The direct action API 116 can transmit the action data structure to a third-party provider device (e.g., the service provider device 108) to cause the third-party provider device 108 to invoke a conversational application programming interface (e.g., the service provider NLP component 142) and establish a communication session between the third-party provider device 108 and the client computing device 104. In response to establishing the communication session between the service provider device 108 and the client computing device 104, the service provider device 108 can transmit the data packet directly to the client computing device 104 via the network 105. In some cases, the service provider device 108 can transmit the data packet to the client computing device 104 via the data processing system 102 and the network 105.

[0060] In some cases, the third party provider device 108 can execute at least a portion of the conversation API 142. For example, the third party provider device 108 can process certain aspects of the communication session or the type of query. The third party provider device 108 can utilize the NLP component 112 executed by the data processing system 102 to facilitate processing the audio signal associated with the communication session and generate a response to the query. In some cases, the data processing system 102 can include a conversation API 142 configured for the third party provider 108. In some cases, the data processing system routes data packets between the client computing device and the third party provider device to establish a communication session. The data processing system 102 can receive from the third party provider device 108 an indication that the third party provider device has established a communication session with the client device 104. The indication can include an identifier of the client computing device 104, a timestamp corresponding to when the communication session is established, or other information associated with the communication session, such as an action data structure associated with the communication session. In some cases, the data processing system 102 can include a session handler component 114 to manage the communication session and a feedback monitor component 120 to measure the characteristics of the communication session.

[0061] The data processing system 102 can include, execute, access, or otherwise communicate with the session handler component 114 to establish a communication session between the client device 104 and the data processing system 102. The communication session can refer to one or more data transmissions between the client device 104 and the data processing system 102, including input audio signals detected by the sensor 134 of the client device 104, and output signals transmitted by the data processing system 102 to the client device 104. The data processing system 102 (e.g., via the session handler component 114) can establish the communication session in response to receiving the input audio signal. The data processing system 102 can set a duration for the communication session. The data processing system 102 can set a timer or counter for the duration set for the communication session. In response to the expiration of the timer, the data processing system 102 can terminate the communication session.

[0062] A communication session can refer to a network-based communication session in which the client device 104 provides authentication information or credentials to establish a session. In some cases, a communication session refers to the topic or context of audio signals carried by data packets during a session. For example, a first communication session can refer to audio signals transmitted between the client device 104 and the data processing system 102 that are related to (e.g., containing keywords, action data structures, or content item objects) taxi services; and a second communication session can refer to audio signals transmitted between the client device 104 and the data processing system 102 that are related to laundry and dry cleaning services. In this example, the data processing system 102 can determine that the contexts of the audio signals are different (e.g., via the NLP component 112) and separate the two groups of audio signals into different communication sessions. The session handler 114 can terminate the first session related to the ride service in response to identifying one or more audio signals related to the dry cleaning and laundry services. Therefore, in response to detecting the context of the audio signals, the data processing system 102 can initiate or establish a second session of audio signals related to the dry cleaning and laundry services.

[0063] The data processing system 102 can include, execute, or otherwise communicate with a content selector component 118 to receive trigger keywords identified by a natural language processor and select content items via a real-time content selection process based on the trigger keywords. In some cases, the direct action API 116 can transmit an action data structure to the content selector component 118 to perform a real-time content selection process and establish a communication session between the content provider device 106 (or third-party provider device 108) and the client computing device 104.

[0064] The content selection process can refer to or include selecting a sponsored content item object provided by a third-party content provider 106. The content selection process can include such a service: wherein the content items provided by multiple content providers are parsed, processed, weighted, or matched so as to select one or more content items to be provided to the computing device 104. The content selection process can be performed in real time or offline. Performing the content selection process in real time can refer to performing the content selection process in response to a request for content received via the client computing device 104. The real-time content selection process can be performed (e.g., initiated or completed) within a time interval (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, or 20 minutes) of receiving the request. The real-time content selection process can be performed during a communication session with the client computing device 104 or within a time interval after the communication session is terminated.

[0065] For example, the data processing system 102 can include a content selector component 118 designed, constructed, configured or operated to select content item objects. In order to select content items to be displayed in a speech-based environment, the data processing system 102 (e.g., via the NLP component 112) can parse the input audio signal to identify keywords (e.g., trigger keywords), and use the keywords to select matching content items based on broad matching, complete matching, or phrase matching. For example, the content selector component 118 can analyze, parse or otherwise process the subject of the candidate content item to determine whether the subject of the candidate content item corresponds to the subject of the keyword or phrase of the input audio signal detected by the microphone of the client computing device 104. The content selector component 118 can use image processing technology, character recognition technology, natural language processing technology, or database query, etc. to identify, analyze, or recognize the voice, audio, word, character, text, symbol, or image of the candidate content item. The candidate content item can include metadata indicating the subject of the candidate content item, in which case the content selector component 118 can process the metadata to determine whether the subject of the candidate content item corresponds to the input audio signal.

[0066] The content provider 106 can provide additional indicators when setting up a content campaign that includes a content item. The content provider 106 can provide information at the content campaign or content group level that the content selector component 118 can identify by performing a search using information about the candidate content items. For example, the candidate content item can contain a unique identifier that can be mapped to a content group, a content campaign, or a content provider. The content selector component 118 can determine information about the content provider 106 based on information stored in a content campaign data structure in the data repository 124.

[0067] Data processing system 102 can receive a request for content for presentation on computing device 104 via a computer network. Data processing system 102 can identify the request by processing an input audio signal detected by a microphone of client computing device 104. The request can contain selection criteria for the request, such as a type of device, location, and keywords associated with the request. The request can contain an action data structure or action data structures.

[0068] In response to the request, data processing system 102 can select a content item object from data repository 124 or a database associated with content provider 106, and provide the content item via network 105 for presentation via computing device 104. The content item object can be provided by a content provider device 108 that is different from service provider device 108. The content item can correspond to a type of service that is different from the service type of the action data structure (e.g., taxi service vs. food delivery service). Computing device 104 can interact with the content item object. Computing device 104 can receive an audio response to the content item. Computing device 104 can receive an indication to select a hyperlink or other button associated with the content item object, the hyperlink or other button enabling or allowing computing device 104 to identify service provider 108, request a service from service provider 108, instruct service provider 108 to perform a service, transmit information to service provider 108, or otherwise query service provider device 108.

[0069] The data processing system 102 can include, execute, or communicate with the audio signal generator component 122 to generate an output signal. The output signal can include one or more parts. For example, the output signal can include a first part and a second part. The first part of the output signal can correspond to the action data structure. The second part of the output signal can correspond to the content item selected by the content selector component 118 during the real-time content selection process.

[0070] The audio signal generator component 122 can generate an output signal using a first portion having a sound associated with a first data structure. For example, the audio signal generator component 122 can generate a first portion of the output signal based on one or more values ​​populated into a field of the action data structure by the direct action API 116. In the taxi service example, the values ​​of the fields can include, for example, 123rd Street for the pickup location, 1234th Street for the destination location, 2 for the number of passengers, and an economy for the level of service. The audio signal generator component 122 can generate a first portion of the output signal to confirm that the end user of the computing device 104 wants to continue transmitting the request to the service provider 108. The first portion can include the following output "Do you want to order an economy car from taxi service provider A, pick up two people at 123rd Street, and drop them off at 1234th Street?"

[0071] In some cases, the first part can include information received from the service provider device 108. The information received from the service provider device 108 can be customized or adjusted for the action data structure. For example, the data processing system 102 (e.g., via the direct action API 116) can transmit the action data structure to the service provider 108 before instructing the service provider 108 to perform the operation. On the contrary, the data processing system 102 can instruct the service provider device 108 to perform initial or preliminary processing on the action data structure to generate preliminary information about the operation. In the example of a taxi service, the preliminary processing of the action data structure can include identifying available taxis that meet the level of service demand around the pickup location, estimating the amount of time for the nearest available taxi to arrive at the pickup location, estimating the time to arrive at the destination, and estimating the price of the taxi service. The estimated preliminary value can include a fixed value, an estimate that may vary based on various conditions, or a range of values. The service provider device 108 can return the preliminary information to the data processing system 102 or send it directly to the client computing device 104 via the network 104. The data processing system 102 can incorporate the preliminary results from the service provider device 108 into an output signal and transmit the output signal to the computing device 104. The output signal can include, for example, "Taxi Service Company A can pick you up at 123rd Street in 10 minutes and drop you off at 1234th Street at 9:00 AM for $10. Do you want to order this ride?" This can form the first part of the output signal.

[0072] In some cases, the data processing system 102 can form a second portion of the output signal. The second portion of the output signal can contain content items selected by the content selector component 118 during the real-time content selection process. The first portion can be different from the second portion. For example, the first portion can contain information corresponding to an action data structure, which directly responds to a data packet carrying an input audio signal detected by the sensor 134 of the client computing device 104, while the second portion can contain content items selected by the content selector component 104 that can be tangentially related to the action data structure, or contain sponsored content provided by the content provider device 106. For example, an end user of the computing device 104 can request a taxi from taxi service company A. The data processing system 102 can generate a first portion of the output signal to contain information about taxis from taxi service company A. However, the data processing system 102 can generate a second portion of the output signal to contain content items selected based on the keyword "taxi service" and information contained in the action data structure that the end user may be interested in. For example, the second portion can contain content items or information provided by different taxi service companies (such as, taxi service company B). Because the user may select to perform an operation with taxi service company B, data processing system 102 may still provide content items from taxi service company B, even though the user may not have specifically requested taxi service company B.

[0073] Data processing system 102 can transmit information from the action data structure to Taxi Service Company B to determine the pickup time, the time to arrive at the destination, and the price of the ride. Data processing system 102 can receive this information and generate a second portion of the output signal as follows: "Taxi Service Company B can pick you up at 123 Main Street in 2 minutes and drop you off at 1234 Main Street at 8:52 AM for $15. Do you want to change this ride?" The end user of computing device 104 can then select a ride provided by Taxi Service Company A or a ride provided by Taxi Service Company B.

[0074] Before providing the sponsored content item corresponding to the service provided by taxi service company B in the second portion of the output signal, the data processing system 102 can notify the end-user computing device that the second portion corresponds to the content item object selected during the real-time content selection process (e.g., by the content selector component 118). However, the data processing system 102 can have limited access to different types of interfaces to provide notifications to the end user of the computing device 104. For example, the computing device 104 may not include a display device, or the display device may be disabled or turned off. The display device of the computing device 104 may consume more resources than the speaker of the computing device 104, so turning on the display device of the computing device 104 may be less efficient in transmitting notifications than using the speaker of the computing device 104. Therefore, in some cases, the data processing system 102 can improve the efficiency and effectiveness of information transmission through one or more interfaces or one or more types of computer networks. For example, the data processing system 102 (e.g., via the audio signal generator component 122) can modularize the portion of the output audio signal including the content item to provide an indication or notification to the end user that the portion of the output signal includes the sponsored content item.

[0075] The data processing system 102 (e.g., via the interface 110 and the network 105) can transmit a data packet including an output signal generated by the audio signal generator component 122. The output signal can cause the audio driver component 138 of the client device 104 or the audio driver component 138 executed by the client device 104 to drive a speaker (e.g., the transducer 136) of the client device 104 to generate a sound wave corresponding to the output signal.

[0076] The data processing system 102 can include a feedback monitor component 120. The feedback monitor component 120 can include hardware or software for measuring the characteristics of a communication session. The feedback monitor component 120 can receive a data packet carrying an auditory signal transmitted between a client device (e.g., a computing device 104) and a conversation application programming interface (e.g., an NLP component 112 executed by a data processing system, or a service provider NLP component 142 executed by a service provider device 108, a third-party provider device, or a content provider device 106), and the conversation application programming interface establishes a communication session with the client device in response to interaction with a content item. In some cases, the content provider device 106 can execute an NLP component including one or more functions or components of the service provider NLP component 142 or the NLP component 112. The NLP component executed by the service provider device 108 or the content provider device 106 can be customized for the service provider device 108 or the content provider device 106. By customizing the NLP component, the NLP component can reduce bandwidth usage and request responses when compared to a generic or standard NLP component because the NLP component can be configured with more precise queries and responses, resulting in reduced round trips between the NLP component and the client computing device 104 .

[0077] The feedback monitor component 120 can measure characteristics of the communication session based on the auditory signal. The feedback monitor component 120 can generate a quality signal based on the measured characteristics. The quality signal can include or refer to a quality level, a quality metric, a quality score, or a quality level. The quality signal can include, for example, a numerical score (e.g., 0 to 10, where 0 is the lowest quality and 10 is the highest quality, and vice versa), a letter rating (e.g., A to F, A is the best quality), a binary value (e.g., yes / no; good / bad; 1 / 0; high / low), a grade, or a percentage. The quality signal can include an average quality signal determined from communications between multiple client devices communicating with the same NLP component or provider device 106 or 108.

[0078] Feedback monitor component 120 can use various measurement techniques, heuristic techniques, strategies, conditions, or tests to measure the characteristics of the communication session. Feedback monitor component 120 can parse the data packets transmitted between client device 104 and content provider equipment, third-party equipment, service provider, or data processing system to determine the characteristics of the communication session. Quality can refer to the quality of the communication channel used to transmit data or the quality of the data being transmitted. For example, the quality of the communication channel can refer to signal-to-noise ratio, ambient noise level, delay, delay, latency, discontinuity, echo, or dropped calls. The quality of the data being transmitted can refer to the quality of the response generated by the NLP component to the audio signal detected by the microphone of the computing device in response. The quality of the data can be based on the responsiveness of the NLP component, the accuracy of the NLP component, or the waiting time between the NLP component receiving the audio signal or query and the transmission response from the client device 104.

[0079] The feedback monitor component 120 can determine the quality of the communication channel by measuring the amount of background noise and the signal level to determine the signal-to-noise ("SNR") ratio. The feedback monitor component 120 can compare the measured or determined SNR to a threshold to determine the quality level. For example, an SNR of 10 dB may be considered good. The threshold can be predetermined or determined via a machine learning model (e.g., based on feedback from multiple devices).

[0080] The feedback monitor component 120 can further determine the quality of the communication channel based on the ping time between the client device 104 and the provider device or data processing system. The data processing system can compare the ping time and the threshold to determine the quality level. For example, the ping threshold can be 20ms, 30ms, 50ms, 100ms, 200ms or more. The feedback monitor component 120 can determine the quality of the communication channel based on the discontinuity of the audio (e.g., a pause or interruption in the audio; audio cutoff). The feedback monitor component 120 can identify echoes in the communication channel to determine a low quality level. The feedback monitor component 120 can determine the number of dropped calls of the NLP component during the time interval or the ratio of dropped calls to all calls, and compare it with the threshold to determine the quality level. For example, the threshold can be 2 dropped calls per hour; or 1 dropped call per 100 calls.

[0081] The feedback monitor component 120 can determine the quality of the communication session based on the quality of the response generated by the NLP component (or the conversation API) in communication with the client computing device 104. The quality of the response can include or be based on, for example, the amount of time it took the NLP component to generate the response, the text of the response, the accuracy of the response, the relevance of the response, the semantic analysis of the response, or the network activity of the client device in response to the response provided by the NLP component. The feedback monitor component 120 can determine the amount of time it took the NLP component to generate the response by distinguishing between a timestamp corresponding to when the NLP component received an audio signal from the client device 104 and a timestamp corresponding to when the NLP transmitted the response. The feedback monitor component 120 can determine the amount of time by distinguishing between a timestamp corresponding to when the client device transmitted the audio signal and a timestamp corresponding to when the client device received the response from the NLP component.

[0082] Feedback monitor component 120 can determine the quality of response by parsing the data packet including the response. For example, feedback monitor component 120 can parse and analyze the text of response, the accuracy of response, or the relevance of the response to the query from the client device. Feedback monitor component 120 can perform this evaluation by providing the query to another NLP component and comparing the responses from the two NLP components. Feedback monitor component 120 can perform this evaluation by providing the query and response to a third-party assessor. Feedback monitor component 120 can determine the consistency of response by comparing multiple responses with multiple similar queries provided by multiple client devices. Feedback monitor component 120 can determine the quality of response based on the number of times (for example, indicating that the response is not yet fully responded to the query submitted by the client device) that the audio signal including the same query is transmitted by the client device.

[0083] The feedback monitor component 120 can determine the quality of the response generated by the NLP based on the network activity of the client device. For example, the NLP component can receive a voice query from the client device, generate a response to the voice query, and transmit a data packet carrying the response to the client device. The client device can perform network activities or change network activities after receiving the response from the NLP component. For example, the client device can terminate the communication session, which can indicate that the NLP component fully responds to the client device, or that the NLP fails to successfully respond to the client device and the client device abandons the NLP component. The feedback monitor component can determine that the client device terminates the call due to good or bad reasons based on the confidence score associated with the response generated by the NLP component. The confidence score can be associated with the probability or statistical semantic analysis used to generate the response.

[0084] Feedback monitor component 120 can determine that client device terminates communication session based on the absence of audio signal transmitted by client device. Feedback monitor component 120 can determine that client device terminates communication session based on termination or end command transmitted by client device. Feedback monitor component 120 can determine quality level based on the quiet volume (e.g., absence of audio signal) from client device. Can identify the absence of audio signal based on SNR from client device less than threshold value (e.g., 6dB, 3dB or 0dB). Feedback monitor component can measure characteristic based on the duration of communication session. For example, duration greater than threshold value can indicate that terminal user of client device is satisfied with communication session. However, because user of client may have spent unnecessary or unnecessary extended time amount related to communication, long duration combined with other characteristics such as amplitude increase of audio signal, inquiry repetition and rhythm slowing down can indicate low quality session.

[0085] The NLP component can perform semantic analysis on queries transmitted by the client device to determine that the client device repeatedly transmits the same or similar queries even if the NLP component generates and provides responses. The feedback monitor component 120 can determine that the quality level is low based on the number of repeated queries (or sequentially repeated queries) within a time interval exceeding a threshold value (e.g., 2, 3, 4, 5, 6, 7, or more).

[0086] In some cases, the feedback monitor component 120 can determine the quality of the communication session at different parts of the communication session (e.g., the beginning, the middle, or the end; or time intervals). For example, the feedback monitor component 120 can determine the quality of a first part or a first time interval of the communication session; and the quality of a second part or a second time interval in the communication session after the first part or the first time interval. The feedback monitor component 120 can compare the quality at the two parts to determine the quality of the entire communication session. For example, a difference in quality between the two parts greater than a threshold can indicate low quality, inconsistent quality, or unreliable quality.

[0087] In some cases, the feedback monitor component 120 can determine the quality based on the characteristics of the communication session or at least a portion thereof. The characteristics can include at least one of, for example, amplitude, frequency, rhythm, pitch, and pitch. For example, the feedback monitor component 120 can use the characteristics to determine the reaction of the user of the client device or the emotion of using the client. For example, if the amplitude of the audio signal transmitted by the client device increases after each response from the NLP, the feedback monitor can determine that the end user is disappointed with the response generated by the NLP component. The feedback monitor component 120 can compare the amplitude of the audio signal detected by the client device with a threshold value or with other audio signals received by the client device during the same communication session or different communication sessions.

[0088] The feedback monitor component 120 can determine the quality based on characteristics such as the rhythm or pitch of the audio signal detected by the client device and transmitted to the NLP component. For example, the feedback monitor component 120 can determine that, for example, a slowing down of the rhythm (e.g., the rate of words spoken per time interval) after each NLP response can indicate that the end user is dissatisfied with the response generated by the NLP component, and slowly repeats to allow the NLP component to better parse the audio signal and improve the response. In some cases, an increased or stable rhythm can indicate that the user of the client device is satisfied with the response generated by the NLP and trusts the response. In some cases, an increase in the pitch of the audio signal detected by the client device can indicate poor quality of the response from the NLP or a lack of confidence in these responses.

[0089] In some cases, the feedback monitor component 120 can transmit a query to the client device to measure or determine quality. For example, the feedback monitor component 120 can transmit a survey question to the end user, asking about the quality of the communication session, the NLP component, or the provider device. In some cases, in response to the feedback monitor component 120 determining that the first quality signal is below a threshold, the feedback monitor component 120 can generate a query. For example, the feedback monitor component 120 can measure the quality based on the use of the characteristics such as the amplitude increase of the audio signal detected by the client device combined with the slowed-down rhythm of the audio signal detected by the client device to determine the first quality signal. The feedback monitor component 120 can generate a quality signal indicating a low quality level based on the combined characteristics of amplitude and rhythm. In response to the low-quality signal determined based on the combined characteristics, the feedback monitor component 120 can generate and transmit a query to the client device, implicitly or explicitly asking about the quality of the communication session (for example, are you satisfied with the response generated by the NLP component? Are you satisfied with the communication session?). In another example, the data processing system can determine the quality based on whether the service provider 108 can provide the requested service. For example, an end user may request a product or service, but service provider 108 responds stating that they do not have the product or cannot perform the service, which may cause the end user to indicate frustration with service provider 108. Data processing system 102 can recognize such frustration and assign quality accordingly.

[0090] In some cases, the feedback monitor component 120 can measure characteristics based on network activity on multiple electronic surfaces, and aggregate the qualities measured from the multiple electronic surfaces to generate a total quality signal. The total quality signal can be an average, a weighted average, an absolute sum, or other combined quality signal values. The feedback monitor component 120 can also generate statistics of the combined quality signal values ​​or perform statistical analysis, such as determining standard deviation, variance, 3 sigma quality, or 6 sigma quality.

[0091] The feedback monitor component 120 can adjust the real-time content selection process performed by the content selector component 118. Adjusting the real-time content selection process can refer to adjusting the weights used to select content items provided by the content provider device 106 or the service provider device 108 or a third-party provider device 108 executing an NLP component for establishing a communication session with the client device 104. For example, if a content item results in a low-quality communication session, the feedback monitor component 120 can adjust the attributes or parameters of the content data 130 comprising the content item to reduce the likelihood of selecting the content item for similar action data structures or similar client devices 104 (or their accounts or profiles).

[0092] In some cases, the feedback monitor component 120 can prevent or block the content selector component 118 from selecting a content item in the real-time selection process in response to the quality signal being less than a threshold. In some cases, the feedback monitor component 120 can allow or permit the content selector component 118 to select a content item in the real-time selection process in response to the quality signal being greater than or equal to the threshold.

[0093] Figure 2 is a diagrammatic representation of the operation of a feedback control system for data transmission over a computer network. The system can include Figure 1 1. The system 100 can include one or more components of the system 100 depicted in FIG. The system 100 can include one or more electronic surfaces 202a-n executed or provided by one or more client computing devices 104a-n. Examples of electronic surfaces 202a-n can include an audio interface, a voice-based interface, a display screen, an HTML content item, multimedia, an image, a video, a text-based content item, SMS, a messaging application, a chat application, or a natural language processor.

[0094] At act 204, the client computing device 104 can receive a data packet, signal, or other information indicative of feedback from or via the electronic surface 202. At act 206, one or more of the client computing devices 104a-n, one or more of the service provider devices 108a-n, or one or more of the content provider devices 106a-n can transmit the data packet to the feedback monitor component 124. The data packet can be associated with a communication session established between the client device 104 and one or more of the service provider device 108 or the content provider device 106. The data packet can be transmitted from the respective device to the feedback monitor component 124.

[0095] In some cases, the feedback monitor component 124 can intercept data packets transmitted from the device 104, 106, or 108 to the corresponding device. The feedback monitor component 124 can analyze the intercepted data packets and route or forward the data packets to their intended destination. Therefore, the feedback monitor component 124 can be an intermediary to the client device 104 and the service / third-party provider device 108 or content provider device 106.

[0096] At act 208, the feedback monitor component 124 can transmit the data packets intercepted or received from the communication session to the NLP component 112. At act 210, the NLP component 112 can perform semantic analysis of the data packets and provide them back to the feedback component 124. In some cases, the NLP component 112 can perform natural language processing on the audio signals from the communication session 206 to compare the responses of the NLP component generated by the provider device 106 or 108. The feedback monitor component 124 can compare the responses generated by the controlling NLP component 112 to determine whether the third-party NLP component is operating at a comparable or satisfactory level.

[0097] At action 212, the feedback monitor component 124 can determine a quality signal of the communication session 206 and adjust the real-time content selection process performed by the content selector component 118 so that the next time the content selector component 118 receives a request for content, the content selector component 118 can appropriately weight the content items (or content providers) associated with the communication session 206 to increase or decrease the likelihood of the content item being selected. For example, if the provider 108 is associated with multiple low-quality communication sessions, the feedback monitor component 124 can instruct the content selector component 118 to prevent the selection of content items that can lead to the establishment of a communication session with the provider 108.

[0098] Figure 3 300 is an illustration of an example method for performing dynamic modulation of a packetized audio signal. Method 300 can be performed by one or more components, systems, or elements of system 100 or system 400. Method 300 can include a data processing system that receives an input audio signal (act 305). The data processing system can receive the input audio signal from a client computing device. For example, a natural language processor component executed by the data processing system can receive the input audio signal from the client computing device via an interface of the data processing system. The data processing system can receive a data packet carrying or containing the input audio signal detected by a sensor of a client computing device (or client device).

[0099] At action 310, method 300 can include a data processing system parsing an input audio signal. A natural language processor component can parse the input audio signal to identify a request and a trigger keyword corresponding to the request. For example, an audio signal detected by a client device can include "Okay device, I need a ride from Taxi Service Company A to go to 1234 Main Street". In this audio signal, an initial trigger keyword can include "Okay device", which can indicate to the client device that the input audio signal is transmitted to the data processing system. A preprocessor of the client device can filter out the term "Okay device" before sending the rest of the audio signal to the data processing system. In some cases, the client device can filter out additional terms or generate keywords to transmit to the data processing system for further processing.

[0100] The data processing system can identify trigger keywords in the input audio signal. The trigger keywords can include, for example, "to go to" or "ride" or variations of these terms. The trigger keywords can indicate the type of service or product. The data processing system can identify a request in the input audio signal. The request can be determined based on the term "I need". The trigger keywords and the request can be determined using semantic processing techniques or other natural language processing techniques.

[0101] In some cases, the data processing system can generate an action data structure. The data processing system can generate the action data structure based on a trigger keyword, a request, a third-party provider device, or other information. The action data structure can respond to a request. For example, if an end user of a client computing device requests a taxi from taxi service company A, the action data structure can contain information for requesting a taxi service from taxi service company A. The data processing system can select a template for taxi service company A and fill in the fields in the template with values ​​to allow taxi service company A to dispatch a taxi for the user of the client computing device to pick up the user and deliver the user to the requested destination.

[0102] At action 315, the data processing system can select content items. For example, the content selector component can receive trigger keywords, request or action data structure, and select content items via real-time content selection process. The selected content items can correspond to content providers, service providers or other third-party providers. Client devices can interact with content items to establish a communication session with the provider of content items or other devices associated with content items. Devices associated with content items can use conversation APIs (such as, NLP) to interact with client devices.

[0103] At act 320, the feedback monitor component can receive a data packet carrying an auditory signal transmitted between a client device and a dialog application programming interface that establishes a communication session with the client device in response to interaction with the content item. At act 325, the feedback monitor component can measure a quality or characteristic of the communication session based on the auditory signal and generate a quality signal based on the measured characteristic. At act 330, the feedback monitor component or the data processing system can adjust the real-time selection process based on the quality signal.

[0104] Figure 4 4 is a block diagram of an example computer system 400. The computer system or computing device 400 can include or be used to implement the system 100 or a component thereof such as the data processing system 102. The data processing system 102 can include an intelligent personal assistant or a voice-based digital assistant. The computing system 400 includes a bus 405 or other communication component for transmitting information and a processor 410 or processing circuit coupled to the bus 405 to process the information. The computing system 400 can also include one or more processors 410 or processing circuits coupled to the bus for processing information. The computing system 400 also includes a main memory 415 such as a random access memory (RAM) or other dynamic storage device coupled to the bus 405 for storing information and instructions to be executed by the processor 410. The main memory 415 can be or include the data repository 145. The main memory 415 can also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 410. The computing system 400 may further include a read-only memory (ROM) 420 or other static storage device coupled to the bus 405 for storing static information and instructions for the processor 410. A storage device 425, such as a solid-state device, magnetic disk, or optical disk, can be coupled to the bus 405 to permanently store information and instructions. The storage device 425 can include or be part of the data repository 145.

[0105] Computing system 400 may be coupled to a display 435, such as a liquid crystal display or an active matrix display, via bus 405 for displaying information to a user. An input device 430, such as a keyboard including alphanumeric and other keys, may be coupled to bus 405 for communicating information and command selections to processor 410. Input device 430 can include a touch screen display 435. Input device 430 can also include a cursor control, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 410 and for controlling cursor movement on display 435. For example, display 435 can be a display of data processing system 102, client computing device 150, or Figure 1 parts of other components.

[0106] The processes, systems, and methods described herein can be implemented by the computing system 400 in response to the processor 410 executing an arrangement of instructions contained in the main memory 415. Such instructions can be read into the main memory 415 from another computer-readable medium, such as the storage device 425. The arrangement of execution instructions contained in the main memory 415 causes the computing system 400 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement can also be employed to execute the instructions contained in the main memory 415. With the systems and methods described herein, hard-wired circuits can be used in place of or in conjunction with software instructions. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0107] Although in Figure 4 An example computing system has been described in the specification, but the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of the foregoing.

[0108] For the situation that the system discussed herein collects personal information about the user or can utilize personal information, the user can be provided with the opportunity to control the program or feature that can collect personal information (for example, information about the user's social network, social action or activity, the user's preference, or the user's location), or control whether or how to receive the content from the content server or other data processing system that can be more relevant to the user. In addition, some data may be anonymized in one or more ways before storage or use, so that personal identification information is removed when generating parameters. For example, the identity of the user can be anonymous, so that the user's personal identifiable information cannot be determined, or the user's geographical location can be generalized when obtaining location information (such as city, zip code, or state level), so that the specific location of the user cannot be determined. Therefore, the user can have control over how to collect information about him or her and used by the content server.

[0109] The subject matter and operations described in this specification can be implemented in digital electronic circuits or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of the above. The subject matter described in this specification can be implemented as one or more computer programs, such as one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal generated to encode information, such as a machine-generated electrical signal, optical signal, or electromagnetic signal, to be transmitted to a suitable receiver device for execution by a data processing device. The computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof, or can be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. Although a computer storage medium is not a propagation signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagation signal. The computer storage medium can also be or be included in one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0110] The terms "data processing system," "computing device," "component," or "data processing apparatus" encompass various apparatuses, devices, and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or multiple or combinations of the foregoing. The apparatus can include dedicated logic circuits, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In addition to hardware, the apparatus can also include code that creates an execution environment for the computer program involved, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as Web services, distributed computing, and grid computing infrastructures. For example, the direct action API 116, the content selector component 118, or the NLP component 112 and other components of the data processing system 102 can include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0111] A computer program (also referred to as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program suitable for a computing environment or as a module, component, subroutine, object, or other unit. A computer program can correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0112] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs (e.g., components of data processing system 102) to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry (e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC)). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0113] The subject matter described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an embodiment of the subject matter described in this specification), or a combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0114] A computing system such as system 100 or system 400 can include a client and a server. The client and the server are usually far away from each other and usually interact through a communication network (e.g., network 165). The relationship between the client and the server is generated by a computer program that runs on each computer and has a client-server relationship with each other. In some embodiments, the server transmits data (e.g., a data packet representing a content item) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). The data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server (e.g., received by the data processing system 102 from the computing device 150 or the content provider computing device 155 or the service provider computing device 160).

[0115] Although operations are described in the drawings in a particular order, these operations need not be performed in the particular order shown or in sequential order, and not all of the operations shown need be performed. The actions described herein can be performed in a different order.

[0116] The separation of various system components need not be separated in all embodiments, and the described program components can be included in a single hardware or software product. For example, the NLP component 110 or the content selector component 125 can be a single component, app, or program, or a logical device having one or more processing circuits, or part of one or more servers of the data processing system 102.

[0117] Now that some illustrative embodiments have been described, it is apparent that the foregoing is illustrative rather than restrictive and has been presented by way of example. Specifically, although many of the examples presented herein relate to specific combinations of method actions or system elements, these actions and elements may be combined in other ways to achieve the same goals. The actions, elements, and features discussed in conjunction with one embodiment are not intended to exclude similar effects in other one or more embodiments.

[0118] The words and terms used herein are for descriptive purposes and should not be considered limiting. "Comprising," "including," "having," "containing," "involving," "characterized by," "characterized by," and variations thereof are intended to encompass the items listed thereafter, their equivalents and additional items, and alternative embodiments consisting of the items specifically listed thereafter. In one embodiment, the systems and methods described herein include one, every combination of more than one, or all of the elements, actions, or components described.

[0119] The embodiment of the system and method mentioned in the singular mode herein or any quoting of element or action can also contain the embodiment comprising a plurality of these elements, and the plural quoting of any embodiment or element or action herein can also contain the embodiment comprising only one element. The quoting of singular form or plural form is not intended to limit the currently disclosed system or method, its assembly, action or element to single or multiple configurations. The quoting of any action or element based on any information, action or element can include the embodiment wherein action or element is based at least in part on any information, action or element.

[0120] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment," "some embodiments," "an embodiment," etc. are not necessarily mutually exclusive, but are intended to indicate that a particular feature, structure, or characteristic described in conjunction with an embodiment can be included in at least one embodiment or example. Such terms used herein do not necessarily all refer to the same embodiment. Any embodiment can be combined, inclusively or exclusively, with any other embodiment in any manner consistent with the aspects and embodiments disclosed herein.

[0121] References to "or" can be interpreted as inclusive, so any term described using "or" can indicate any of a single, more than one, and all of the described terms. For example, reference to "at least one of 'A' and 'B'" can include only "A," only "B," and both "A" and "B." Such references used in conjunction with "including" or other public terms can include additional items.

[0122] Where reference numerals are attached to technical features in the drawings, detailed description or any claims, the reference numerals have been included to increase the intelligibility of the drawings, detailed description and claims. Accordingly, neither the reference numerals nor their absence will have any limiting effect on the scope of any claim element.

[0123] The system and method described herein can be implemented in other specific forms without departing from its characteristics. For example, data processing system 102 can select content items for subsequent actions (for example, 3rd action 215) based in part on the data of the previous action in the action sequence from thread 200 (such as, from the data indicating that the second action 210 is completed or about to start of the second action 210). The aforementioned embodiments are illustrative rather than limiting the described system and method. The scope of the system and method described herein is therefore indicated by the appended claims rather than the preceding description, and the changes in the meaning and scope of the equivalents falling into the claims are also included therein.

Claims

1. A system for data transmission over a computer network, comprising: A data processing system including one or more processors and memory; A natural language processor component executed by the data processing system for: receiving, via an interface of the data processing system, a data packet including an input audio signal detected by a sensor of a client device; establishing a communication session in response to the input audio signal; as well as parsing the input audio signal to identify a request and keywords associated with the request; Feedback monitor assembly for: determining a characteristic of the communication session based on the input audio signal; as well as generating a quality signal based on the characteristic; as well as A content selector executed by the data processing system for: receiving the keyword identified by the natural language processor component; adjusting a real-time content selection process based on the quality signal; as well as A content item is selected in response to the request and the keyword via the real-time content selection process adjusted based on the quality signal.

2. The system according to claim 1, comprising: the data processing system being configured to receive a second data packet carrying an auditory signal transmitted between the client device and a conversational application programming interface, the conversational application programming interface establishing a second communication session with the client device in response to interaction with the content item; as well as A second quality signal is determined based on a second characteristic associated with the second communication session.

3. The system according to claim 2, wherein: The dialog application programming interface is executed by a third party provider device or the data processing system.

4. The system according to claim 1, comprising: the feedback monitor component to forward a data packet carrying one or more input audio signals to the natural language processor component to determine a first characteristic of the one or more input audio signals at a first time interval and a second characteristic of the one or more input audio signals at a second time interval after the first time interval; as well as The characteristic is measured based on a comparison of the first characteristic and the second characteristic.

5. The system according to claim 1, wherein: The characteristic includes at least one of amplitude, frequency, rhythm, pitch, and pitch.

6. The system according to claim 1, wherein: The characteristic is indicative of a mood of a user associated with the client device.

7. The system according to claim 1, comprising the data processing system being configured to: determining the characteristic based on at least one of amplitude, frequency, rhythm, pitch, and pitch; and A mood of a user associated with the client device is determined based on the characteristic.

8. The system according to claim 1, comprising the data processing system being configured to: transmitting a plurality of voice-based queries to the client device; and The characteristic is measured based on responses to the plurality of speech-based queries.

9. The system according to claim 1, comprising the data processing system being configured to: generating a query based on the quality signal being less than a threshold; receiving a response to the query from the client device; and A second quality signal is generated based on the response.

10. The system according to claim 1, comprising: The data processing system is used to measure the characteristic for the duration of the communication session.

11. The system according to claim 1, comprising the data processing system being configured to: measuring the characteristic based on network activity on a plurality of electronic surfaces; and The mass signals measured from the plurality of electronic surfaces are aggregated to generate a summed mass signal.

12. The system according to claim 1, comprising: The data processing system is configured to prevent the content selector component from selecting content items during the communication session in response to the quality signal being less than a threshold.

13. The system according to claim 1, comprising: The data processing system enables the content selector component to select content items during the communication session in response to the quality signal being greater than or equal to a threshold.

14. The system according to claim 1, comprising: The feedback monitor component is for determining a user's reaction associated with the input audio signal based on the characteristic of the communication session.

15. A method for feedback control of data transmission over a computer network, comprising: receiving, by a data processing system including one or more processors, a data packet including an input audio signal detected by a sensor of a client device; establishing, by the data processing system, a communication session in response to the input audio signal; parsing, by the data processing system, the input audio signal to identify a request and a keyword associated with the request; determining, by the data processing system, a characteristic of the communication session based on the input audio signal; generating, by the data processing system, a quality signal based on the characteristic; adjusting, by the data processing system, a real-time content selection process based on the quality signal; as well as A content item is selected by the data processing system in response to the request and the keyword via the real-time content selection process adjusted based on the quality signal.

16. The method according to claim 15, comprising: receiving a second data packet carrying an auditory signal transmitted between the client device and a conversational application programming interface, the conversational application programming interface establishing a second communication session with the client device in response to interaction with the content item; as well as A second quality signal is determined based on a second characteristic associated with the second communication session.

17. The method according to claim 15, comprising: receiving a data packet carrying one or more input audio signals; determining a first characteristic of the one or more input audio signals at a first time interval and a second characteristic of the one or more input audio signals at a second time interval after the first time interval; as well as The characteristic is measured based on a comparison of the first characteristic and the second characteristic.

18. The method according to claim 15, wherein: The characteristic includes at least one of amplitude, frequency, rhythm, pitch, and pitch.

19. The method according to claim 15, wherein: The characteristic is indicative of a mood of a user associated with the client device.

20. The method according to claim 15, comprising: transmitting a plurality of voice-based queries to the client device; as well as The characteristic is measured based on responses to the plurality of speech-based queries.

Images