Graphical user interface rendering management via voice-driven computing infrastructure

Abstract

The present application discloses graphical user interface rendering management by voice-driven computing infrastructure. A system for managing the rendering of a graphical user interface is provided. A system receives a data packet including an input audio signal. The system determines an application identifier and a query. The system provides the query to the application so that the application generates a second query for transmission to a third-party server, and identifies a response to the query. The system intercepts the response and generates keywords based on the response. The system selects a digital component using the keyword, performs a de-duplication process and determines to add the digital component to the response. The system constructs a display output using a graphical user interface template that integrates multiple responses generated by the application with the digital component, and provides the display output to a computing device for rendering.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201880061283.7, with a filing date of June 15, 2018. Technical Field

[0003] The present disclosure relates to graphical user interface rendering management via a voice-driven computing infrastructure. Background Art

[0004] A computing device may receive data transmitted over a computer network. The device may present the data. Effectively presenting data from multiple sources may be challenging due to excessive data transmission. Summary of the invention

[0005] The present disclosure generally relates to graphical user interface rendering management performed by voice-driven computing infrastructure. Responses or outputs may be received from multiple, disparate sources. Each source may provide output in a format. Performing multiple renderings of outputs from different sources may be inefficient or processor-intensive.

[0006] At least one aspect relates to a system for managing rendering of a graphical user interface. The system may include a data processing system having one or more processors and a memory. The data processing system may execute a natural language processor component, an interface, an output merging component, and a content selector component. The data processing system may receive a data packet including an input audio signal detected by a sensor of a computing device via an interface. The data processing system may determine an application identifier and a query to be input into an application corresponding to the application identifier from the input audio signal. The data processing system may provide a query to the application so that the application generates a second query to be transmitted to a third-party server and identifies multiple responses to the query to be displayed via a computing device. The data processing system may obtain from the application multiple responses identified by the application in response to the second query transmitted to the third-party server before displaying on the computing device. The data processing system may parse the multiple responses to generate keywords based on the multiple responses. The data processing system may perform a real-time content selection process using the keywords generated based on the multiple responses obtained from the application in response to the generation of the keywords. The data processing system may identify a digital component provided by a content provider device based on the real-time content selection process. The data processing system may perform a deduplication process using the digital component and the multiple responses generated by the application. The data processing system may determine to add the digital component to the multiple responses in response to the deduplication process performed using the digital component and the multiple responses generated by the application. The data processing system may retrieve a graphical user interface template that defines fonts, colors, and layouts for the application to render the multiple responses. The data processing system may construct a display output using the graphical user interface template that integrates the multiple responses generated by the application with the digital component identified based on the real-time content selection process. The data processing system may provide the display output to a computing device so that the computing device renders the display output for display via a display device communicatively coupled to the computing device.

[0007] At least one aspect relates to a method for managing rendering of a graphical user interface. The method may be performed by a data processing system having one or more processors and a memory. The method may include the data processing system receiving, via an interface, a data packet including an input audio signal detected by a sensor of a computing device. The method may include the data processing system determining, by the data processing system, an application identifier and a query to be input into an application corresponding to the application identifier from the input audio signal. The method may include: the data processing system providing the query to the application so that the application generates a second query for transmission to a third-party server and identifies a plurality of responses to the query for display via the computing device. The method may include: the data processing system obtaining, from the application, a plurality of responses identified by the application in response to the second query transmitted to the third-party server before displaying on the computing device. The method may include the data processing system parsing the plurality of responses to generate keywords based on the plurality of responses. The method may include the data processing system, in response to the generation of the keywords, performing a real-time content selection process using the keywords generated based on the plurality of responses obtained from the application. The method may include the data processing system identifying, based on the real-time content selection process, a digital component provided by a content provider device. The method may include the data processing system performing a deduplication process using the digital component and the plurality of responses generated by the application. The method may include determining, by a data processing system, to add a digital component to a plurality of responses in response to a deduplication process performed using the digital component and a plurality of responses generated by an application. The method may include retrieving, by the data processing system, a graphical user interface template defining fonts, colors, and layouts for rendering the plurality of responses for the application. The method may include constructing a display output using the graphical user interface template by the data processing system, the graphical user interface template integrating the plurality of responses generated by the application with the digital component identified based on a real-time content selection process. The method may include providing, by the data processing system, the display output to a computing device, so that the computing device renders the display output for display via a display device communicatively coupled to the computing device.

[0008] At least one aspect relates to a digital assistant device. The digital assistant device may include a display device, a graphics driver, a sensor, and a preprocessor component. The sensor may detect an input audio signal. The preprocessor component may be coupled to the display device, the graphics driver, and the sensor. The preprocessor component may filter the input audio signal to create a filtered input audio signal. The preprocessor component may convert the filtered input audio signal into a data packet. The preprocessor component may transmit the data packet to a data processing system. The data processing system may have one or more processors and memories that execute a natural language processor component, an interface, an output merging component, and a content selector component. The data processing system may receive a data packet including an input audio signal detected by a sensor of a computing device via an interface. The data processing system may determine an application identifier and a query to be input into an application corresponding to the application identifier from the input audio signal. The data processing system may provide a query to the application so that the application generates a second query to be transmitted to a third-party server, and identifies multiple responses to the query to be displayed via the computing device. The data processing system may obtain multiple responses from the application in response to the second query transmitted to the third-party server before displaying on the computing device. The data processing system may parse multiple responses to generate keywords based on multiple responses. The data processing system may perform a real-time content selection process using keywords generated based on multiple responses obtained from the application in response to the generation of keywords. The data processing system may identify digital components provided by a content provider device based on the real-time content selection process. The data processing system may perform a deduplication process using digital components and multiple responses generated by the application. The data processing system may determine to add digital components to multiple responses in response to the deduplication process performed using digital components and multiple responses generated by the application. The data processing system may retrieve a graphical user interface template defining fonts, colors, and layouts for the application to render multiple responses. The data processing system may construct a display output using a graphical user interface template that integrates multiple responses generated by the application with digital components identified based on the real-time content selection process. The data processing system may provide the display output to a digital assistant device so that the digital assistant device renders the display output to be displayed via a display device communicatively coupled to the digital assistant device. The graphics driver of the digital assistant device may render the display output so that the display device presents a visual indication corresponding to the display output.

[0009] These and other aspects and embodiments are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of the various aspects and embodiments, and provide an overview or framework for understanding the nature and character of the claimed aspects and embodiments. The accompanying drawings provide illustration and further understanding of the various aspects and embodiments, and are incorporated into and constitute a part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings are not intended to be drawn to scale. Like reference numbers and names refer to like elements in the various drawings. For clarity, not every component is labeled in every drawing. In the drawings:

[0011] Figure 1 is an illustration of a system for managing rendering of a graphical user interface.

[0012] Figure 2 is an illustration of the operation of a system for managing rendering of a graphical user interface.

[0013] Figure 3 is an illustration of a graphical user interface provided by a system for managing rendering of graphical user interfaces.

[0014] Figure 4 is an illustration of a method for managing rendering of a graphical user interface.

[0015] Figure 5 is a block diagram illustrating the overall architecture of a computer system that may be used to implement elements of the systems and methods described and illustrated herein. DETAILED DESCRIPTION

[0016] Below is a more detailed description of various concepts related to methods, apparatuses, and systems for routing packetized actions via a computer network, and their implementation.The various concepts introduced above and discussed in more detail below may be implemented in any of numerous ways.

[0017] The present disclosure generally relates to managing the rendering of a graphical user interface. Responses or outputs may be received from multiple disparate sources. Each source may provide output in a format. Performing multiple renderings of outputs from different sources may be inefficient or processor-intensive.

[0018] The systems and methods of the present solution generally involve integrating a digital assistant with a third-party user experience. A computing system can provide digital components from multiple sources. However, it may be challenging to effectively present digital components from completely different sources via a single interface. The systems and methods of the present solution involve integrating a digital assistant with a third-party user experience. For example, the present disclosure may allow a digital assistant to be integrated with a chatbot application to provide a user experience and interface consistent with the interface of the chatbot, thereby providing a third-party look and feel within a digitally driven visual display. A chatbot application may refer to a computer program that simulates a conversation using audio output or displayed text. In some cases, a digital assistant may add content from other sources for presentation with third-party content. For example, a user may submit a voice query "Chatbot, suggest some restaurants nearby (chatbot, suggest some restaurants nearby)", and the digital assistant may respond using organic content from the chatbot and integrated sponsored digital components.

[0019] Figure 1 An example system 100 for managing the rendering of a graphical user interface is illustrated. The system 100 may include a content selection infrastructure. The system 100 may include a data processing system 102. The data processing system 102 may communicate with one or more of a content provider computing device 106, a chatbot provider device 108, or a client computing device 104 via a network 105. The network 105 may include a computer network such as the Internet, a local area network, a wide area network, a metropolitan area network, or other regional networks, an intranet, a satellite network, and other communication networks such as voice or data mobile phone networks. The network 105 may be used to access information resources such as web pages, websites, domain names, or uniform resource locators, which may be presented, output, rendered, or displayed on at least one computing device 104, such as a laptop computer, a desktop computer, a tablet computer, a digital assistant, a personal digital assistant, a smart watch, a wearable device, a smart phone, a portable computer, or a speaker. For example, via the network 105, a user of the computing device 104 may access information or data provided by a chatbot provider 108 or a content provider computing device 106. The computing device 104 may or may not include a display; for example, the computing device may include a limited type of user interface, such as a microphone and a speaker. In some cases, the primary user interface of the computing device 104 may be a microphone and a speaker.

[0020] Network 105 may include or constitute a display network, such as a subset of information resources available on the Internet that are associated with a content placement or search engine result system or that are eligible to include a third-party digital component as part of a digital component placement activity. Data processing system 102 may use network 105 to access information resources, such as web pages, websites, domain names, or uniform resource locators that may be presented, output, rendered, or displayed by client computing device 104. For example, via network 105, a user of client computing device 104 may access information or data provided by content provider computing device 106 or chatbot provider computing device 108.

[0021] The network 105 may be any type or form of network, and may include any of the following: a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an ATM (asynchronous transfer mode) network, a SONET (synchronous optical network) network, an SDH (synchronous digital hierarchy) network, a wireless network, and a wired network. The network 105 may include a wireless link, such as an infrared channel or a satellite band. The topology of the network 105 may include a bus, star, or ring network topology. The network may include a mobile phone network using any one or more protocols for communicating between mobile devices, including the Advanced Mobile Phone Protocol ("AMPS"), Time Division Multiple Access ("TDMA"), Code Division Multiple Access ("CDMA"), Global System for Mobile Communications ("GSM"), General Packet Radio Service ("GPRS"), or Universal Mobile Telecommunications System ("UMTS"). Different types of data may be transmitted via different protocols, or the same type of data may be transmitted via different protocols.

[0022] System 100 may include at least one data processing system 102. Data processing system 102 may include at least one logical device such as a computing device with a processor to communicate with, for example, computing device 104, content provider device 106 (content provider computing device 106), or chatbot provider device 108 (or chatbot provider 108) via network 105. Data processing system 102 may include at least one computing resource, server, processor, or memory. For example, data processing system 102 may include multiple computing resources or servers located in at least one data center. Data processing system 102 may include multiple servers in logical groupings and facilitate distributed computing techniques. Logical groups of servers may be referred to as data centers, server farms, or machine farms. Servers may also be geographically dispersed. A data center or machine farm may be managed as a single entity, or a machine farm may include multiple machine farms. Servers within each machine farm may be heterogeneous—one or more of the servers or machines may operate according to one or more types of operating system platforms.

[0023] The servers in the machine farm may be stored in a high-density rack system along with associated storage systems and may be located in an enterprise data center. Consolidating servers in this manner may improve system manageability, data security, system physical security, and system performance, for example, by locating servers and high-performance storage systems on a localized, high-performance network. Centralizing all or some of the data processing system 102 components, including servers and storage systems, and combining them with advanced system management tools allows for more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage.

[0024] The computing device 104 may include, interface with, or otherwise communicate with at least one display 132, at least one sensor 134, at least one transducer 136, at least one audio driver 138, or at least one pre-processor 140. The display 132 may include one or more hardware or software components configured to provide a visual indication or optical output, such as a light emitting diode, an organic light emitting diode, a liquid crystal display, a laser, or a display 535. The sensor 134 may include, for example, a camera, an ambient light sensor, a proximity sensor, a temperature sensor, an accelerometer, a gyroscope, a motion detector, a GPS sensor, a location sensor, a microphone, a video, an image detection, or a touch sensor. The transducer 136 may include or be part of a speaker or a microphone. The audio driver 138 may provide a software interface to the hardware transducer 136. The audio driver may execute an audio file or other instructions provided by the data processing system 102 to control the transducer 136 to generate corresponding sound waves or sound waves. The pre-processor 140 may include one or more processors (e.g., processor 510), a logic array, or a memory. The preprocessor 140 can detect keywords and perform actions based on the keywords. The preprocessor 140 can filter out one or more terms or modify the terms before transmitting the terms to the data processing system 102 for further processing. The preprocessor 140 can convert the analog audio signal detected by the microphone into a digital audio signal and transmit one or more data packets carrying the digital audio signal to the data processing system 102 via the network 105. In some cases, the preprocessor 140 can transmit data packets carrying some or all of the input audio signal in response to detecting an instruction for performing such transmission. The instruction can include, for example, a trigger keyword or other keyword, or can authorize the transmission of a data packet including the input audio signal to the data processing system 102.

[0025] The preprocessor 140 can perform pre-filtering or pre-processing on the input audio signal to remove audio of certain frequencies. Pre-filtering can include filters such as low-pass filters, high-pass filters, or band-pass filters. The filter can be applied in the frequency domain. The filter can be applied using digital signal processing techniques. The filter can be configured to retain frequencies corresponding to human speech or human speech while eliminating frequencies outside the typical frequencies of human speech. For example, the band-pass filter can be configured to remove frequencies below a first threshold (e.g., 70Hz, 75Hz, 80Hz, 85Hz, 90Hz, 95Hz, 100Hz, or 105Hz) and above a second threshold (e.g., 200Hz, 205Hz, 210Hz, 225Hz, 235Hz, 245Hz, or 255Hz). Applying a band-pass filter can reduce computing resource utilization in downstream processing. In some cases, the preprocessor 140 on the computing device 104 can apply a band-pass filter before transmitting the input audio signal to the data processing system 102, thereby reducing network bandwidth utilization. However, based on the computing resources available to computing device 104 and the available network bandwidth, it may be more efficient to provide the input audio signal to data processing system 102 to allow data processing system 102 to perform filtering.

[0026] Preprocessor 140 may apply additional preprocessing or prefiltering techniques, such as noise reduction techniques, to reduce ambient noise levels that may interfere with the natural language processor. Noise reduction techniques may improve the accuracy and speed of the natural language processor, thereby improving the performance of data processing system 102 and managing the rendering of a graphical user interface provided via display 132.

[0027] The client computing device 104 (or computing device 104) may be associated with an end user who inputs a voice query as an audio input (via sensor 134) into the client computing device 104 and receives an audio output in the form of a computer-generated speech, which may be provided from the data processing system 102 (or content provider computing device 106 or chatbot provider computing device 108) to the client computing device 104 and output from a transducer 136 (e.g., a speaker). The computer-generated speech may include a recording from a real person or computer-generated speech. The client computing device 104 may provide a visual output via a display device 132 communicatively coupled to the computing device 104.

[0028] The client computing device 104 may be associated with an end user who provides an image or video that may be indicative of a query as input to the client computing device 104 (via the sensors 134). The end user may receive, in response to the input, an audio output in the form of computer-generated speech, which may be provided from the data processing system 102 (or the content provider computing device 106 or the chatbot provider computing device 108) to the client computing device 104 and output from the transducer 136 (e.g., a speaker). The end user may receive, in response to the input, a visual output in the form of a computer-generated graphical user interface, which may be provided from the data processing system 102 (or the content provider computing device 106 or the chatbot provider computing device 108) to the client computing device 104 and output to the display 132. The input detected by the one or more sensors 134 may include one or more of an audio input (e.g., an acoustic signal), a visual input (e.g., an image or video data), a motion input, or other input. Input to the computing device 104 (e.g., one or more of audio, image, visual, or motion input) may be converted to a digital file and provided to the data processing system 102 for further processing or action generation. For example, input to the computing device 104 (e.g., one or more of audio, image, visual, or motion input) may trigger selection of a computer program including a chatbot, trigger generation of a query to be input to the chatbot, and the chatbot may provide output (e.g., one or more of audio, image, visual, or motion input) to the computing device 104 in response to the generated query or corresponding to the input.

[0029] The computing device 104 may receive an input audio signal detected by a sensor 134 (e.g., a microphone) of the computing device 104. The input audio signal may include, for example, a query, question, command, instruction, or other statement provided in language. The input audio signal may include an identifier or name of a chatbot to which the question or query is directed. For example, a query may include the name of a chatbot followed by the query so as to instruct the data processing system 102 to input the query into a specified chatbot. For example, the input audio signal may include "Foodbot, suggest some good restaurants nearby". The input audio signal may or may not include an identifier of a chatbot.

[0030] The system 100 may include, access or otherwise interact with at least one third-party chatbot provider device 108. The third-party chatbot provider device 108 may include one or more servers and may be referred to as a third-party chatbot server. The third-party chatbot provider device 108 may be integrated with the data processing system 102, or at least partially executed by the data processing system 102. The third-party chatbot provider device 108 may include at least one logical device such as a computing device with a processor to communicate with, for example, a computing device 104, a data processing system 102, or a content provider computing device 106 via a network 105. The chatbot provider device 108 may include at least one computing resource, server, processor, or memory. For example, the chatbot provider device 108 may include multiple computing resources or servers located in at least one data center. The chatbot provider device 108 may include one or more components or functions of the data processing system 102.

[0031] The chatbot provider device 108 may include or refer to a chatbot developer, such as an entity that designs, develops, manages or maintains a computer program that forms or provides one or more chatbots. A chatbot may include a computer program that conducts a conversation via an auditory, image, or text method. A chatbot may be designed to simulate how a human being would behave as a conversation partner. A chatbot may be used for customer service or information acquisition in a conversation system. A chatbot may include or use a natural language processing system (e.g., a natural language processor component 112). The chatbot may scan the keywords in the input and then extract the reply with the most matching keywords or the most similar wording pattern from a database. The chatbot may be programmed with a process that uses application pattern matching to find a predefined conversation data structure. The chatbot may be programmed with natural language processing technology to identify the grammar and syntax of the input, tokenize the input, or otherwise process the input to determine a response.

[0032] The chatbot provider device 108 may include, be coupled to or otherwise communicate with at least one chatbot provider natural language processor component 142 and a chatbot provider interface 144. The chatbot provider computing device 108 may include at least one chatbot provider natural language processor (NLP) component 142 and at least one chatbot provider interface 144. The chatbot provider NLP component 142 (or other components of the chatbot provider computing device 108) may interact with the client computing device 104 (via the data processing system 102 or bypassing the data processing system 102) to create a back-and-forth real-time voice or audio-based dialogue (e.g., a session) between the client computing device 104 and the chatbot provider computing device 108. The chatbot provider NLP 142 may include one or more functions or features of the NLP component 112 of the data processing system 102. For example, the chatbot provider interface 144 may receive data messages or provide data messages to the interface 110 of the data processing system 102. The chatbot provider computing device 108 and the content provider computing device 106 may be associated with the same entity. For example, the content provider computing device 106 may create, store, or provide a digital component for a chatbot, and the chatbot provider computing device 108 may establish a session with the client computing device 106 to communicate via the chatbot via the client computing device 104. The data processing system 102 may also establish a session with the client computing device 104 via the interface 110, the chatbot component 114, or other components, including or bypassing the chatbot provider computing device 104.

[0033] The third-party chatbot provider device 108 may refer to one or more servers of an entity that is different from the entity that manages or provides the data processing system 102. The third-party chatbot device 108 may receive a computer program for a chatbot. The third-party chatbot device 108 may provide natural language processing and other functions. The third-party chatbot device 108 may be coupled or communicated with the computing device 104 to provide chatbot functionality. For example, the third-party chatbot device 108 may execute or run the chatbot to conduct a conversation with a user of the computing device 104. The third-party chatbot device 108 may be executed on a server remote from the data processing system 102 and the computing device 104. In some cases, the third-party chatbot device 108 may be executed at least partially on the computing device 104 (e.g., as part of the preprocessor 140).

[0034] The data processing system 102 may include a content placement system having at least one computing resource or server. The data processing system 102 may include at least one interface 110, interfaced with or otherwise communicating with at least one interface 110. The data processing system 102 may include at least one natural language processor component 112, interfaced with or otherwise communicating with at least one natural language processor component 112. The data processing system 102 may include at least one chatbot component 114, interfaced with or otherwise communicating with at least one chatbot component 114. The data processing system 102 may include at least one hook component 116, interfaced with or otherwise communicating with at least one hook component 116. The data processing system 102 may include at least one content selector component 118, interfaced with or otherwise communicating with at least one content selector component 118. The data processing system 102 may include at least one output merging component 120, interfaced with at least one output merging component 120 or otherwise in communication with at least one output merging component 120. The data processing system 102 may include at least one data repository 122, interfaced with at least one data repository 122 or otherwise in communication with at least one data repository 122. At least one data repository 122 may include or store templates 124, historical data 126, content data 128, or chatbot data 130 in one or more data structures or databases. The data repository 122 may include computer data storage or memory, and may store one or more templates 124, historical data 126, content data 128, or chatbot data 130, as well as other data. The template 124 may include information that facilitates the generation of a display output. The template 124 may include a graphical user interface template. The graphical user interface template may define fonts, colors, or layouts to generate a display output, such as a display output via a chatbot interface. The template 124 may be stored as, for example, a data structure, a photo editor file, or a style sheet.

[0035] The historical data 126 may be stored in one or more data structures. The historical data 126 may include historical network activity associated with the computing device 104, an identifier of a chatbot utilized by the computing device 104, a configuration of the computing device 104, device capabilities, preferences, or other information associated with the computing device 104 that may facilitate content selection or selection of a location to insert a digital component in a chatbot result. The content data 130 may include, for example, content activity information, content groups, content selection criteria, digital component objects, or other information provided by the content provider computing device 106 or acquired or determined by the data processing system to facilitate content selection. The content data 130 may include, for example, historical performance of content activity. The content data 128 may include digital components for audio output, display output, or associated metadata, and input audio messages that may be part of one or more communication sessions with the client computing device 104. Digital component (or digital component object) content may include, for example, content items, online documents, audio, images, videos, multimedia content, or sponsored content.

[0036] The chatbot data 130 may include an identifier of the chatbot, information about the type of chatbot (e.g., category, restrictions, or topics). The chatbot data 130 may be stored in a data structure and may be indexed based on the identifier of the chatbot. The chatbot data 130 may be further indexed based on an identifier associated with the computing device 104. The data repository 122 may include one or more local or distributed databases and may include a database management system.

[0037] The interface 110, the natural language processor component 112, the chatbot component 114, the hook component 116, the content selector component 118, or the output merging component 120 may each include at least one processing unit or other logic device such as a programmable logic array engine, or a module configured to communicate with a database repository or database 122. The interface 110, the natural language processor component 112, the chatbot component 114, the hook component 116, the content selector component 118, the output merging component 120, and the data repository 122 may be separate components, a single component, or part of the data processing system 102. The system 100 and its components such as the data processing system 102 may include hardware elements such as one or more processors, logic devices, or circuits.

[0038] Data processing system 102 may obtain anonymous computer network activity information associated with multiple computing devices 104. A user of computing device 104 may affirmatively authorize data processing system 102 to obtain network activity information corresponding to the user's computing device 104. For example, data processing system 102 may prompt the user of computing device 104 to obtain consent to obtain one or more types of network activity information. The identity of the user of computing device 104 may remain anonymous, and computing device 104 may be associated with a unique identifier (e.g., a unique identifier of the user or computing device provided by the data processing system or the user of the computing device). The data processing system may associate each observation with a corresponding unique identifier.

[0039] The content provider computing device 106 can provide a digital component based on audio, vision or multimedia to be presented as an audio output digital component or a visual output digital component by the client computing device 104. The digital component can be or include a digital component. The digital component can be or include a digital object. The digital component can include a brand name or company name of a commodity or service. The digital component can be configured for parameter-driven text-to-speech technology. The digital component can be configured for a text-to-speech (TTS) implementation that converts ordinary language text into speech. The digital component can be input to an application programming interface that utilizes speech synthesis capabilities to synthesize text into natural-sounding speech of various languages, accents and voices. The digital component can be encoded as plain text or speech synthesis markup language (SSML). SSML can include parameters that can be set to control various aspects of speech, such as pronunciation, volume, pitch or rate that can form an acoustic fingerprint or native voice.

[0040] The content provider computing device 106 may provide content selection criteria for the digital component, such as values, keywords, concepts, or other metadata or information that facilitates the content selection process. The content provider computing device 106 may also provide the audio-based digital component (or other digital components) to the data processing system 102, where they may be stored in the data repository 122. The data processing system 102 may select the audio digital component (or a component configured for parameter-driven text, image, or video-to-speech technology) and provide the audio digital component (or instruct the content provider computing device 106 to provide the audio digital component) to the client computing device 104. The audio-based digital component may be exclusively audio, or may be combined with text, image, or video data.

[0041] Content provider computing device 106 may provide digital components to data processing system 102 for storage in content data structure 128 in data repository 122. Data processing system 102 may retrieve digital components in response to a request for content or otherwise determine that digital components are to be provided.

[0042] The content provider computing device 106 can establish an electronic content campaign. The electronic content campaign can be stored in the data repository 122 as content data 128. The electronic content campaign can refer to one or more content groups corresponding to a common theme. The content campaign can include a hierarchical data structure, which includes content groups, digital component data objects (e.g., digital components or digital objects), and content selection criteria. In order to create a content campaign, the content provider computing device 106 can specify values ​​for activity level parameters for the content campaign. The activity level parameters can include, for example, the name of the campaign, the preferred content network for placing digital component objects, the value of the resources for the content campaign, the start and end dates of the content campaign, the duration of the content campaign, the schedule for the placement of digital component objects, the language, the geographic location, and the type of computing device on which the digital component object is to be provided. In some cases, an impression can refer to the time when the digital component object is obtained from the source of the digital component object (e.g., the data processing system 102 or the content provider computing device 106), and the impression is countable. In some cases, due to the possibility of click fraud, robot activities can be filtered and excluded as impressions. Thus, in some cases, an exposure may refer to a measure of a response from a web server to a page request from a browser, filtered from robot activity and error codes, and recorded as close as possible to the opportunity for rendering a digital component for display on computing device 104. In some cases, an exposure may refer to a visible or audible exposure; for example, a digital component object or digital component is at least partially (e.g., 20%, 30%, 30%, 40%, 50%, 60%, 70%, or more) visible on a display device of client computing device 104, or audible via speaker 136 of computing device 104. A click or selection may refer to a user interaction with a digital component object, such as a voice response to an audible exposure, a mouse click, a touch interaction, a gesture, a shake, an audio interaction, or a keyboard click. A conversion may refer to a user taking a desired action with respect to a digital component object; for example, purchasing a product or service, completing a survey, visiting a physical store corresponding to a digital component, or completing an electronic transaction.

[0043] The content provider computing device 106 may further establish one or more content groups for the content campaign. A content group includes one or more digital component objects and corresponding content selection criteria, such as keywords, words, terms, phrases, geographic locations, types of computing devices, time of day, interests, topics, or verticals. Content groups under the same content campaign may share the same campaign-level parameters, but may have customized specifications for content group-level parameters, such as keywords, negative keywords (e.g., preventing placement of digital components in the presence of negative keywords on primary content), keyword bidding, or parameters associated with bidding or content campaigns.

[0044] To create a new content group, the content provider computing device 106 may provide values ​​for content group level parameters for the content group. Content group level parameters include, for example, a content group name or content group theme, and bids or results (e.g., clicks, impressions, or conversions) for different content placement opportunities (e.g., automatic placement or managed placement). The content group name or content group theme may be one or more terms that the content provider computing device 106 may use to capture a topic or theme for which a digital component object of a content group is to be selected for display. For example, a food and beverage company may create different content groups for each brand of food or beverage it sells, and may further create different content groups for each model of vehicle it sells. Examples of content group themes that a food and beverage company may use include, for example, "Brand A Cola", "Brand B Ginger Ale", "Brand C Orange Juice", "Brand D Sports Drink", or "Brand E Pure Water". For example, an example content campaign theme may be "Soda Water", and include content groups for "Brand A Cola" and "Brand B Ginger Ale". A digital component (or digital component object or digital component) may include "Brand A," "Brand B," "Brand C," "Brand D," or "Brand E." A digital component object or digital component may refer to a digital component configured for parameter-driven text-to-speech technology.

[0045] The content provider computing device 106 can provide one or more keywords and digital component objects to each content group. Keywords can include terms related to products or services associated with or identified by digital component objects. Keywords can contain one or more terms or phrases. For example, a food and beverage company can use "soda", "cola", "soft drink" as keywords for content groups or content activities, which can describe the goods or services provided by the brand. In some cases, the content provider can specify negative keywords to avoid, prevent, block or disable content placement for certain terms or keywords. The content provider can specify a match type for selecting a digital component object, such as an exact match, a phrase match, or a broad match.

[0046] The content provider computing device 106 may provide one or more keywords for use by the data processing system 102 to select digital component objects provided by the content provider computing device 106. The content provider computing device 106 may identify one or more keywords to bid on and further provide bid amounts for various keywords. The content provider computing device 106 may provide additional content selection criteria for use by the data processing system 102 to select digital component objects. Multiple content providers 106 may bid on the same or different keywords, and the data processing system 102 may run a content selection process or an advertisement auction in response to receiving an indication of a keyword of an electronic message.

[0047] The content provider computing device 106 may provide one or more digital component objects for selection by the data processing system 102. When a content placement opportunity becomes available, the data processing system 102 may (e.g., via the content selector component 118) select a digital component object that matches resource allocation, content schedule, highest bid, keyword, and other selection criteria specified for the content group. Different types of digital component objects may be included in the content group, such as voice digital components, audio digital components, text digital components, image digital components, video digital components, multimedia digital components, or digital component links. After selecting a digital component, the data processing system 102 may transmit the digital component object to be rendered on the computing device 104 or a display device of the computing device 104. Rendering may include displaying the digital component on the display device, or playing the digital component via a speaker of the computing device 104. The data processing system 102 may provide instructions to the computing device 104 or the chatbot component 114 or the third-party chatbot provider device 108 to present the digital component object. Data processing system 102 may instruct computing device 104 , display 132 , or audio driver 138 of computing device 104 to generate audio signals, sound waves, or visual output.

[0048] The data processing system 102 may include an interface component 110 designed, configured, constructed, or operated to receive and transmit information using, for example, data packets. The interface 110 may receive and transmit information using one or more protocols, such as a network protocol. The interface 110 may include a hardware interface, a software interface, a wired interface, or a wireless interface. The interface 110 may facilitate converting or formatting data from one format to another. For example, the interface 110 may include an application programming interface that includes definitions for communicating between various components, such as software components.

[0049] The data processing system 102 may include an application, script, or program installed on the client computing device 104, such as an application for transmitting an input audio signal to the interface 110 of the data processing system 102 and driving a component of the client computing device to render an output audio signal or a visual output. The data processing system 102 may receive a data packet, a digital file, or other signal including or identifying an audio input signal (or an input audio signal). The computing device 104 may detect an audio signal via a transducer 136 and convert the analog audio signal into a digital file via an analog-to-digital converter. For example, the audio driver 138 may include an analog-to-digital converter component. In some cases, the preprocessor component 140 may convert the audio signal into a digital file that can be transmitted via a data packet over the network 105.

[0050] The data processing system 102 may execute or run the NLP component 112 to receive or obtain data packets including input audio signals detected by the sensor 134 of the computing device 104. The data packets may provide digital files. The NLP component 112 may receive or obtain digital files or data packets including audio signals and parse the audio signals. For example, the NLP component 112 may provide interaction between a person and a computer. The NLP component 112 may be configured with techniques for understanding natural language and allowing the data processing system 102 to derive meaning from human or natural language input. The NLP component 112 may include or be configured with techniques based on machine learning such as statistical machine learning. The NLP component 112 may parse the input audio signal using a decision tree, a statistical model, or a probabilistic model. The NLP component 112 can perform, for example, functions such as named entity recognition (e.g., given a stream of text, determine which items in the text map to appropriate names such as people or places, and what the type of each such name is, such as person, location, or organization), natural language generation (e.g., converting information or semantic intent from a computer database into understandable human language), natural language understanding (e.g., converting text into a more formal representation, such as a first-order logic structure that a computer module can manipulate), machine translation (e.g., automatically translating text from one human language to another human language), morpheme segmentation (e.g., separating words into individual morphemes and identifying categories of morphemes, which can be challenging based on the morphological or structural complexity of the words in the language under consideration), question answering (e.g., determining answers to human language questions, which can be specific or open-ended), and semantic processing (e.g., processing can be performed after identifying a word and encoding its meaning in order to associate the identified word with other words of similar meaning).

[0051] The NLP component 112 can convert the audio input signal into recognized text by comparing the input signal with a stored (e.g., in a data repository 122) representative audio waveform set and selecting the closest match. The audio waveform set can be stored in the data repository 122 or other databases accessible to the data processing system 102. Representative waveforms are generated across a large set of users and can then be enhanced with voice samples from the users. After the audio signal is converted into recognized text, the NLP component 112 matches the text with a word, which is associated with an action that the data processing system 102 can serve, for example, via training across users or by manually specifying it. The NLP component 112 can convert an image or video input into a text or digital file. The NLP component 112 can process, analyze, or interpret the image or video input to perform an action, generate a request, or select or recognize a data structure.

[0052] The audio input signal may be detected by a sensor 134 or transducer 136 (e.g., a microphone) of the client computing device 104. The client computing device 104 may provide the audio input signal to the data processing system 102 (e.g., via the network 105) via the transducer 136, the audio driver 138, or other components, where it may be received as a digital file or digital format (e.g., via the interface 110) and provided to the NLP component 112 or stored in the data repository 122. In some cases, in addition to or in place of the input acoustic signal, the data processing system 102 may receive an image or video input signal. The data processing system 102 may process the image or video input signal using, for example, image interpretation techniques, computer vision, a machine learning engine, or other techniques to recognize or interpret the image or video to convert the image or video into a digital file. One or more image interpretation techniques, computer vision techniques, machine learning techniques may be collectively referred to as imaging techniques. In addition to or in place of the audio processing techniques, the data processing system 102 (e.g., the NLP component 112) may be configured with imaging techniques.

[0053] The NLP component 112 can obtain an input audio signal. From the input audio signal, the NLP component 112 can identify at least one request or at least one trigger keyword corresponding to the request. The request can indicate the intention or theme of the input audio signal. The trigger keyword can indicate the type of action that may be taken. For example, the NLP component 112 can parse the input audio signal to identify at least one request to leave home for a dinner party and watch a movie in one night. The trigger keyword may include at least one word, phrase, root or partial word or derivative indicating the action to be taken. For example, the trigger keyword "go" or "to go to" from the input audio signal can indicate the need for transportation. In this example, the input audio signal (or the identified request) does not directly express the intention of the transportation, but the trigger keyword indicates that riding the transportation is an auxiliary action of at least one other action indicated by the request.

[0054] The NLP component 112 can parse the input audio signal to identify, determine, retrieve or otherwise obtain the request and trigger keywords. For example, the NLP component 112 can apply semantic processing techniques to the input audio signal to identify trigger keywords or requests. The NLP component 112 can apply semantic processing techniques to the input audio signal to identify trigger phrases including one or more trigger keywords such as a first trigger keyword and a second trigger keyword. For example, the input audio signal may contain the sentence "Foodbot, suggest some good restaurants nearby". The NLP component 112 can determine that the input audio signal includes an application identifier and a query. The application identifier may refer to a chatbot identifier or name. The NLP component 112 can apply semantic processing techniques or other natural language processing techniques to a data group including a sentence to identify the application identifier "Foodbot" and the query "suggest some good restaurants nearby". The NLP component 112 can further identify multiple keywords, such as "good", "restaurant", and "nearby", which can be used to generate queries for input into the application. For example, the NLP component 112 may determine that a trigger phrase includes a trigger keyword and a plurality of additional keywords that may be used to form a query.

[0055] The NLP component 112 can filter the input audio signal to identify the application identifier and the query. In some cases, the input audio signal may include a query, but may not include the application identifier or the name of the chatbot. For example, the data packet carrying the input audio signal may include "It would be great if I could get help finding some good restaurants nearby", in which case the NLP component 112 can filter out one or more terms as follows: "it", "would", "be", "great", "if", "I", "could", "get" or "help". By filtering out these terms, the NLP component 112 can more accurately and reliably identify queries or keywords such as "finding", "good", "restaurant", "nearby", and determine that this is a request to start a restaurant chatbot.

[0056] The input audio signal may include an application identifier, such as an identifier of a desired chatbot. The input audio signal may include a unique identifier of the chatbot, a name of the chatbot, or other indication of which chatbot to launch or invoke. For example, the input audio signal may include a name of the chatbot, such as "food robot". The application identifier may be an alphanumeric identifier.

[0057] Thus, the data processing system 102 may receive data packets corresponding to an input audio signal carrying speech content detected by the sensor 134 or transducer 136 of the computing device 104. The input audio signal may be converted to a digital file by an analog-to-digital converter (e.g., an audio driver 138) of the computing device 104. The data processing system 102 may receive data packets corresponding to the digital file via the network 105, and then parse the data packets or digital file to select a computer program including a chatbot. For example, the data processing system 102 may include a chatbot component 114 that is designed and constructed to select a computer program including a chatbot in response to a data packet or digital file for execution by the data processing system 102 or the computing device 104 or a third-party chatbot provider device 108.

[0058] The chatbot component 114 can identify keywords, tokens, terms, concepts, or other information in the digital file. The chatbot component 114 can utilize the natural language processor component 112 to identify keywords, tokens, terms, concepts, or other information in the digital file. The natural language processor component 112 can provide the parsed keywords, tokens, terms, or concepts to the chatbot component 114. The chatbot component 114 can select a chatbot that responds to the keywords or concepts of the digital file. The chatbot component 114 can select a chatbot that is mapped to an application identifier provided in the input audio signal.

[0059] The data processing system 102 (e.g., via the NLP component 112 or the chatbot component 114) may determine that the input audio signal includes the application identifier "food robot". The NLP component 112 or the chatbot component 114 may further identify a query in the input audio signal. The NLP component 112 may provide the application identifier to the chatbot component 114. The chatbot component 114 may perform a lookup in the chatbot data structure 130 to identify the chatbot mapped to the application identifier "food robot". In some cases, the input audio signal may not include a chatbot identifier, but may include a query indicating the type of chatbot that can provide a response to the query. For example, the chatbot data structure 130 may include keywords or other information that describes, for each chatbot, the goods, services, or functions that the chatbot can provide (e.g., restaurant search). The chatbot component 114 may use the identifier determined via the chatbot data structure 130 to start, initiate, execute, or otherwise activate the corresponding chatbot. In some cases, the identifier may include a file name or file path, a pointer, a web address, an Internet Protocol address, a uniform resource locator, or other identification information of the chatbot, or may be associated with it. For example, the data processing system 102 may determine that a recipe chatbot is provided via a third-party chatbot provider device 108 and instruct the third-party chatbot provider device 108 to launch a restaurant search chatbot and interact with the computing device 104 directly or via the data processing system 102 (e.g., via the chatbot component 114).

[0060] Thus, the data processing system 102 can determine the application identifier and the query to be input into the application corresponding to the application identifier from the input audio signal. The data processing system 102 can provide the query to the application or chatbot so that the application generates a second query for transmission to a third-party server and identifies multiple responses to the query for display via the computing device 104. For example, the chatbot component 114 can at least partially execute the chatbot or facilitate the interface between the chatbot and the computing device 104. The data processing system 102 can call the chatbot and input the query received via the input audio signal. The chatbot can identify the query after execution and then generate a second query for transmission to the third-party chatbot provider device 108. The third-party chatbot provider device 108 can receive the second query, perform the search, and then generate one or more responses to the query.

[0061] In some cases, before initiating or causing the initiation or execution of a chatbot, the data processing system 102 may determine whether the computing device 104 is authorized to access the chatbot. The data processing system 102 (e.g., via the chatbot component 114) may perform a lookup in the data repository 122 (e.g., chatbot data 130) using the identifier of the computing device 104 to determine whether the computing device 104 is authorized to access the computer program containing the chatbot. Authorization may be based on subscriptions, plans, restrictions, resource requirements, version control, or device capabilities. For example, if the computing device 104 is configured with a predefined version of the operating system, the data processing system 102 may grant the computing device 104 access to the chatbot. In another example, if the computing device 104 is associated with a valid account or profile, the data processing system 102 may grant the computing device 104 access to the chatbot. In some cases, if the data processing system 102 determines that the computing device 102 is not authorized to access the chatbot, the data processing system 102 may terminate the thread, prompt the user, or identify another chatbot that the computing device 104 is authorized to access. Thus, data processing system 102 may select a chatbot in response to determining that computing device 104 is authorized to access the chatbot.

[0062] The interface 110 may initiate the chatbot itself, or transmit instructions to the third-party chatbot provider device 108 to cause the third-party chatbot provider device 108 to call a conversation application programming interface associated with the chatbot (e.g., the NLP component 142) and establish a communication session between the data processing system 102 or the third-party chatbot provider device 108 and the client computing device 104. In response to establishing a communication session between the data processing system 102 or the third-party chatbot provider device 108 and the client computing device 104, the data processing system 102 or the third-party chatbot provider device 108 may transmit data packets directly to the client computing device 104 via the network 105. In some cases, the third-party chatbot provider device 108 may transmit data packets to the client computing device 104 via the data processing system 102 and the network 105.

[0063] The chatbot provider device 108 or the data processing system 102 may execute at least a portion of the chatbot provider NLP component 142. For example, the third-party chatbot provider device 108 may handle certain aspects of the communication session or query types. The third-party chatbot device 108 may utilize the NLP component 112 executed by the data processing system 102 to facilitate processing audio signals associated with the communication session and generating responses to the query. In some cases, the data processing system 102 may include a session API 142 configured for the third-party chatbot device 108. In some cases, the data processing system routes data packets between the client computing device and the third-party provider device 108 to establish a communication session. The data processing system 102 may receive an indication from the third-party chatbot device 108 that the third-party device has established a communication session with the client device 104. The indication may include an identifier of the client computing device 104, a timestamp corresponding to the time when the communication session was established, or other information related to the communication session, such as a data structure associated with the communication session.

[0064] The chatbot provider NLP component 142 may be a second NLP that includes one or more components or functionality of the first NLP 112. The chatbot provider NLP component 142 may interact with or utilize the first NLP 112. In some cases, the system 100 may include a single NLP 112 executed by the data processing system 102. The single NLP 112 may support the data processing system 102 and the chatbot. In some cases, the interface 110 generates or constructs data structures to facilitate performing the service, and the conversation API generates responses or queries to advance the communication session with the end user or obtain other information to improve or enhance the end user's experience or the performance of the service.

[0065] The computer program including the chatbot may be executed on the data processing system 102, the chatbot provider device 108, or the computing device 104. The chatbot may receive and process one or more digital files or a portion of one or more digital files to determine a response. For example, the chatbot may be executed on the data processing system 102 as a chatbot component 114.

[0066] The chatbot can recognize the input query after execution. The data processing system 102 can call, execute or start the chatbot and input a query received or determined based on the input audio signal from the computing device 104. For example, the data packet received by the data processing system 102 from the computing device 104 can include an application identifier and a query. The data processing system 102 (e.g., via the chatbot component 114) can call the chatbot corresponding to the application identifier and then input the query "good restaurant nearby". The chatbot component 114 can generate a second query, which is to be input to the chatbot or provided to the third-party chatbot provider device 108 in order to perform a search and generate one or more responses.

[0067] The chatbot component 114 can generate a second query based on the input query in the input audio signal. The second query can be transmitted to the third-party chatbot provider device 108. The second query can be formatted to facilitate processing by the third-party chatbot provider device 108. The second query may include information different from the first query received in the input audio signal. The second query may include additional information relative to the first query, or less information relative to the first query. The chatbot component 112 can perform initial processing or preprocessing on the input query to reduce the amount of processing that the third-party chatbot provider device 108 may need to perform. The chatbot component 112 can perform initial processing or preprocessing on the input query to reduce the number of remote procedure calls between the third-party chatbot provider device 108, the data processing system 102, and the computing device 104.

[0068] For example, an input query received in an input audio signal from the computing device 104 may be “Foodbot, suggest some good restaurants nearby.” The chatbot component 114 may determine that there is a location component of the input query based on the term “nearby.” Instead of passing the query as is to the third-party chatbot provider device 108, which may cause the third-party chatbot provider device 108 to subsequently query the data processing system 102 or the computing device 104 for location information, the chatbot component 114 may construct a second query and input location information corresponding to the computing device 104. The data processing system 102 (e.g., via the chatbot component 114) may determine the location of the computing device 104. The data processing system 102 may determine the location preferences established in the profile of the computing device 104. The location preferences may include, for example, a location resolution (e.g., address, within a radius of 100 meters, 200 meters, 300 meters, 500 meters, 1000 meters, 1 mile, zip code, city, town, or county) that blocks transmission of a location or allows transmission. Based on the location preference, the data processing system 102 may identify the current location of the computing device 104 and construct a second query to include the following location information: “identify good restaurants near 123 Main Street, Anytown, USA”. Thus, by pre-processing the input query to identify missing information, then determining the missing information and generating a second query including the determined information, the data processing system 102 may improve overall system efficiency by reducing excessive remote procedure calls performed by the third-party chat provider device 108, thereby reducing computing resource utilization or battery consumption of the computing device 104.

[0069] The chatbot component 114 can provide the generated second query to the third-party chatbot provider device 108. The third-party chatbot provider device 108 can use natural language processing technology, search engine technology, pattern matching technology, or semantic analysis technology to identify one or more responses to the query. For example, the chatbot can perform a search using the query and then generate a response containing the search results.

[0070] The third-party chatbot provider device 108 may implement or perform a search using a second query provided by the chatbot component 114. The third-party chatbot provider device 108 may access one or more databases internal or external to the third-party chatbot provider device 108. In some cases, the third-party chatbot provider device 108 may utilize one or more components, functions, resources, or databases of the data processing system 102 to perform the search. The third-party chatbot provider device 108 may identify multiple responses to the second query. The multiple responses may include search results. The multiple responses may include, for example, a list of restaurants that meet the search criteria defined by the second query. For example, the list of restaurants may include restaurants that are within a radius of the current location of the computing device 104 and are "good" based on a rating of the restaurant being greater than or equal to 4 stars.

[0071] The data processing system 102 may include, execute, or otherwise communicate with a hook component 116 to obtain one or more responses to the second query from the chatbot application (or a third-party chatbot provider device 108 that at least partially executes the chatbot application or generates a response to the second query) before displaying on the computing device 104. In the case where the data processing system 102 executes the chatbot application to identify the response to the second query, the hook component 116 may obtain the response before the chatbot application transmits the response to the computing device 104. In some cases, the hook component 116 may intercept the response from the third-party chat provider device 108 because the response may be intended to be directed to the chatbot application or computing device 104 for display. Interception may refer to the hook component 116 obtaining the response and forwarding it to the content selector component 118 instead of the chatbot component 114 or computing device 104 for display.

[0072] The chatbot application may generate a response to the second query by executing on at least one of the data processing system 102, the chatbot component 114, or the third-party chatbot provider device 108. After generating or identifying the response to the second query, the chatbot application (whether or not executing at least in part on one or more of the data processing system 102, the third-party chatbot provider device 108, or the computing device 104) may transmit or provide the response for presentation via the display 132 of the computing device 104. However, before presenting the result for display, the hook component 116 may obtain or intercept the response to the second query. The hook component 116 may obtain or intercept the response from a component of the data processing system 102, the third-party chatbot provider device 108, or the computing device 104.

[0073] System 100 may include a hook component 118 executing on one or more of data processing system 102 or client computing device 104 to obtain or intercept a response to the second query before providing the response for display on computing device 104. Hook component 116 may obtain or intercept the response and redirect the response to output merger component 120.

[0074] The hook component 116 can use hook technology to obtain or intercept the response to the second query, which is designed to intercept function calls, responses, digital components, messages or events passed between components of the data processing system 102, between the data processing system 102 and the computing device 104, between the data processing system 102 and the third-party provider device 108, or between the third-party provider device 108 and the computing device 104. The hook component 116 can insert an event hook configured to intercept the response when the chatbot application is running. In some embodiments, the hook component 118 can be executed in the network stack, for example, before passing data to the application layer of the stack. In other embodiments, the hook component 118 can be executed in the kernel level, user level of the operating system, or can be executed by a hypervisor independent of one or more virtual machines.

[0075] For example, a hook component 118 executing on the data processing system 102 or computing device 104 may obtain or intercept a response provided via the chatbot application or third-party chatbot provider device 108 and forward the response to the content selector component 118. The hook component may be configured to reroute or redirect the response to the content selector component 118. The interception may occur at any convenient or appropriate part of the processing, such as in the network stack (e.g., at or below the application layer); after the response has been written to the sandbox media environment's memory or buffer, but before the sandbox media environment's processing thread reads the digital component; and so on.

[0076] The data processing system 102 may include, execute, or otherwise communicate with a content selector component 118 to receive a response to the second query from the chatbot application and select a digital component. The content selector component 118 may parse the response received from the chatbot application and generate keywords. The content selector component 118 may perform a real-time content selection process using keywords generated based on the response intercepted by the hook component 116. The content selector component 118 may use a real-time content selection process to identify a digital component. The digital component may be provided by the content provider device 106.

[0077] The content selector component 118 may generate one or more keywords based on the response from the chatbot component 118. The content selector component 118 may parse the response to identify one or more keywords. The content selector component 118 may use semantic analysis techniques, machine learning models, pattern matching techniques, or other keyword generation techniques to identify or generate one or more keywords. The content selector component 118 may identify a topic vertical field, concept, category, commodity, service, or entity associated with the response to generate keywords. For example, the response to the query "good restaurant nearby" may include the following three results: i) restaurant A, ii) restaurant B, and iii) restaurant C. The results may further include a description of the type of food provided in each restaurant. The type of food may include burgers and fries at restaurant A, pizza and pasta at restaurant B, and pancakes and waffles at restaurant C. The content selector component 118 may parse these results to generate the following keywords: burgers, fries, pizza, pasta, pancakes, and waffles. The content selector component 118 may input these automatically generated keywords into the real-time content selection process to select the sponsored digital component.

[0078] The real-time content selection process may refer to or include selecting sponsored digital component objects provided by third-party content providers 106. The real-time content selection process may include a service in which digital components provided by multiple content providers are parsed, processed, weighted, or matched to select one or more digital components to be provided to the computing device 104. The content selector component 118 may perform the content selection process in real-time. Performing the content selection process in real-time may refer to performing the content selection process in response to a request for content received via the client computing device 104. The real-time content selection process may be performed (e.g., initiated or completed) within a time interval (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, or 20 minutes) of receiving the request. The real-time content selection process may be performed during a communication session with the client computing device 104 or within a time interval after the communication session is terminated.

[0079] For example, the data processing system 102 may include a content selector component 118 designed, constructed, configured or operated to select digital component objects. The content selector component 118 may use image processing techniques, character recognition techniques, natural language processing techniques or database searches to identify, analyze or recognize the voice, audio, word, character, text, symbol or image of the candidate digital component. The candidate digital component may include metadata indicating the subject matter of the candidate digital component, in which case the content selector component 118 may process the metadata to determine whether the subject matter of the candidate digital component corresponds to the input audio signal.

[0080] When establishing a content campaign that includes a digital component, the content provider 106 may provide additional indicators. The content provider computing device 106 may provide information that the content selector component 118 may identify by performing a search using information about candidate digital components at the content campaign or content group level. For example, a candidate digital component may include a unique identifier that may be mapped to a content group, content campaign, or content provider. The content selector component 118 may determine information about the content provider computing device 106 based on information in a content campaign data structure stored in the data repository 122.

[0081] Data processing system 102 may automatically generate a request for content in response to intercepting a response to the second query from the chat application. Data processing system 102 may receive a request for content to be provided via computing device 104. The request may include selection criteria for the request, such as device type, location, and keywords associated with the request.

[0082] In response to the request, the data processing system 102 may select a digital component object from a data repository 122 or a database associated with the content provider computing device 106, and provide the digital component via the network 105 for presentation via the computing device 104. The digital component object may be provided by a content provider device 108 that is different from the chatbot provider device 108. The computing device 104 may interact with the digital component object. The computing device 104 may receive an audio response to the digital component. The computing device 104 may receive an indication for selecting a hyperlink or other button associated with the digital component object, which enables or allows the computing device 104 to identify the content provider computing device 106, request a service from the content provider computing device 106, instruct the content provider computing device 106 to perform a service, transmit information to the content provider computing device 106, or otherwise identify a good or service associated with the content provider computing device 106.

[0083] The request for content may include content selection criteria, such as the format of the content, keywords, concepts, profile information, or other information that may facilitate content selection. The content selector component 118 may perform a real-time content selection process. Real-time content selection may refer to performing content selection in response to a request for content. After the chatbot identifies a dialog data structure in response to voice input, a request for content may be generated, transmitted, or otherwise provided.

[0084] The content selector component 118 can select a digital component including text, a string or a character, which can be processed by a text-to-speech system or can be presented via a display. The content selector component 118 can select a digital component in a parameterized format configured for a parameterized driven text-to-speech technology. In some cases, the dialog data structure can be in SSML format, or can be configured with voice parameters. The data processing system 102 can configure the voice parameters of the digital component to match the voice parameters of the dialog data structure identified by the chatbot, so that the digital component can be presented to the user of the computing device 104 with native voice, image or acoustic fingerprint (e.g., compared with the dialog data structure without the digital component, the digital component has the same or similar acoustic properties).

[0085] The content selector component 118 may select a digital component in a parameterized format configured for text to speech, rather than a digital component in an audio file format. For example, the content selector component 118 may not select a digital component in an audio file in an audio file format or an audio coding format (such as, .WAV, .AIFF or .AU), because the digital component already in the audio file format may not be configured for seamless insertion into the placeholder field of the conversation data structure identified by the chatbot computer program. Compared with the native voice of the computing device or the acoustic fingerprint set for the chatbot, the digital component in the audio file format may have a different acoustic fingerprint. If the digital component audio file has a different acoustic fingerprint (e.g., words are spoken at different rates, different frequencies, different pitches, different tones, different volumes or different accents) compared to the native voice or acoustic fingerprint of the chatbot or the conversation data structure, the insertion or integration of the digital component audio file into the placeholder field in the conversation data structure may not be seamless, smooth or continuous. For example, a digital component audio file with different acoustic fingerprints may cause awkward transitions or signs of inconsistency. Thus, by providing digital components configured for text-to-speech technology, where a chatbot or computing device can play the digital components in a manner that corresponds to the acoustic fingerprint or native voice of the chatbot or computing device, the data processing system 102 can facilitate providing seamless modification of the chatbot computer program output.

[0086] The content selector component 118 may select a digital component that is in a format configured for display via the display device 132. The content selector component 118 may select a digital component that may be reformatted to match a native output format of the chatbot application.

[0087] The content selector component 118 can select the plurality of digital components via a real-time content selection process. The content selector component 118 can score and rank the digital components and provide the plurality of digital components to the output merger component 120 to allow the output merger component 120 to select the highest ranked digital component.

[0088] The content selector component 118 may provide the selected digital component to an output merger component 120. The data processing system 102 may include, execute, or otherwise communicate with the output merger component 120 to combine or integrate the selected digital component with a response received from the chatbot application for presentation via the computing device 104. The output merger component 120 may provide the combined digital component and response for presentation via an audio output or a visual output.

[0089] To reduce redundant entries, thereby preventing or reducing network bandwidth utilization or processor utilization of the computing device 104, the output merging component 120 can perform a deduplication process on the digital components and multiple responses generated by the chatbot application. Deduplication can refer to eliminating or removing duplicate or redundant entries in computer data. Deduplication techniques can ensure that only one unique data instance is retained in the output. Redundant data entries can be replaced with new data entries.

[0090] For example, the digital component selected by the content selector component 118 may include a digital component for restaurant A, which is also one of the responses generated by the chatbot application in response to the second query. In this example, the output merge component 120 may remove or delete the digital component and select a second digital component (e.g., a second ranked digital component output from the real-time content selection process). Thus, based on the deduplication process results, the data processing system 102 may request an additional digital component, prevent the insertion of a digital component, or determine to add the digital component if it is not a duplicate entry.

[0091] The output merging component 120 can determine to add the digital component to the response in response to a deduplication process performed on the digital component and the multiple responses of the chatbot application. The output merging component 120 can determine that the digital component is unique or different from the response. The output merging component 120 can then construct a display output that includes the digital component and the response from the chatbot application.

[0092] To manage the rendering of display outputs via a graphical user interface and reduce graphics processor utilization, CPU utilization, or memory utilization, and network bandwidth utilization when transmitting digital components and results, the output merging component 120 can construct the display output using a unified graphical layout. For example, instead of using a first graphical theme that requires different layouts, styles, colors, and responses for the graphics and a second, different graphical theme for the digital components, the output merging component 120 can determine a single, unified graphical theme and layout that uses the merged data set.

[0093] In order to construct the display output, the output merging component 120 can retrieve the graphical user interface template from the template data structure 124 stored in the data repository 122. The graphical user interface template can define, provide, set or otherwise indicate that the output merging component 120 can be used to generate the font, color and layout of the display output. The output merging component 120 can select the graphical user interface template based on the application identifier of the chatbot application. For example, each application identifier (e.g., each chatbot application) can be established with a graphical user interface template. The third-party chatbot provider device 108 can set, configure, establish or design a graphical user interface template. The graphical user interface template may include fonts, colors, layouts or other graphical user interfaces or display elements for text. In some cases, the graphical user interface template may include or define animation, transparency, positioning, size or sound.

[0094] The output merging component 120 can retrieve the graphical user interface template from the template data structure 124 by performing a lookup in the template data structure 124. The template data structure 124 can store the graphical user interface templates in an index data structure. In some cases, the computing device 104 can build a customized graphical user interface for the computing device 104, which can include a preferred theme, layout, number of responses, or font. The output merging component 120 can generate a tuple formed by the application identifier and the identifier of the computing device 104, and use the tuple to perform a lookup in the template data structure 124 to retrieve a graphical user interface template built for a specific chatbot application and customized for use with the computing device 104.

[0095] In some cases, the output merging component 120 may request a graphical user interface template from a third-party chatbot provider device 108. For example, the output merging component 120 may determine that the template data structure 124 does not include a graphical user template corresponding to the application identifier of the chatbot, or that the stored graphical user template has expired, invalidated, or obsolete. The output merging component 120 may transmit the request to the third-party chat provider device 108 corresponding to the application identifier, and then receive a graphical user interface template in response to the request for constructing a display output. The output merging component 120 may then update the template data structure 124 in the data repository 122 with the received graphical user interface template and use it to construct a subsequent display output.

[0096] The output merging component 120 can select a location in the response received from the chatbot application where the digital component is to be inserted or placed. The graphical user interface template of the chatbot application can indicate the location where the digital component is to be inserted. The graphical user interface template can indicate that the digital component is appended to the list of responses. The graphical user interface template can indicate that the digital component is added before the list of responses or within the list of responses (e.g., after the first response, after the second response, or after the third response). The graphical user interface template can indicate that the digital component is added to the location so that the digital component looks consistent or similar to the response to provide an appearance that the response and the digital component are provided by the same source, such as a chatbot application.

[0097] In some cases, the output merging component 120 can dynamically select the positioning. For example, the output merging component 120 can use historical interactions with the digital component to determine the positioning based on performance information. The output merging component 120 can use a machine learning model to determine where to insert the digital component. The machine learning model can be generated using historical performance data collected from multiple instances of the chatbot application. The performance data can indicate features associated with clicks, selections, or conversions. The data processing system 102 can receive an indication of an interaction with the digital component. The interaction with the digital component can occur via an interface or component of the computing device 104. The data processing system 102 can record, store, or otherwise monitor and track the interaction with the digital component and information about or associated with the interaction.

[0098] The machine learning model may more heavily weight targeting associated with clicks, selections, or conversions on the digital component, while reducing weights associated with targeting that does not receive clicks, selections, or conversions. The machine learning model may generate different models for different types of digital components, topics, keywords, computing devices, or other parameters in order to customize the machine learning model to determine targeting for inserting digital components.

[0099] The output merging component 120 can use the retrieved graphical user interface template to construct the display output. The output merging component 120 can construct the display output by integrating the digital component selected via the real-time content selection process with the response generated by the chatbot application. The output merging component 120 can consistently apply the graphical user template to both the digital component and the response to improve the efficiency of rendering the display output by the computing device 104. In order to construct the display output, the output merging component 120 can modify the font or color of the digital component to integrate the digital component with the response. The output merging component 120 can provide the display output to the computing device 104 so that the computing device 104 renders the display output to be presented via the display 132.

[0100] The computing device 104 may include, execute, or be referred to as a digital assistant device. The digital assistant device (or computing device 104) may include a display device 132, a graphics driver 146, a sensor for detecting an input audio signal, and a preprocessor component 140 coupled to the display device, the graphics driver, and the sensor. The graphics driver 146 may receive a display output from the data processing system 102 and render the display output on the display device 132. The graphics driver 146 may include a hardware or software component that controls or enhances how graphics or visual output is displayed on the display 132. The graphics driver 146 may include, for example, a program that controls how the graphics component works with the rest of the computing device 104 (or digital assistant). The preprocessor component 140 filters the input audio signal to create a filtered input audio signal, converts the filtered input audio signal into data packets, and transmits the data packets to a data processing system including one or more processors and memory.

[0101] The digital assistant device may include an audio driver 138 and a speaker assembly (e.g., transducer 136). The pre-processor assembly 140 receives the indication of the display output and instructs the audio driver 138 to generate an output audio signal to cause the speaker assembly (e.g., transducer 136) to transmit an audio output corresponding to the indication of the display output. For example, the audio output may include Figure 3 A text-to-speech representation of the text described in text box 312 .

[0102] Figure 2 is an illustration of the operation of a system 200 for managing rendering of a graphical user interface. The operation of the system 200 may include Figure 1200. For example, the operation of system 200 may be performed by data processing system 102, computing device 104, third-party chatbot provider device 108, or content provider device 106. At action 202, computing device 104 may detect or receive an input audio signal. The input audio signal may include a voice input spoken by an end user. The input audio signal may include a first query. The input audio signal may or may not include an application identifier. The input audio signal may include an identifier of a chatbot application for executing the query. The input audio signal may not include an application identifier, but data processing system 102 may automatically determine the chatbot application to be called to execute the query. In some cases, if the input audio signal does not include an application identifier, computing device 104 or data processing system 102 may provide an audio or visual prompt that requires the end user to provide an indication of the chatbot application for executing the query.

[0103] At action 204, the computing device 104 transmits a data packet indicating a first query. The data processing system 102 may receive the first query. The NLP component 112 of the data processing system 102 may process the data packet to identify the first query or application identifier. The data processing system 102 may provide the first query to the chatbot component 114, or may provide the first query directly to the third-party chatbot provider server 108.

[0104] At action 206, the data processing system 102 may provide a first query to the chatbot component 114. The chatbot component 114 may identify a query and an application identifier corresponding to a chatbot application that may execute the query to generate a response. The chatbot component 114 may generate a second query based on the first query. The second query may be in a format that facilitates the chatbot application to generate a response to the first query. The second query may filter out information that may not be needed for the chatbot application to generate a response to the query. The chatbot component 114 may add information relative to the first query to generate the second query. The chatbot component 114 may construct or generate the second query in other ways to facilitate the third-party chatbot application to generate a response to the query while reducing remote procedure calls. For example, the first query may not include location information of the computing device 104, but the location information may be helpful or necessary to generate a response. The chatbot component 114 may detect that the location information is missing in the first query, and may also determine to include the location information when generating the second query. The chatbot component 114 may determine to include the location information because the chatbot component 114 may determine that the lack of location information in the second query may cause the chatbot application or third-party chatbot provider 108 to return an error or generate a prompt for location information. Therefore, in order to prevent the chatbot application or third-party chatbot provider 108 from encountering a runtime error or generating a subsequent prompt for additional information, the chatbot component 114 may proactively and automatically determine the location information of the computing device and generate the second query including the location information. By generating a complete second query without missing information, the chatbot component 114 may help reduce latency or delay in generating a response to the input audio signal.

[0105] At action 208, the chatbot component 114 provides the second query to the third-party chatbot provider 108. In some cases, the NLP component 112 or the chatbot component 114 may forward the first query to the third-party chatbot provider server 108 without changing, modifying, or otherwise adjusting the first query. For example, the chatbot component 114 may determine that the first query includes sufficient information to allow the third-party chatbot provider 108 to generate a response. In some cases, the chatbot component 114 may forward the first query to the third-party chatbot provider 108, and the third-party chatbot provider may generate a request for additional information to facilitate identifying a response. The third-party chatbot provider 108 may transmit the request for additional information to the chatbot component 114, the data processing system 102, or directly to the computing device 104. After receiving the requested additional information, the third-party chatbot provider 108 may generate a second query and identify a response to the second query, which may also be responsive to the first query.

[0106] The third-party chatbot provider device 108 may perform a second query to identify a response and then provide the response to the data processing system 102 at action 210. The hook component 116 may intercept the response. For example, the response may have been intended to go directly to the output merger component 122 or directly to the computing device 104. The hook component 116 may intercept the response and then provide the response to the content selector component 118 at action 212. The content selector component 118 may parse the response, generate one or more keywords based on the response, and then perform a real-time content selection process using the one or more keywords. The content selector component 118 may provide the selected one or more digital components to the output merger component 122 at action 214. The output merger component 122 may perform a deduplication process, retrieve a graphical user interface template, and construct a unified display output that combines the digital components with the response from the third-party chatbot provider device 108. The output merger component 122 may transmit or provide the display output to the computing device 104 at action 216. Computing device 104 may render a display output via display device 132 to visually present the output to an end user who provided the input audio signal.

[0107] Figure 3 is an illustration of an embodiment of a graphical user interface provided by a system for managing rendering of a graphical user interface. Graphical user interface 300 may be provided by Figure 1 , Figure 2 or Figure 5. A user may input an audio signal to a digital assistant 306 (e.g., a digital assistant executed by computing device 102). A graphical user interface 300 provided by the digital assistant 306 executed by computing device 102 may indicate an icon 302 to the user and visually present the text of the input audio signal in a text box 304, as shown below: “Foodbot, suggest some good restaurants nearby.” The input audio signal may include an application identifier and a query. In this example, the application identifier may be “Foodbot” and the query may be “suggest some good restaurants nearby”. The digital assistant 306 may identify the application identifier. The digital assistant 306 may identify the application identifier by locally parsing the input audio signal, or by transmitting the data packets to the data processing system 102, and then receiving a response with an indication of the application identifier from the data processing system 102. The digital assistant 306 may provide a status indication in a text box 308, such as “Connecting you to Foodbot”. In some cases, digital assistant 306 may provide an audio indication of the status.

[0108] The data processing system 102 or the digital assistant 306 can interact with the third-party chatbot provider device 108 to obtain a response to the query. The digital assistant 306 or the data processing system 102 can transmit the received query as is to the third-party chatbot provider device 108, or generate a second query based on the first query that can facilitate the third-party chatbot provider device 108 to identify a response.

[0109] The data processing system 102 may receive an indication of a response from the third-party chatbot provider device 108, generate keywords, perform a real-time content selection process using the keywords to identify a digital component, and then use a graphical user interface template to merge the digital component and the response. The data processing system 102 may provide a display output including the digital component integrated with the response.

[0110] The digital assistant 306 can receive the display output and render the display output 314. The digital assistant 306 can display an icon 310 of the food robot and include a text box 312 indicating a response. For example, the text box 312 can state "We suggest the following restaurants. All three are 4+stars, and within 5 miles of your location." The text can correspond to the input query 304. For example, the input term "good" may have caused the food robot 310 to identify restaurants with more than 4 stars. The input term "nearby" may have caused the food robot 310 to identify restaurants within 5 miles of the current location of the computing device 104.

[0111] A display output 314 may be constructed using a graphical user interface template for the food robot 310. The display output 314 may include responses from the third-party chatbot provider device 108 and sponsored digital components. The graphical user interface template may define the layout, colors, and fonts for the responses and digital components. For example, the graphical user template may define: i) a layout including a single column with each response having a separate row (e.g., 4 rows, one row for each restaurant); ii) icons 320 for the responses located on the left side of the row; iii) two lines of text per response, where the first line of text includes the restaurant name in bold and underline, followed by a colon, then a description of the food served by the restaurant, and the second line of text includes the street address of the restaurant.

[0112] To integrate the digital component with the response from the third-party chatbot server, the data processing system 102 may apply the graphical user template to both the response provided by the chatbot and the digital component selected via the real-time content selection process to generate display output 314. Integrating the digital component with the response may refer to adding the digital component to the response in a manner that preserves or maintains the look and feel or other user interface features. Integrating the digital component with the response may refer to combining the digital component with the response in a manner that preserves or maintains the look and feel or other user interface features.

[0113] The data processing system may integrate the digital component with the response using a graphical user interface template. The graphical user interface template may indicate that the digital component is to be attached to the response. Thus, the display output 314 may include: i) a first response 314a for restaurant A, which includes the description "Burgers & Fries" on the first text line and the address "123 Main Street" on the second line; ii) a second response 314b for restaurant B, which includes the description "Pizza & Pasta" in the first text line and the address "456 Main Street" in the second text line; iii) a third response 314c for restaurant C, which includes the description "Pancakes & Waffles" in the first text line and the address "321 Main Street" in the second text line. The data processing system 102 may attach the digital component 314d for restaurant D using the same graphical user interface template as used for responses 314a-c. The data processing system 102 can select the digital component of restaurant D by generating keywords based on the responses 314a-c and inputting the keywords into the real-time content selection process. The data processing system 102 can construct a display output to integrate the digital component with the responses so that the digital component matches the same graphical user interface element of the responses 314a-c. For example, the digital component can match the layout with two lines of text and an icon 322. The text of the digital component can include the name of the restaurant 318 in bold and underline, followed by a colon and the description "Pizza&Subs", and the second line of text includes the address. Similar to the text 316 of restaurant A, the text 318 can be bold and underlined to match the style. However, the display of the digital component can be changed by including an indication that it is a sponsored digital component 324.

[0114] Figure 4 is a diagram of a method for managing rendering of a graphical user interface. Method 400 may be performed by Figure 1 , Figure 2 or Figure 5 The method 400 may include, at 402, a data processing system receiving a data packet. The data processing system may receive the data packet via an interface. The data packet may include an input audio signal detected by a sensor of a computing device. For example, a microphone of a digital assistant device may detect the input audio signal.

[0115] At action 404, the data processing system may determine an identifier and a query. The data processing system may determine an application identifier and a first query to be input into an application corresponding to the application identifier. The data processing system may determine the application identifier and the first query based on processing or parsing an input audio signal. The data processing system may apply natural language processing techniques or semantic analysis techniques to identify the application identifier and the first query. The application identifier may identify or indicate a chatbot application. The chatbot application may be called to execute or process the query. The chatbot application may include a conversation application programming interface configured to process the query.

[0116] At action 406, the data processing system may generate and provide a second query. The data processing system may generate the second query based on the first query. The data processing system may generate the second query based on the first query and information associated with the computing device that detects the input audio signal. The data processing system may supplement the first query with additional information to generate the second query. For example, the first query may include an indication of a term related to location, such as, "near me", "nearby", "close by", "within walking distance" or "in my neighborhood". The data processing system (e.g., via a chatbot component) may parse the first query to identify the term related to location. The data processing system may determine information associated with the term, such as the current location of the computing device. The data processing system may then generate a second query including location information. For example, if the first query is "find restaurants near me", the second query may be "find restaurants in San Jose, California". The data processing system may transmit or provide the second query to the chatbot application or a third-party chatbot provider server.

[0117] In some cases, the data processing system may provide the first query directly to the third-party chatbot provider server. The third-party chatbot provider server may parse the first query and determine additional information that may be required to generate a relevant response to the first query. In response to parsing the first query and identifying that missing information or additional information may facilitate generating a response, the third-party server may generate a prompt or request for the additional information and transmit the prompt or request to the data processing system or computing device. The data processing system or computing device may respond to the request from the third-party server with the requested additional information.

[0118] In some cases, the data processing system may provide a query to a chatbot application. The chatbot application may receive a first query and generate a second query based on the first query. The chatbot application may be executed at least in part on a data processing system. The chatbot application may include a query input interface query processing component (e.g., a chatbot component) that receives an input query or a first query, identifies lost information, or otherwise pre-processes the first query to generate a second query that helps generate a response. The chatbot application may transmit the second query to a third-party chatbot provider server to obtain a response to the second query. The data processing system may receive a response to the second query to be displayed via a computing device.

[0119] At action 408, the data processing system may obtain a response. The data processing system may obtain or receive a response from a third-party chatbot provider server. In some cases, the data processing system may intercept the response before the response is displayed on the computing device. The data processing system may use a hook component to intercept the response. The response may be transmitted by the third-party chatbot provider to a chatbot application executed on the data processing system or computing device. The hook component may intercept a response to a chatbot application executed on the data processing system or computing device. After intercepting the response, the hook component may redirect or forward the response to the content selector component.

[0120] At action 410, the data processing system may generate keywords and identify digital components. The data processing system may parse the intercepted responses to generate keywords based on the responses. The data processing system may select terms in the responses to use as keywords. The data processing system may apply semantic processing to the responses to identify the topic of the responses. The data processing system may use the automatically generated keywords to perform a real-time content selection process to identify digital components provided by a content provider.

[0121] At act 412, the data processing system may de-duplicate the digital component with the response to verify the digital component for integration with the response. The data processing system may perform de-duplication to confirm that the digital component does not match one of the organic responses provided by the third-party content provider device. If the digital component is different from the response, the data processing system may determine to add the digital component to the response. If the digital component matches one of the responses, the data processing system may determine to prevent the addition of the digital component and then select a second digital component (e.g., a second highest ranked digital component based on a content selection process).

[0122] At action 414, the data processing system can retrieve a template and construct a display output that includes a digital component integrated with the response. The data processing system can retrieve a template (e.g., a graphical user interface template) that defines fonts, colors, and layouts to render multiple responses. The template can include an identifier for a font, such as "Times New Roman" or an alphanumeric identifier for a font. The template can further include an indication of the font size, font style, character spacing, or line spacing. The template can include or define the color of the font, the color of the text, the foreground color, the background color, the color of the icon, or other color themes to be used when constructing the display output. The template can define a layout, such as the positioning of a graphical element, such as text, an icon, a response, a border, a row, or a column.

[0123] The graphical user interface template may be provided in various formats. For example, the graphical user interface template may be provided using a cascading style sheet or elements thereof to define how elements are displayed on a display screen. The graphical user interface template may use a user interface toolkit. The graphical user interface may use a *.PSD file (e.g., a photoshop editor program file format).

[0124] The data processing system may provide a display output at act 416. The data processing system may transmit the generated display output to the computing device to cause the computing device to render the display output for display via a display device communicatively coupled to the computing device.

[0125] Figure 5is a block diagram of an example computer system 500. The computer system or computing device 500 may include or be used to implement the system 100 or its components, such as a data processing system 102. The data processing system 102 may include an intelligent personal assistant or a voice-based digital assistant. The computing system 500 includes a bus 505 or other communication component for transmitting information, and a processor 510 or processing circuit coupled to the bus 505 for processing information. The computing system 500 may also include one or more processors 510 or processing circuits coupled to the bus to process information. The computing system 500 also includes a main memory 515 (such as a random access memory (RAM) or other dynamic storage device) coupled to the bus 505 to store information and instructions to be executed by the processor 510. The main memory 515 may be or include the data repository 122. The main memory 515 may also be used to store location information, temporary variables, or other intermediate information in the process of executing instructions by the processor 510. The computing system 500 may also include a read-only memory (ROM) 520 or other static storage device coupled to the bus 505 for storing static information and instructions for the processor 510. A storage device 525, such as a solid-state device, a magnetic disk, or an optical disk, may be coupled to the bus 505 to persistently store information and instructions. The storage device 525 may include the data repository 122 or may be part of the data repository 122.

[0126] Computing system 500 may be coupled to a display 535, such as a liquid crystal display or an active matrix display, via bus 505 for displaying information to a user. An input device 530, such as a keyboard including alphanumeric and other keys, may be coupled to bus 505 for communicating information and command selections to processor 510. Input device 530 may include a touch screen display 535. Input device 530 may also include a cursor control, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to processor 510 and controlling cursor movement on display 535. Display 535 may be, for example, a computer program product of data processing system 102, client computing device 150, or the like. Figure 1 part of other components.

[0127] The processes, systems, and methods described herein may be implemented by the computing system 500 in response to the processor 510 executing an arrangement of instructions contained in the main memory 515. These instructions may be read into the main memory 515 from another computer-readable medium, such as the storage device 525. Execution of the arrangement of instructions contained in the main memory 515 causes the computing system 500 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement may also be employed to execute the instructions contained in the main memory 515. Hardwired circuitry may be used in place of software instructions with the systems and methods described herein, or hardwired circuitry may be used in combination with software instructions with the systems and methods described herein. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0128] Although already Figure 5 An example computing system is described in the specification, but the subject matter including the operations described in this specification may be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more thereof.

[0129] For situations where the systems discussed herein collect personal information about a user or may make use of personal information, the user may be provided with the opportunity to control whether a program or feature may collect personal information (e.g., information about the user's social network, social behavior or activities, the user's preferences, or the user's location) or to control whether or how content is received from a content server or other data processing system that is more relevant to the user. In addition, certain data may be anonymized in one or more ways prior to storage or use so that personally identifiable information is removed when parameters are generated. For example, the identity of a user may be anonymized so that no personally identifiable information can be determined for the user, or the user's geographic location may be generalized (such as to a city, zip code, or state level) when location information is obtained so that the user's specific location cannot be determined. Thus, the user may control how information about him or her is collected and used by the content server.

[0130] The subject matter and operation described in this specification can be implemented in digital electronic circuits or computer software, firmware or hardware, including the structure disclosed in this specification and its structural equivalent, or one or more combinations thereof. The subject matter described in this specification can be implemented as one or more computer programs, for example, one or more circuits of computer program instructions, encoded on one or more computer storage media to be executed by a data processing device or to control the operation of the data processing device. Alternatively or in addition, program instructions can be encoded on artificially generated propagation signals, for example, machine-generated electrical, optical or electromagnetic signals, which are generated to encode information to be transmitted to a suitable receiver device to be executed by a data processing device. Computer storage media can be or be included in a computer-readable storage device, a computer-readable storage matrix, a random or serial access memory array or device, or a combination of one or more of them. Although computer storage media is not a propagation signal, computer storage media can be the source or destination of computer program instructions encoded with artificially generated propagation signals. Computer storage media can also be one or more separate components or media (for example, multiple CDs, disks or other storage devices) or included therein. The operation described in this specification can be implemented as an operation performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0131] The terms "data processing system", "computing device", "component" or "data processing apparatus" cover various apparatuses, devices and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or multiple chips, or a combination of the above. The apparatus may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform operating environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures. For example, the interface 110, the chatbot component 114, the content selector component 118, the output merging component 120 or the NLP component 112 and other data processing system 102 components may include or share one or more data processing apparatuses, systems, computing devices or processors.

[0132] A computer program (also referred to as a program, software, software application, application, script, or code) may be written in any form of programming language (including compiled or interpreted languages, declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system. A computer program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer, or a computer program may be deployed to execute on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0133] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs (e.g., components of data processing system 102) to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and the apparatus may also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or may be incorporated into, special purpose logic circuitry.

[0134] The subject matter described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or a combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0135] A computing system such as system 100 or system 500 may include a client and a server. The client and server are usually remote from each other and usually interact through a communication network (e.g., network 105). The relationship between the client and the server is generated by computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., data packets representing digital components) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from a user interacting with the client device). Data (e.g., the result of user interaction) generated at the client device can be received from the client device at the server (e.g., received by the data processing system 102 from the computing device 104 or the content provider computing device 106 or the chatbot provider computing device 108).

[0136] Although operations are depicted in the drawings in a particular order, such operations need not be performed in the particular order shown or in sequential order, and not all illustrated operations need be performed. The actions described herein may be performed in a different order.

[0137] The separation of the various system components need not be separated in all embodiments, and the described program components may be included in a single hardware or software product. For example, the NLP component 112 or the content selector component 118 may be a single component, application or program, a logic device having one or more processing circuits, or part of one or more servers of the data processing system 102.

[0138] Now that some illustrative embodiments have been described, it is apparent that the foregoing is illustrative rather than restrictive, and has been given by way of example. In particular, although many of the examples presented herein relate to specific combinations of method actions or system elements, those actions and those elements may be combined in other ways to achieve the same goals. The actions, elements, and features discussed in conjunction with one embodiment are not intended to be excluded from similar roles in other embodiments or embodiments.

[0139] The words and terms used herein are for descriptive purposes and should not be considered limiting. The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized by," and variations thereof herein is intended to encompass the items listed thereafter, their equivalents, and additional items, as well as alternative embodiments consisting solely of the items listed thereafter. In one embodiment, the systems and methods described herein consist of one, more than one, each combination, or all of the described elements, acts, or components.

[0140] Any reference to the implementation or element or action of the system and method mentioned in the singular form herein may also include the implementation including multiple of these elements, and any reference to the plural form of any implementation or element or action herein may also include the implementation including only a single element. Reference in the singular or plural form is not intended to limit the currently disclosed system or method, their components, actions or elements to a single or multiple configurations. Reference to any action or element based on any information, action or element may include the implementation where the action or element is based at least in part on any information, action or element.

[0141] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment," "some embodiments," "an embodiment," etc. are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in conjunction with an embodiment may be included in at least one embodiment or example. Such terms as used herein do not necessarily all refer to the same embodiment. Any embodiment may be inclusively or exclusively combined with any other embodiment in any manner consistent with the aspects and embodiments disclosed herein.

[0142] References to "or" may be interpreted as inclusive, such that any term described using "or" may indicate any of a single, more than one, and all of the described terms. For example, reference to at least one of "A" and "B" includes only "A," only "B," and both "A" and "B." Such references used in conjunction with "including" or other open-ended terms may include additional items.

[0143] Where a technical feature in the drawings, detailed description or any claims is followed by a reference numeral, the reference numeral has been included to increase the intelligibility of the drawings, detailed description and claims. Therefore, a reference numeral or its absence shall have no limiting effect on the scope of any claim element.

[0144] The systems and methods described herein may be implemented in other specific forms without departing from their characteristics. For example, the data processing system 102 may recognize a first query based on an input audio signal, and transmit the first query to a third-party chatbot provider device to generate a response to the first query without generating a second query. The foregoing embodiments are illustrative rather than limiting of the described systems and methods. Therefore, the scope of the systems and methods described herein is indicated by the appended claims rather than the foregoing description, and changes within the equivalent meaning and range of the claims are included therein.

Claims

1. A system comprising: A data processing system comprising one or more processors for: generating a query based on input from a computing device remote from the data processing system; identifying one or more responses to the query for output via an application executed by the computing device; generating keywords based on the one or more responses; selecting a digital component provided by a content provider device via a real-time content selection process utilizing the keyword; integrating the digital component with the one or more responses; and The digital component integrated with the one or more responses is provided to the computing device to cause the computing device to render the digital component integrated with the one or more responses via the application on a display device communicatively coupled to the computing device.

2. The system according to claim 1, wherein: The application includes a third-party chat robot provided by a third-party server.

3. The system according to claim 1, comprising: The data processing system is configured to receive the input, the input comprising an input audio signal detected by a sensor of the computing device.

4. The system according to claim 1, comprising the data processing system to: determining a first query based on the input; and The query is generated based on the first query in the input and information associated with the computing device.

5. The system of claim 1, comprising the data processing system to: determining a location of the computing device; and The query is generated using the location and a first query identified from the input.

6. The system of claim 1, comprising the data processing system to: providing the query to a third party server associated with the application; and The one or more responses are received from the third party server.

7. The system according to claim 1, comprising: A data processing system is provided for determining to integrate the digital component with the one or more responses.

8. The system of claim 1, comprising the data processing system to: performing a deduplication process using the digital components and the one or more responses; and In response to the deduplication process being performed using the digital component and the one or more responses, a determination is made to add the digital component to the one or more responses.

9. The system of claim 1, comprising the data processing system to: retrieving a graphical user interface template defining fonts, colors, and layout for the application to render the one or more responses; and At least one of the font or the color of the digital component is modified based on the graphical user interface template to integrate the digital component with the one or more responses based on the layout.

10. The system of claim 1, comprising the data processing system to: constructing a display output using a graphical user interface template that integrates the digital component with the one or more responses; and The display output is provided to the computing device so that the computing device renders the display output and the digital component integrated with the one or more responses via the application.

11. A method comprising: generating, by a data processing system including a memory and one or more processors, a query based on input from a computing device remote from the data processing system; identifying, by the data processing system, one or more responses to the query for output via an application executed by the computing device; generating, by the data processing system, keywords based on the one or more responses; selecting, by the data processing system, a digital component provided by a content provider device via a real-time content selection process utilizing the keyword; integrating, by the data processing system, the digital component with the one or more responses; as well as The digital component integrated with the one or more responses is provided by the data processing system to the computing device so that the computing device renders the digital component integrated with the one or more responses via the application on a display device communicatively coupled to the computing device.

12. The method according to claim 11, wherein: The application includes a third-party chat robot provided by a third-party server.

13. The method according to claim 11, comprising: An input is received by the data processing system, the input comprising an input audio signal detected by a sensor of the computing device.

14. The method according to claim 11, comprising: determining, by the data processing system, a first query based on the input; as well as The query is generated by the data processing system based on the first query in the input and information associated with the computing device.

15. The method according to claim 11, comprising: determining, by the data processing system, a location of the computing device; as well as The query is generated by the data processing system using the location and a first query identified from the input.

16. The method according to claim 11, comprising: providing, by the data processing system, the query to a third-party server associated with the application; as well as The one or more responses are received by the data processing system from the third party server.

17. The method according to claim 11, comprising: A determination is made by the data processing system to integrate the digital component with the one or more responses.

18. The method according to claim 11, comprising: performing, by the data processing system, a deduplication process using the digital components and the one or more responses; as well as A determination is made by the data processing system to add the digital component to the one or more responses in response to the deduplication process being performed using the digital component and the one or more responses.

19. The method according to claim 11, comprising: retrieving, by the data processing system, for the application, a graphical user interface template defining fonts, colors, and layout for rendering the one or more responses; as well as At least one of the font or the color of the digital component is modified by the data processing system based on the graphical user interface template to integrate the digital component with the one or more responses based on the layout.

20. The method according to claim 11, comprising: constructing, by the data processing system, a display output using a graphical user interface template that integrates the digital component with the one or more responses; as well as The display output is provided by the data processing system to the computing device so that the computing device renders the display output and the digital component integrated with the one or more responses via the application.

Images