SYSTEM AND METHOD FOR INTERFERING WITH KNOWLEDGE GRAPH IN A NETWORKED SYSTEM - Patent application

Abstract

Generating domain-specific models in networked systems. The present disclosure is generally directed to creating a domain-specific voice-activated system in an interconnected network. The system can receive an input signal detected at a client device. The input signal can be a voice-based input signal, a text-based input signal, an image-based input signal, or other type of input signal. Based on the input signal, the system can select a domain-specific knowledge graph and generate a response based on the selected knowledge graph.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Non-Provisional Patent Application No. 16 / 008,988, filed June 14, 2018, and entitled "GENERATION OF DOMAIN-SPECIFIC MODELS IN NETWORKED SYSTEM," which is incorporated herein by reference in its entirety. [Background technology]

[0002] In a networked environment, such as the Internet or other network, first-party content providers may provide information for public presentation in resources, such as web pages, documents, applications, and other resources. The first-party content may include text, video, and audio information provided by the first-party content providers. A user accessing a resource may wish to request or receive additional information about information associated with the resource. To view or receive the additional information, the user must leave the currently viewed resource. Summary of the Invention [Means for solving the problem]

[0003] According to at least one aspect of the present disclosure, a system for generating a natural language processing model in a networked system may include a data processing system. The data processing system may include one or more processors and a memory. The one or more processors may execute a natural language processor component and a digital component selector. The data processing system may receive an input audio signal detected by a sensor in a first client device via the natural language processor component and an interface of the data processing system. The data processing system may parse the input audio signal to identify a first search request in the input audio signal via the natural language processor component. The data processing system may select search results based on at least the first search request via the digital component selector executed by the data processing system. The data processing system may select a digital component based on the first search request via the digital component selector. The digital component may include a link to a data source. The data source may include multiple entities. The digital component may include an input interface for requesting a response based on a knowledge graph.

[0004] The data processing system can transmit, via the interface, a digital component including a link to the data source and associated with the knowledge graph along with the search results to the first client device. The data processing system can receive a second search request via the interface and through the input interface of the digital component rendered by the client device. The data processing system can select a response based on the second search request via the digital component selector and based on the knowledge graph. The data processing system can transmit the response via the interface to the first client device for rendering the response in the digital component.

[0005] According to at least one aspect of the present disclosure, a method for generating a natural language processing model in a networked system may include receiving, by a natural language processor component executed by a data processing system and via an interface of the data processing system, an input audio signal detected by a sensor in a first client device. The method may include parsing, by the natural language processor component, the input audio signal to identify a first search request in the input audio signal. The method may include selecting, by a digital component selector executed by the data processing system, search results based on at least the first search request. The method may include selecting, by the digital component selector, a digital component based on the first search request. The digital component may be associated with a data source and a knowledge graph based on the data source. The method may include transmitting, by the digital component selector, the digital component associated with the data source and the knowledge graph based on the data source along with the search results to the first client device.

[0006] According to at least one aspect of the present disclosure, a system for generating a natural language processing model in a networked system may include a data processing system. The data processing system may include one or more processors and a memory. The one or more processors may execute a digital component selector and a natural language processor component. The data processing system may receive a content request from a client device via the digital component selector. The data processing system may select a digital component based on the content request via the digital component selector. The digital component may be associated with a data source and a knowledge graph based on the data source. The data processing system may transmit the digital component to the client computing device via the digital component selector for rendering the digital component in a content slot. The data processing system may receive an input audio signal detected by a sensor in the client device via the natural language processor component and via an interface of the data processing system. The data processing system may parse the input audio signal to identify a request in the input audio signal via the natural language processor component. The data processing system may select a response to the request based on the knowledge graph via the natural language processor component. The data processing system may transmit the response to the client computing device via the interface.

[0007] According to at least one aspect of the present disclosure, a method for generating a natural language processing model in a networked system can include receiving a content request from a client computing device by a digital component selector executed by a data processing system. The method can include selecting, by the digital component selector, a digital component based on the content request. The digital component can be associated with a data source and a knowledge graph based on the data source. The method can include transmitting, by the digital component selector, the digital component to the client computing device for rendering the digital component in a content slot. The method can include receiving, by a natural language processor, an input audio signal detected by a sensor in the client device. The method can include parsing, by the natural language processor component, the input audio signal to identify a request in the input audio signal. The method can include selecting, by the natural language processor component, a response to the request based on the knowledge graph. The method can include transmitting, by an interface, the response to the client computing device.

[0008] These and other aspects and implementations are described in detail below. The above information and the following detailed description, including illustrative examples of the various aspects and implementations, provide an overview or framework for understanding the nature and characteristics of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations, and are incorporated in and constitute a part of this specification.

[0009] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For clarity, not every component may be labeled in every drawing. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an exemplary system for generating responses based on domain-specific natural language models in a networked computer environment, according to an example of the present disclosure. [Figure 2] FIG. 2 is a block diagram of the system shown in FIG. 1 for generating and using an exemplary knowledge graph, according to an example of the present disclosure. [Figure 3] FIG. 1 is a block diagram of an example method for generating responses based on a domain-specific natural language processing model in a networked system, according to an example of the present disclosure. [Figure 4] FIG. 1 is a block diagram of an example method for generating responses based on a domain-specific natural language processing model in a networked system, according to an example of the present disclosure. [Figure 5] 2 is a block diagram of an exemplary computer system that can be used in the system shown in FIG. 1 according to an example of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] Below follows a more detailed description of various concepts related to methods, apparatus, and systems for generating voice-activated data flows in interconnected networks and their implementations. The various concepts introduced above and described in further detail below may be implemented in any of numerous ways.

[0012] The present disclosure is generally directed to generating a domain-specific voice-activated system in an interconnected network. The system can receive an input signal detected at a client device. The input signal can be a voice-based input signal, a text-based input signal, an image-based input signal, or other type of input signal. The input signal can include a request, such as a search request. Due to the conversational nature of the input signal, the request can be vague, broad, or lacking in context. The system can generate a response to the request using a domain-specific knowledge graph. An interface to the domain-specific knowledge graph can be incorporated into a digital component that is provided in response to a search request or provided in first-party content on a web page.

[0013] The system and method of the technical solution enable network bandwidth reduction by reducing the number of network transmissions required to complete a voice-based request. The solution can enable a digital assistant to select a specific response to an unclear request and reduce the number of follow-up input audio signals required to complete the request. For example, when a first search request is received, a digital component associated with a domain-specific knowledge graph can be selected based on the first search request. When a second search request is received, the domain-specific knowledge graph can be used to process the second search request. The domain-specific knowledge graph can help provide a response to the second query even if the second query is a natural language query that is ambiguous, unclear, broad, or lacks context. This, in turn, can reduce the number of follow-up input audio signals required to complete the request. In some embodiments, the digital component can include an input interface capable of receiving the second search request. Providing the input interface to the digital component can eliminate the need for a user to leave the currently viewed resource to submit a second search request related to the digital component. In some embodiments, the digital component can be configured to render a response to the second search request within the digital component. By rendering the response within the digital component, the user does not have to leave the currently viewed resource to view or receive the response to the second search request.

[0014] 1 illustrates an exemplary system 100 for generating responses based on domain-specific natural language models in a networked computing environment. System 100 may include a data processing system 102. Data processing system 102 may communicate with one or more digital component provider devices 106 (e.g., content provider devices) or client computing devices 104 via a network 105.

[0015] The system 100 may include one or more networks 105. The networks 105 may include computer networks such as the Internet, local, wide, metro, or other area networks, intranets, satellite networks, other communication networks such as voice or data cellular networks, and combinations thereof.

[0016] Data processing system 102 and client computing devices 104 can access digital components and data sources 135 via network 105. Network 105 may be used to access data sources such as web pages, websites, domains (e.g., collections of web pages), or uniform resource locators. Digital components may be presented, output, rendered, or displayed on at least one computing device 104, such as a laptop, desktop, tablet, digital assistant, personal digital assistant, smart watch, wearable device, smartphone, portable computer, or speaker. For example, via network 105, a user of client computing device 104 can access a website (an exemplary data source 135) provided by digital component provider device 106. The website may include one or more digital components, such as first-party and third-party content.

[0017] Network 105 may include or comprise a display network, such as a subset of information resources available on the Internet that may be associated with a content placement or search engine results system, or that may include third-party digital components. Network 105 may be used by data processing system 102 to access information resources, such as web pages, websites, domain names, or uniform resource locators, that may be presented, output, rendered, or displayed by client computing device 104.

[0018] Network 105 may be any type or form of network, including a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, an SDH (Synchronous Digital Hierarchy) network, a wireless network, and a wireline network. Network 105 may include wireless links such as infrared channels or satellite bands. The topology of network 105 may include a bus, star, or ring network topology. The network may include a cellular network using any protocol used to communicate between mobile devices, including Advanced Mobile Phone Protocol ("AMPS"), Time Division Multiple Access ("TDMA"), Code Division Multiple Access ("CDMA"), Global System for Mobile Communications ("GSM"), General Packet Radio Service ("GPRS"), or Universal Mobile Telecommunications System ("UMTS"). Different types of data may be transmitted over different protocols, or the same type of data may be transmitted over different protocols.

[0019] The system 100 may include at least one data processing system 102. The data processing system 102 may include at least one logical device, such as a computing device having a processor, for communicating with, for example, a computing device 104 or a digital component provider device 106 over a network 105. The data processing system 102 may include at least one computational resource, server, processor, or memory. For example, the data processing system 102 may include multiple computational resources or servers located in at least one data center. The data processing system 102 may include multiple servers logically grouped to facilitate distributed computing techniques. A logical group of servers may be referred to as a data center, server farm, or machine farm. The servers may also be geographically distributed. A data center or machine farm may be managed as a single entity, or a machine farm may include multiple machine farms. The servers within each machine farm may be heterogeneous, and one or more servers or machines may operate according to one or more types of operating system platforms.

[0020] Servers in a machine farm, along with associated storage systems, may be housed in high-density rack systems and located in an enterprise data center. For example, consolidating servers in this manner may improve system manageability, data security, system physical security, and system performance by locating the servers and high-performance storage systems on a local, high-performance network. Centralizing all or some of the components of data processing system 102, including servers and storage systems, and combining them with advanced system management tools may enable more efficient use of server resources, conserving power and processing requirements and reducing bandwidth usage.

[0021] The client computing device 104 may include, execute, interface with, or otherwise communicate with one or more of at least one local digital assistant 134, at least one sensor 138, at least one transducer 140, at least one audio driver 142, or at least one display 144. The client computing device 104 may interface with one or more interfaces, such as graphical or physical interfaces.

[0022] The sensors 138 may include, for example, a camera, ambient light sensor, proximity sensor, temperature sensor, accelerometer, gyroscope, motion detector, GPS sensor, position sensor, microphone, video, image detection, or touch sensor. The transducer 140 may include or be part of a speaker or microphone. The audio driver 142 may provide a software interface to the hardware transducer 140. The audio driver 142 may execute audio files or other instructions provided by the data processing system 102 to control the transducer 140 to generate corresponding acoustic or sound waves. The display 144 may include one or more hardware or software components configured to provide a visual display or optical output, such as a light emitting diode, organic light emitting diode, liquid crystal display, laser, or display.

[0023] The client computing device 104 may or may not include a display. For example, the client computing device 104 may include a limited type of user interface, such as a microphone and speaker (e.g., the client computing device 104 may include a voice-activated or audio-based interface). The client computing device 104 may be a speaker-based digital assistant. The primary user interface of the computing device 104 may include a microphone and speaker.

[0024] The client computing device 104 can include, execute, interface with, or otherwise communicate with a local digital assistant 134. The local digital assistant 134 can detect an input signal, such as an audio input signal, at the client computing device 104. The input signal can include a request or a search request. The local digital assistant 134 can be an instance of the remote digital assistant component 112 executing on the data processing system 102 or can perform any of the functions of the remote digital assistant component 112.

[0025] The local digital assistant 134 can filter or modify one or more terms before transmitting the terms as data to the data processing system 102 (e.g., the remote digital assistant component 112) for further processing. The local digital assistant 134 can convert the analog audio signal detected by the transducer 140 into a digital audio signal and transmit one or more data packets carrying the digital audio signal to the data processing system 102 over the network 105. The local digital assistant 134 can transmit the data packets carrying some or all of the input audio signal in response to detecting an instruction to perform such transmission. The instruction can include, for example, a trigger keyword or other keyword or authorization to transmit the data packet including the input audio signal to the data processing system 102.

[0026] The local digital assistant 134 can interface with one or more of the knowledge graphs 126 hosted or generated by the data processing system 102. The local digital assistant 134 can provide or render an interface to the knowledge graphs 126. For example, the local digital assistant 134 can receive an input signal sent to the data processing system 102. The remote digital component assistant 112 can determine a response to the request based at least on the knowledge graph 126. The local digital assistant 134 can interface with the knowledge graph 126 to provide a result or response to the request parsed from the input signal.

[0027] The local digital assistant 134 can perform pre-filtering or pre-processing on the input audio signal to remove certain frequencies of the audio. Pre-filtering can include filters such as low-pass, high-pass, or band-pass filters. The filters can be applied in the frequency domain. The filters can be applied using digital signal processing techniques. The filters can be configured to remove frequencies that fall outside the typical frequencies of human speech while maintaining frequencies corresponding to human voice or human speech. For example, a band-pass filter can be configured to remove frequencies below a first threshold (e.g., 70 Hz, 75 Hz, 80 Hz, 85 Hz, 90 Hz, 95 Hz, 100 Hz, or 105 Hz) and above a second threshold (e.g., 200 Hz, 205 Hz, 210 Hz, 225 Hz, 235 Hz, 245 Hz, or 255 Hz). Applying a band-pass filter can reduce computing resource utilization in downstream processing. The local digital assistant 134 on the computing device 104 can apply a bandpass filter to the input audio signal before transmitting it to the data processing system 102, thereby reducing network bandwidth utilization. However, based on the computing resources available to the computing device 104 and the available network bandwidth, it may be more efficient to provide the input audio signal to the data processing system 102 to allow the data processing system 102 to perform the filtering.

[0028] The local digital assistant 134 can apply additional pre-processing or pre-filtering techniques, such as noise reduction techniques, to reduce ambient noise levels that may interfere with the natural language processor. The noise reduction techniques can improve the accuracy and speed of the natural language processor, thereby improving the performance of the data processing system 102 and managing the rendering of the graphical user interface provided via the display 144.

[0029] The client computing device 104 may be associated with an end user who inputs a voice query as audio input to the client computing device 104 (via a sensor 138 or transducer 140) and receives audio (or other) output from the data processing system 102 or digital component provider device 106 for presentation, display, or rendering to the end user of the client computing device 104.

[0030] The digital component may include computer-generated audio that may be provided to the client computing device 104 from the data processing system 102 or the digital component provider device 106. The client computing device 104 may render the computer-generated audio to the end user via a transducer 140 (e.g., a speaker). The computer-generated audio may include recordings of real people or computer-generated speech. The client computing device 104 may provide visual output via a display device 144 communicatively coupled to the computing device 104. The client computing device 104 may receive a query from the end user via a keyboard. The query may be a request or a search request.

[0031] The client computing device 104 may receive an input audio signal detected by a sensor 138 (e.g., a microphone) of the computing device 104. The input audio signal or other form of input signal may include, for example, a query, a question, a command, an instruction, a request, a search request, or other statement provided in spoken language.

[0032] The client computing device 104 can include, execute, or be referred to as a digital assistant device. The digital assistant device can include one or more components of the computing device 104. The digital assistant device can include a graphics driver that can receive display output from the data processing system 102 and render the display output on the display 132. The graphics driver can include hardware or software components that control or enhance how graphics or visual output is displayed on the display 144. The graphics driver can include, for example, a program that controls how the graphics component operates with the rest of the computing device 104 (or digital assistant). The local digital assistant 134 can filter an input audio signal to create a filtered input audio signal, convert the filtered input audio signal into data packets, and send the data packets to a data processing system including one or more processors and memories.

[0033] The digital assistant device may include an audio driver 142 and a speaker component (e.g., transducer 140). The preprocessor component 140 receives an instruction for display output and instructs the audio driver 142 to generate an output audio signal, causing the speaker component (e.g., transducer 140) to transmit an audio output corresponding to the instruction for display output.

[0034] The system 100 includes, can access, or otherwise interact with at least a digital component provider device 106. The digital component provider device 106 can include one or more servers that can provide digital components to the client computing device 104 or the data processing system 102. The digital component provider device 106 can provide or be associated with a data source 135. The data source 135 can be a website or a landing page. The digital component provided by the digital component provider device 106 can be associated with the data source 135 provided by the digital component provider device 106. For example, the digital component can be third-party content provided by the digital component provider device 106 that includes a link to the data source 135, such as a landing page.

[0035] Digital component provider device 106, or components thereof, may be integrated with or at least partially executed by data processing system 102. Digital component provider device 106 may include at least one logical device, such as, for example, a computing device having a processor for communicating with computing device 104, data processing system 102, or digital component provider device 106 over network 105. Digital component provider device 106 may include at least one computational resource, server, processor, or memory. For example, digital component provider device 106 may include multiple computational resources or servers located in at least one data center.

[0036] The digital component provider device 106 can provide audio, visual, or multimedia-based digital components for presentation by the client computing device 104 as an output digital component or a visual output digital component. The term “digital component” generally refers to data that can be rendered by the client computing device 104. A digital component can be a website, a web page, an application, text-based content, audio-based content, video-based content, other digital document, or any combination thereof. A digital component can be or include digital content. A digital component can be or include digital objects. A digital component can include multiple digital content items or other digital components. For example, a digital component can be a website that includes other digital components, such as advertisements or content from third parties. A digital component can include an instance of a local digital assistant 134, or the client computing device 104 can have an instance of a local digital assistant 134 running on it.

[0037] The digital component provider device 106 can provide digital components to the client computing device 104 over the network 105, bypassing the data processing system 102. The digital component provider device 106 can provide digital components to the client computing device 104 and the data processing system 102 over the network 105. For example, the digital component provider device 106 can provide the digital components to the data processing system 102, which can store the digital components and provide the digital components to the client computing device 104 upon request by the client computing device 104. The digital components can be from a data source 135. The data source can be a server hosting a web page, a landing page, or other content.

[0038] The data processing system 102 may include at least one computational resource or server. The data processing system 102 may include, interface with, or otherwise communicate with at least one interface 110. The data processing system 102 may include, interface with, or otherwise communicate with at least one remote digital assistant component 112. The remote digital assistant component 112 may include, interface with, or otherwise communicate with at least one NLP component 114 and at least one domain processor 117. The data processing system 102 may include, interface with, or otherwise communicate with at least one digital component selector 120. The data processing system 102 may include, interface with, or otherwise communicate with at least one data repository 124. The at least one data repository 124 may include or store a knowledge graph 126 and content data 132 in one or more data structures or databases.

[0039] The components of data processing system 102 may each include at least one processing unit or other logic device, such as a programmable logic array engine or module configured to communicate with a database repository or database 124. The components of data processing system 102 may be separate components, a single component, or part of multiple data processing systems 102. System 100 and its components, such as data processing system 102, may include hardware elements, such as one or more processors, logic devices, or circuits.

[0040] The data processing system 102 may include an interface 110. The interface 110 may be configured, constructed, or operative to send and receive information using, for example, data packets. The interface 110 may send and receive information using one or more protocols, such as network protocols. The interface 110 may include a hardware interface, a software interface, a wired interface, or a wireless interface. The interface 110 may facilitate the conversion or formatting of data from one format to another. For example, the interface 110 may include an application programming interface ("API") that includes definitions for communicating between various entities, such as software components.

[0041] The remote digital assistant component 112 of the data processing system 102 can execute or operate the NLP component 114 to receive or acquire data packets containing input signals. The input signals can include input audio signals detected by the sensors 138 of the computing device 104 or other input signals, such as entered text. For example, the input signals can include text entered by a user into the client computing device 104 via a keyboard or other text entry system. The data packets can provide digital files. The NLP component 114 can receive or acquire digital files or data packets containing the input signals and parse the input signals. For example, the NLP component 114 can enable human-computer interaction. The NLP component 114 can be configured with techniques for converting input signals to text and understanding natural language to enable the data processing system 102 to derive meaning from human or natural language input.

[0042] The NLP component 114 may include or consist of machine learning-based techniques, such as statistical machine learning. The NLP component 114 may utilize decision trees, statistical models, or probabilistic models to parse the input audio signal. The NLP component 114 may perform functions such as named entity recognition (e.g., given a stream of text, determining whether items in the text map to names of people, places, etc., and what the type of each such name is—person, place (e.g., “house”), organization, etc.), natural language generation (e.g., translating information from a computer database or semantic intent into understandable human language), natural language understanding (e.g., converting text into a more formal representation, such as a first-order logical structure that a computer module can manipulate), machine translation (e.g., automatically translating text from one human language to another), morphological segmentation (e.g., dividing words into individual morphemes and identifying classes of morphemes, which may be difficult based on the complexity of the word morphology or structure of the language being considered), question answering (e.g., determining answers in human language, which may be concrete or open-ended), or semantic processing (e.g., identifying words and encoding their meaning to associate the identified word with other words of similar meaning).

[0043] The NLP component 114 can convert an input audio signal into recognized text by comparing the input signal to a stored set of representative audio waveforms (e.g., in the data repository 124) and selecting the closest match. The set of audio waveforms can be stored in the data repository 124 or other database accessible to the data processing system 102. The representative waveforms can be generated across a large set of users and augmented with voice samples from the users. After the audio signal is converted into recognized text, the NLP component 114 matches the text to words associated with actions that the data processing system 102 can service, for example, through user-wide training or through manual specification. The NLP component 114 can convert image or video input into text or a digital file. The NLP component 114 can process, analyze, or interpret the image or video input to perform an action, generate a request, or select or identify a data structure.

[0044] The NLP component 114 can obtain an input signal. From the input signal, the NLP component 114 can identify at least one request. The request can indicate an intent or a digital component, or can be a search request. The request can be an explicitly stated request for information. For example, the request can be the question, "What color is car model X?" An intent can be derived or not explicitly stated. For example, in the input signal "Car Model X 2018," the NLP component 114 can derive an intent, even though the input signal does not explicitly state that the user wants information about 2018 car model X.

[0045] The NLP component 114 may parse the input signal to identify, determine, search, or otherwise obtain a request from the input signal. For example, the NLP component 114 may apply semantic processing techniques to the input signal to identify a search request within the input signal.

[0046] Data processing system 102 may include or interface with an instance of domain processor 117. Domain processor 117 may be any script, file, program, application, set of instructions, or computer-executable code configured to enable a computing device on which domain processor 117 executes to generate knowledge graph 126. As described in more detail below, a "knowledge graph" may be a data structure (e.g., a graph data structure) that represents relationships between multiple entities. An entity may be any data associated with (e.g., stored and / or referenced by) a data source.

[0047] The domain processor 117 can generate domain-specific knowledge graphs 126. For example, the domain processor 117 can generate knowledge graphs 126 for different specific websites, domains, collections of data, and other data sources 135. The domain processor 117 can generate a knowledge graph 126 for each digital component that the data processing system 102 receives from the digital component provider device 106. The domain processor 117 can store the knowledge graph 126 in the data repository 124. The domain processor 117 can store the knowledge graph 126 in a relational database.

[0048] A data source 135, such as a website under a particular domain, may contain terms, phrases, or other data that may generally be referred to as entities. The knowledge graph 126 of a given data source 135 may include nodes that represent the entities in the data source 135.

[0049] The knowledge graph 126 may include edges or links connecting related nodes in the knowledge graph 126. The edges may represent relationships between entities. For example, two nodes linked by an edge may indicate that the entities represented by the nodes are related. The domain processor 117 may assign weights to the edges. The weights may indicate the degree of relationship between the nodes connected by the link. For example, an edge with a high weight may indicate that the two entities represented by the nodes are more related to each other than two entities connected by an edge with a relatively low weight. The edges may represent semantic relationships between the entities represented by the nodes connected by the edges. The domain processor 117 may process text, phrases, or other entities from the data source with the NLP component 114 to generate the knowledge graph 126.

[0050] The domain processor 117 can generate a knowledge graph 126 based on entities in the data sources. The data sources can also be related to or associated with the digital components. For example, the digital components can be third-party content displayed along with first-party content in a web page. The digital components can include links to landing pages, websites, or other data sources. The domain processor 117 can generate a knowledge graph 126 for the data sources (e.g., landing pages) to which the digital components link. The domain processor 117 can store the knowledge graph 126 in the data repository 124 in association with an indication of the data sources. The domain processor 117 can generate a different knowledge graph 126 for each digital component sent to the client computing device 104 for rendering.

[0051] The knowledge graph 126 of a digital component can be generated from only primary data. For example, the knowledge graph 126 can be generated based only on entities and other data contained in data sources 135 associated with the digital component. The knowledge graph 126 of a digital component can be generated from primary and secondary data. The primary data can be entities and other data contained in data sources 135 associated with the digital component. The secondary data can be entities and other data associated with a different data source 135 or a web search. The domain processor 117 can assign different weighting factors to the primary data and the secondary data. For example, an entity in the primary data can have a relatively greater influence on the edge strength between two nodes compared to an entity in the secondary data.

[0052] The digital component provider device 106 can transmit the digital component to the data processing system 102. The data processing system 102 can store the digital component as content data 132. The digital component can be used as third-party content on a website. The digital component can include a uniform resource locator or a link to a landing page or other data source 135. When the data processing system 102 receives the digital component from the digital component provider device 106, the domain processor 117 can generate a knowledge graph 126 of the data sources 135 linked by the digital component. The domain processor 117 can store the knowledge graph 126 in the data repository 124 in association with an indication of the data source or digital component.

[0053] The domain processor 117 can generate and include in the digital component an interface to the knowledge graph 126. The interface can be a link or a deep link that causes an instance of the local digital assistant 134 to run on the client computing device 104. For example, the client computing device 104 can receive the digital component in response to a request. By rendering the digital component, the client computing device 104 can launch or run an instance of the local digital assistant 134. Responses to requests presented in the rendered digital component can be generated based on the knowledge graph 126.

[0054] The data processing system 102 may execute or operate an instance of a digital component selector 120. The digital component selector 120 may select digital components including text, strings of characters, characters, video files, image files, or audio files that may be processed by the client computing device 104 and presented to a user via a display 144 or a transducer 140 (e.g., a speaker).

[0055] The digital component selector 120 can respond to or select a digital component associated with a request identified by the NLP component 114 in the input audio signal. The digital component selector 120 can select which digital component provider device 106 should or can fulfill the request and can forward the request to the digital component provider device 106. For example, the data processing system 102 can initiate a session between the digital component provider device 106 and the client computing device 104 to enable the digital component provider device 106 to send a digital component to the client computing device 104. The digital component selector 120 can request a digital component from the digital component provider device 106. The digital component provider device 106 can provide the digital component to the data processing system 102, which can store the digital component in a data repository 124. In response to the request for the digital component, the digital component selector 120 can retrieve the digital component from the data repository 124.

[0056] The digital component selector 120 can select multiple digital components through a real-time content selection process. The digital component selector 120 can score and rank the digital components and select a digital component from the multiple digital components based on the score or rank of the digital component. The digital component selector 120 can select one or more additional digital components to be transmitted to the second client computing device 104 based on the input audio signal (or keywords and requests contained therein). The digital component selector 120 can select additional digital components (e.g., advertisements) associated with a different digital component provider device 106.

[0057] Digital component selector 120 can provide digital components selected in response to a request identified in the input signal to computing device 104, or local digital assistant 134, or an application running on computing device 104 for presentation. Thus, digital component selector 120 can receive content requests from client computing device 104, select digital components in response to the content request, and send the digital components for presentation to client computing device 104. Digital component selector 120 can send the selected digital components to local digital assistant 134 for presentation by the local digital assistant 134 itself or a third-party application running by client computing device 104. For example, local digital assistant 134 can play or output an audio signal corresponding to the selected digital component.

[0058] The data repository 124 may store content data 132, which may include digital components provided by the digital component provider device 106 or obtained or determined by the data processing system 102, for example, to facilitate content selection. The content data 132 may include digital components (or digital component objects) that may include, for example, content items, online documents, audio, images, video, multimedia content, or third-party content. The content data 132 may include digital components, data, or information provided by the client computing device 104 (or its end user). For example, the content data 132 may include user preferences, user information stored by the user, or data from a previously input audio signal.

[0059] FIG. 2 illustrates a block diagram of a system 100 that generates and uses an exemplary knowledge graph 126 to generate a response. As shown in FIG. 2, the system 100 includes a client computing device 104. The client computing device 104 is in communication with a data processing system 102 and a digital component provider device 106. The digital component provider device 106 may host, serve, or otherwise be associated with a data source 135. As shown in FIG. 2, the data source 135 may be a landing page. The landing page may be a website associated with the digital component. For example, the digital component may include a link to the landing page. The digital component may be third-party content presented on a website that includes first-party content. In this example, the digital component may be an image, video, audio clip, or text related to ACME coffee makers, and the data source 135 may be the landing page for ACME coffee makers.

[0060] The digital component provider device 106 can provide a digital component (for an ACME coffee maker in this example) to the data processing system 102. The digital component selector 120 can receive the digital component and store the digital component in a data repository 124. The data processing system 102 can generate a knowledge graph 126 based on the digital component. The data processing system 102 can generate the knowledge graph 126 based on data sources 135 associated with the digital component. For example, the domain processor 117 can process text or other content of a landing page associated with the digital component via the NLP component 114. The data processing system 102 can store the knowledge graph 126 in the data repository 124. The digital component provider device 106 can generate the knowledge graph 126 associated with the digital component and provide the knowledge graph 126 along with the digital component to the data processing system 102.

[0061] 2, the client computing device 104 may be a mobile phone. The client computing device 104 may receive an input signal 200. The client computing device 104 may receive the input signal as an input audio signal. For example, a user may speak a request into the client computing device 104. An instance of the NLP component 114 (executing on the client computing device 104 or the data processing system 102) may parse the input signal to determine the text of the input signal. The client computing device 104 may receive the input signal via a keyboard. For example, a user may type a request. The NLP component 114 may parse the input signal to identify a request, such as a search request, in the input signal 200. As shown in FIG. 2, the NLP component 114 may parse the input signal 200 to identify a search request as, "Where is the nearest coffee shop?"

[0062] The client computing device 104 may send an input signal to the data processing system 102. The data processing system 102 may select a response to the search request parsed from the input signal. The data processing system 102 may send the search response to the client computing device 104 for rendering. As shown in FIG. 2, the response may be rendered to the user as a search response 202. The search response 202 may be rendered as text, an image, a video, an audio, or any combination thereof. For example, as shown in FIG. 2, the search response 202 is rendered as text. When the client computing device 104 is a speaker-based digital assistant, the search response 202 may be rendered as an audio output signal.

[0063] Based on the search request, the digital component selector 120 of the data processing system 102 can select the digital component 204. The data processing system 102 can send the digital component 204 to the client computing device 104. The client computing device 104 can render the digital component 204 for display or presentation to the user along with the search response 202. The digital component 204 can be included in another digital component 206. The digital component 206 can be executed by the local digital assistant 134. The digital component 206 can include an interface 208 to the knowledge graph 126. The interface 208 can be configured to receive an input signal, such as a text- or audio-based input signal. The interface 208 can be referred to as an input interface 208. The digital component 204 or 206 can include a link to a landing page 135. Upon selecting the digital component 204 or 206, the client computing device 104 can activate a web browser that loads the address identified in the link on the landing page 135.

[0064] The interface 208 can accept input signals, such as audio-based or text-based input signals. The client computing device 104 can receive the input signals through the interface 208 and transmit the input signals to the data processing system 102. To receive the input signals, the interface 208 can be activated by a user. For example, a user can select, click, or tap the digital component 206 or the interface 208 to activate the interface 208 so that the interface 208 begins receiving the input signals. Activating the interface 208 can cause the input signals to go to the interface 208 rather than the interface through which the input signal 200 was received. The local digital assistant 134 can determine whether the input signal should be provided to the interface 208 or to the data processing system 102 (as the input signal 200 was). For example, the local digital assistant 134 can process the input signal with a local instance of the NLP component 114 and provide the input signal to the interface 208 based on one or more terms parsed from the input signal and based on the context or keywords of the digital component 204.

[0065] When the data processing system 102 receives an input signal via the interface 208, it can generate a response based on the knowledge graph 126. For example, the NLP component 114 can process the input signal to parse a request in the input signal. The request can be a request for additional information related to the digital component 204. For example, in the example shown in FIG. 2, the request can be for additional information about ACME coffee makers. The data processing system 102 can use the knowledge graph 126 associated with the digital component 204 to generate a response to the request. For example, the knowledge graph 126 can be based on the landing page 135 associated with the digital component 204 such that the response provides a response specific to the entities, text, and other data contained within the landing page 135. For example, a request received via the interface 208 can be, "How much does a coffee maker cost?" The data processing system 102 can use the knowledge graph 126 to generate a response to the request. The response can include the cost of the coffee maker shown on the landing page 135.

[0066] The digital component 206 can be sent to the client computing device 104 in response to a search request. The digital component 206 can be sent to the client computing device 104 in response to a request for third-party content. For example, the client computing device 104 can load a web page including first-party content. The web page can include a slot for the third-party content. The web page can include scripts or other processor-executable instructions that, when executed, cause the client computing device 104 to request the third-party content from the data processing system 102. The data processing system 102 can select the digital component 206 via the digital component selector 120 based on entities, keywords, content, or data associated with the first-party content of the web page. The data processing system 102 can generate the digital component 206 to include the digital component 204 from the digital component provider device 106 and an interface 208. The interface 208 can be processor-executable instructions that cause the client computing device 104 to launch an instance of a local digital assistant. While a user is viewing a web page that includes first-party content and digital component 206, the user can input signals into interface 208. Interface 208 can transmit the input signals to data processing system 102. Interface 208 can transmit the input signals to data processing system 102 without the browser leaving the web page. This allows the user to gather or request additional information about digital component 204 without leaving the web page.

[0067] The client computing device 104 may include an audio-only interface. For example, rather than displaying the search response 202, the client computing device 104 may render the search response 202 as an output audio file. Also, the digital components 206 and 204 may be rendered to a user as an output audio signal, for example, after rendering the search response 202. The user may speak into the client computing device 104 to provide input signals to the interface 208.

[0068] 3 shows a block diagram of an example method 300 for generating a response based on a domain-specific natural language processing model in a networked system. The method 300 may include receiving an input audio signal (ACT 302). The method 300 may include parsing the input audio signal (ACT 304). The method 300 may include selecting a search result (ACT 306). The method 300 may include selecting a digital component (ACT 308). The method 300 may include transmitting the search result and the digital component (ACT 310). The method 300 may include receiving a request (ACT 312). The method 300 may include selecting a response (ACT 314). The method 300 may include transmitting the response (ACT 316).

[0069] The method 300 may include receiving an input signal (ACT 302). The method may include receiving the input signal by an NLP component executed by the data processing system. The input signal may be an input audio signal detected by a sensor in a first client device and transmitted to the data processing system. The sensor may be a microphone of the first client device. The input signal may be an input request. A digital assistant component executed at least in part by the data processing system including one or more processors and memory may receive the input signal. The input signal may include a conversation facilitated by the digital assistant. The conversation may include one or more inputs and outputs. The conversation may be voice-based, text-based, or a combination of voice and text. The input audio signal may include text input or other types of input that may provide conversation information. The data processing system may receive session input corresponding to the conversation.

[0070] The method 300 may include parsing an input signal (ACT 304). An NLP component of the data processing system may parse the input signal to identify a request. The request may be a search request. The request may be data, information, a web page, or a search intent or request. The NLP component may identify one or more entities, such as terms or phrases, within the request.

[0071] The method 300 may include selecting search results (ACT 306). The data processing system may select search results based on at least the search request parsed from the input signal. The data processing system may include or interface with a search engine or search data processing system that may select one or more search results and provide the search results to the client computing device.

[0072] The method 300 may include selecting a digital component (ACT 308). The data processing system may select the digital component based on the search request. For example, a digital component provider device may provide digital component candidates for the data processing system. The digital component provider device may associate or label the digital component candidates with keywords. The digital component selector may select one or more digital components from the digital component candidates based on similarity between the keywords of the digital components and entities (e.g., terms) identified in the request.

[0073] The similarity may be a match. For example, the data processing system may select a digital component that has a keyword that is present as an entity in the request. For example, with reference to FIG. 2, digital component 204 may be labeled with the keyword "coffee." Because the term "coffee" is present in input signal 200, the digital component selector may select digital component 204.

[0074] The similarity can be based on semantic relationships, for example, a search can be for vacation rentals and the data processing system can match the search to digital components that include the keyword "flight booking" because the phrase "flight booking" can be semantically related to vacation rentals.

[0075] A digital component can be associated with a data source, such as a landing page or other website. A digital component can be associated with a data source when the digital component includes a link to the data source. A digital component can include a link that, when activated by a client computing device, causes a web browser executed by the client computing device to retrieve the data source. A data processing system can generate a knowledge graph based on the data source. For example, the knowledge graph can be generated from terms, phrases, or other entities included in the data source. A digital component provider device can generate the knowledge graph and provide the knowledge graph along with candidate digital components to the data processing system.

[0076] The method 300 can include transmitting the results and the digital component (ACT 310). The data processing system can transmit the search results and the digital component to the client computing device that sent the input signal to the data processing system. When the client computing device receives the results and the digital component, the client computing device can render the results and the digital component. Rendering the digital component can activate or execute an instance of a local digital assistant on the client computing device. The local digital assistant can render or otherwise display the digital component. The local digital assistant can render or otherwise display the results.

[0077] The digital component can include an interface to the knowledge graph. For example, the digital component, when rendered, can present an input interface, such as a graphical interface, to a user. Through the input interface, a user can input a request. The request can be transmitted through the digital component to a data processing system.

[0078] The method 300 may include receiving a request (ACT 312). The request may be in an input signal. The input signal may be an audio-based or text-based input signal. For example, a user may speak a question into an input interface that may be detected by a microphone of the client computing device. The local digital assistant may receive the input signal and transmit the input signal to a data processing system. When the input signal is an input audio signal, the NLP component may receive the input signal and parse the request from the input signal.

[0079] The method 300 may include selecting a response (ACT 314). The digital component selector may select a response to the request included in the input signal. The data processing system may generate the response based on a knowledge graph associated with the digital component sent to the client computing device in ACT 310.

[0080] Method 300 may include transmitting a response (ACT 316). The data processing system may transmit the response to the client computing device. The data processing system may include the response in a digital component that includes the response and instructions for how to render the response at the client computing device. The client computing device may render or display the response within the digital component transmitted to the client computing device in ACT 310. Rendering the response within a digital component previously transmitted to the client computing device may present the response to the user without modifying or altering the content currently displayed to the user. For example, referring also to FIG. 2 , rendering the results within digital component 206 may allow the input signal 200 and search response 202 to remain displayed to the user while the user requests and receives additional information regarding digital component 204.

[0081] 4 illustrates a block diagram of an example method 400 for generating a response based on a domain-specific natural language processing model in a networked system. The method 400 may include receiving a request (ACT 402). The method 400 may include selecting a digital component (ACT 404). The method 400 may include transmitting the digital component (ACT 406). The method 400 may include receiving an input signal (ACT 408). The method 400 may include parsing the input signal (ACT 410). The method 400 may include selecting a response (ACT 412). The method 400 may include transmitting the response (ACT 414).

[0082] The method 400 may include receiving a request (ACT 402). The request may be for third-party content. The request may be received from a client computing device. For example, the client computing device may include a web browser. The web browser may receive and render a website that includes the first-party content. The website may include a slot for the third-party content. The slot may include processor-executable instructions that enable the web browser to send a content request to the data processing system. The request may include content parameters. The content parameters may be size requirements for the digital components to be returned or keywords that the data processing system can use to select the digital components.

[0083] The method 400 may include selecting a digital component (ACT 404). The data processing system may select the digital component from a plurality of digital components. A digital component selector of the data processing system may select the digital component. The digital component selector may select the plurality of digital components via a real-time content selection process. The digital component selector may score and rank the digital components. The digital component selector may select the digital component from the plurality of digital components based on the score and rank of the digital component. For example, the digital component selector may select the digital component with the highest rank or score. The data processing system may include the digital component in another digital component having an interface to a knowledge graph associated with the selected digital component.

[0084] The method 400 may include transmitting the digital component (ACT 406). The data processing system may transmit the selected digital component to a client computing device. The client computing device may receive the digital component and render the digital component in one of the slots of a web page as third-party content. Rendering the digital component including the first-party content may present an interface to a knowledge graph associated with the digital component to an end user. For example, rendering the digital component may cause the client computing device to launch or run a local instance of a digital assistant. The interface may be configured to receive an input signal. The input signal may be text-based or audio-based. The client computing device may transmit the input signal received via the digital component to the data processing system.

[0085] The method 400 may include receiving an input signal (ACT 408). The method 400 may include receiving the input signal by an NLP component executed by a data processing system. The input signal may be an input audio signal detected by a sensor in the first client device and transmitted to the data processing system. The sensor may be a microphone of the first client device. The input signal may be an input request. A digital assistant component executed at least in part by the data processing system, including one or more processors and memory, may receive the input signal.

[0086] The method 400 may include parsing an input signal (ACT 410). An NLP component of the data processing system may parse the input signal to identify a request. The request may be a search request. The request may be data, information, a web page, or a search intent or request. The NLP component may identify one or more entities, such as terms or phrases, within the request. For example, the request may be for information or data related to a digital component provided to the client computing device as third-party content.

[0087] The method 400 may include selecting a response (ACT 412). The digital component to which the input signal was sent may be associated with a knowledge graph. The knowledge graph may be generated from terms or entities contained within a data source (e.g., a landing page) associated with the digital component. The data processing system may select the response based on the knowledge graph associated with the digital component. For example, an NLP component of the data processing system may use the knowledge graph to generate a response based on the entities and data contained in the landing page.

[0088] Method 400 may include transmitting a response (ACT 414). The data processing system may transmit the response to the client computing device. The client computing device may receive the response and render the response within the digital component transmitted to the client computing device in ACT 406. By rendering the response within the digital component, the response may be presented to the user without modifying or altering the first-party content presented to the user. For example, the user need not leave the original website displaying the first-party content to view or receive the response.

[0089] 5 is a block diagram of an exemplary computer system 500. The computer system or computing device 500 may include or be used to implement system 100 or components thereof, such as, for example, data processing system 102. Data processing system 102 may include an intelligent personal assistant or a voice-based digital assistant. Computing system 500 includes a bus 505 or other communication component for communicating information and a processor 510 or processing circuitry coupled to bus 505 for processing information. Computing system 500 may also include one or more processors 510 or processing circuits coupled to the bus for processing information. Computing system 500 also includes a main memory 515, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 505 for storing information and instructions to be executed by processor 510. Main memory 515 may be or include data repository 124. Main memory 515 may also be used to store location information, temporary variables, or other intermediate information during execution of instructions by processor 510. Computing system 500 may further include a read-only memory (ROM) 520 or other static storage device coupled to bus 505 for storing static information and instructions for processor 510. A storage device 525, such as a solid-state device, magnetic disk, or optical disk, may be coupled to bus 505 for persistent storage of information and instructions. Storage device 525 may include or be part of data repository 124.

[0090] Computing system 500 may be coupled via bus 505 to a display 535, such as a liquid crystal display or active matrix display, for displaying information to a user. An input device 530, such as a keyboard including alphanumeric and other keys, may be coupled to bus 505 for communicating information and command selections to processor 510. Input device 530 may include a touchscreen display 535. Input device 530 may also include a cursor control, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 510 and for controlling cursor movement on display 535. Display 535 may be part of data processing system 102, client computing device 104, or another component of FIG. 1 .

[0091] The processes, systems, and methods described herein can be implemented by computing system 500 in response to processor 510 executing an arrangement of instructions contained in main memory 515. Such instructions can be read into main memory 515 from another computer-readable medium, such as storage device 525. Execution of the arrangement of instructions contained in main memory 515 causes computing system 500 to perform the example processes described herein. One or more processors in a multiprocessing configuration can also be used to execute the instructions contained in main memory 515. Hardwired circuitry can be used in place of or in combination with software instructions with the systems and methods described herein. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0092] Although an exemplary computing system is illustrated in FIG. 5, the subject matter, including the operations described herein, can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof.

[0093] In situations where the systems described herein may collect or utilize personal information about a user, the user may be given the opportunity to control programs or features that may collect personal information (e.g., information about the user's social network, social behavior or activities, the user's preferences, or the user's location) or to control whether and / or how to receive content from content servers or other data processing systems that may be more relevant to the user. Additionally, some data may be anonymized in one or more ways before it is stored or used, such that personally identifiable information is removed when generating parameters. For example, a user's identity may be anonymized so that personally identifiable information cannot be determined about that user, or the user's geographic location may be generalized, and location information (such as to the city, zip code, or state level) obtained so that the user's specific location cannot be determined. Thus, the user may control how information about the user is collected and used by the content server.

[0094] The subject matter and operations described herein can be implemented in digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or any combination or combinations thereof. The subject matter described herein can be implemented as one or more computer programs (e.g., one or more circuits of computer program instructions) encoded on one or more computer storage media for execution by or to control the operation of a data processing device. Alternatively, or additionally, the program instructions can be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) generated to encode information for transmission to an appropriate receiver device for execution by the data processing device. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Although a computer storage medium is not a propagating signal, a computer storage medium can be the source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be, or may be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0095] The terms “data processing system,” “computing device,” “component,” or “data processing apparatus” encompass various apparatuses, devices, and machines for processing data, including, for example, a programmable processor, a computer, a system-on-chip, multiple ones, or a combination of the above. An apparatus can include special-purpose logic circuitry (e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit)). In addition to hardware, an apparatus can also include code that generates an execution environment for the computer program in question (e.g., code comprising processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof). The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. For example, the interface 110, the digital component selector 120, the domain processor 117, or the NLP component 114, and other data processing system 102 components can include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0096] A computer program (also referred to as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be arranged in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored in a single file dedicated to the program, in multiple coordinated files (e.g., a file storing one or more modules, subprograms, or portions of code), or in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document). A computer program can be arranged to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0097] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs (e.g., components of data processing system 102) to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry (e.g., an FPGA or ASIC). Devices suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0098] The subject matter described herein can be implemented in a computing system that includes back-end components (e.g., as data servers), or includes middleware components (e.g., such as application servers), or includes front-end components (e.g., such as a client computer having a graphical user interface or a client computer having a web browser through which a user can interact with an implementation of the subject matter described herein), or includes a combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network, etc.). Examples of communications networks include local area networks (“LANs”) and wide area networks (“WANs”), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0099] A computing system such as system 100 or system 500 may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communications network (e.g., network 105). The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data (e.g., data packets representing digital components) to client devices (e.g., for the purpose of displaying the data and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of a user interaction) may be received at the server from the client device (e.g., received by data processing system 102 from client computing device 104 or digital component provider device 106).

[0100] Although operations are shown in the figures in a particular order, such operations do not have to be performed in the particular order or sequence shown, and not all illustrated operations need be performed. Actions described herein may be performed in different orders.

[0101] The separation of various system components does not require separation in all implementations, and the described program components may be included in a single hardware or software product. For example, NLP component 114, domain processor 117, or digital component selector 120 may be part of a single component, app, or program, or a logic device having one or more processing circuits, or one or more servers of data processing system 102.

[0102] While several exemplary implementations have been described, it should be clear that the foregoing is illustrative and not limiting, and is presented by way of example. In particular, while many of the examples presented herein include specific combinations of method operations or system elements, those operations and those elements can be combined in other ways to achieve the same purpose. Operations, elements, and features discussed in connection with one implementation are not intended to be excluded from other implementations or similar roles in implementations.

[0103] The phraseology and terminology used herein are for purposes of description and should not be regarded as limiting. The use herein of "including," "comprising," "having," "including," "involving," "characterized by," "featured by," and variations thereof, is intended to include the items listed thereafter, equivalents thereof, and additional items, as well as alternative implementations consisting exclusively of the items listed thereafter. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, operations, or components.

[0104] Any reference herein to system and method implementations or elements or acts in the singular may also encompass implementations that include a plurality of those elements, and any reference herein to any implementation or element or act in the plural may also encompass implementations that include only a single element. References in the singular or plural are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to single or multiple configurations. References to any act or element that is based on any information, act, or element may include implementations in which the act or element is based at least in part on any information, act, or element.

[0105] Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to "implementations," "several implementations," "one implementation," etc. are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with an implementation may be included in at least one implementation or embodiment. As used herein, such terms do not necessarily all refer to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0106] References to "or" may be construed as inclusive, such that any term described using "or" may refer to either one, more than one, or all of the listed terms. For example, a reference to "at least one of 'A' and 'B'" can include "A" only, "B" only, and both "A" and "B." Such references used in conjunction with "comprising" or other open terminology can include additional items.

[0107] Where a reference sign follows a technical feature in a drawing, the detailed description, or any claim, the reference sign is included to enhance comprehension of the drawing, the detailed description, and the claim, and therefore neither the reference sign nor its absence has any limiting effect on the scope of any claim element.

[0108] The systems and methods described herein may be embodied in other specific forms without departing from their characteristics. For example, the computing device 104 may generate a packaged data object and transfer it to a third-party application when the application is launched. The above implementations are illustrative rather than limiting of the described systems and methods. Accordingly, the scope of the systems and methods described herein is indicated by the appended claims, rather than the foregoing description, and changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. [Explanation of symbols]

[0109] 100 systems 102 Data Processing System 104 Client Computing Devices 105 Network 106 Digital Component Provider Devices 110 Interface 112 Remote Digital Assistant Components 114 NLP Components 117 Domain Processor 120 Digital Component Selector 124 Data Repositories 126 Knowledge Graph 132 Content Data 134 Local Digital Assistants 135 Data Sources 138 Sensors 140 Transducer 140 Preprocessor Components 142 Audio Driver 144 displays 200 input signals 202 Search Response 204 Digital Components 206 Digital Components 208 Interface 500 Computer Systems 505 Bus 510 processor 515 main memory 520 Read-Only Memory (ROM) 525 Storage Devices 530 Input Devices 535 Display

Claims

1. 1. A system for interfacing with a knowledge graph in a networked system, comprising: a data processing system having one or more processors coupled to a memory; the data processing system stores a plurality of candidate digital components and a plurality of knowledge graphs; the data processing system, Identifying a first query from an input audio signal obtained via a client device; determining, based on the first query, a digital component from the plurality of candidate digital components to be presented via the client device; transmitting the digital component to the client device for presentation via the client device, the digital component comprising: an input interface for processing a second query obtained via the client device according to at least one of the plurality of knowledge graphs, the at least one of the plurality of knowledge graphs being generated using a plurality of entities identified from data sources linked by the digital component, the input interface being activated by a user; and a link for causing the client device to access an information resource of the data source in response to activation of the link; receiving the second query passed via an activated input interface of the digital component; generating a response corresponding to at least one of the plurality of entities based on the second query according to at least one of the plurality of knowledge graphs; sending the response presented with the digital component to the client device; The system.

2. analyzing the input audio signal obtained via the client device to identify the first query; using the first query to identify search results to be presented along with the digital component via the client device; The system of claim 1 , further comprising the data processing system:

3. generating at least one of the plurality of knowledge graphs using the plurality of entities identified from the data source, the knowledge graph defining semantic relationships between the plurality of entities; The system of claim 1 , further comprising the data processing system:

4. determining a score for each of the plurality of candidate digital components according to a content selection process based on the first query obtained via the client device; selecting a digital component from the plurality of candidate digital components for presentation via the client device; The system of claim 1 , further comprising the data processing system:

5. generating at least one of the plurality of knowledge graphs to include a plurality of nodes and a plurality of edges, the plurality of nodes corresponding to the plurality of entities and the plurality of edges corresponding to a corresponding plurality of relationships between each pair of the plurality of entities; assigning to each edge of the plurality of edges a weight for a corresponding relation of the plurality of relations based on the semantic distance between the respective pair; The system of claim 1 , further comprising the data processing system:

6. 1. A method of interfacing with a knowledge graph in a networked system, comprising: identifying a first query from an input audio signal acquired via a client device by a data processing system, the data processing system storing a plurality of candidate digital components and a plurality of knowledge graphs; determining, by the data processing system, a digital component from the plurality of candidate digital components to be presented via the client device based on the first query; transmitting, by the data processing system, the digital component to be presented via the client device to the client device, the digital component comprising: an input interface for processing a second query obtained via the client device according to at least one of the plurality of knowledge graphs, the at least one of the plurality of knowledge graphs being generated using a plurality of entities identified from data sources linked by the digital component, the input interface being activated by a user; and a link for causing the client device to access an information resource of the data source in response to activation of the link; receiving, by the data processing system, the second query passed through an activated input interface of the digital component; generating, by the data processing system, a response corresponding to at least one of the plurality of entities based on the second query according to at least one of the plurality of knowledge graphs; transmitting, by the data processing system, the response presented with the digital component to the client device; A method comprising:

7. analyzing, by the data processing system, an input audio signal obtained via the client device to identify the first query; using the first query, by the data processing system, to identify search results to be presented along with the digital component via the client device; The method of claim 6, comprising:

8. generating, by the data processing system, at least one of the plurality of knowledge graphs using the plurality of entities identified from the data source, the knowledge graph defining semantic relationships between the plurality of entities; The method of claim 6, comprising:

9. determining, by the data processing system, a score for each of the plurality of candidate digital components according to a content selection process based on the first query obtained via the client device; selecting, by the data processing system, a digital component from the plurality of candidate digital components for presentation via the client device; The method of claim 6, comprising:

Images