Systems and methods for remote control of a device

Abstract

Provides virtual remote control between digital assistant devices. A first computing device detects a second computing device and determines the capabilities of the second computing device. The first computing device generates a prompt indicating that the first computing device is capable of controlling the second computing device. In response to the prompt, the first computing device receives an instruction for controlling the second computing device. The first computing device establishes a communication channel with the second computing device. The first computing device invokes a virtual controller on the first computing device. The virtual controller forwards queries received by the first computing device via the communication channel to the second computing device to control the second computing device.

Description

Technical Field

[0001] The inventive concepts of the present application generally relate to systems and methods for remote control of devices. Background Art

[0002] A computing device such as a television may receive instructions from a remote control dedicated to the computing device. However, the computing device or remote control may have limited functionality or a limited interface. Summary of the invention

[0003] The present disclosure is generally directed to virtual remote control between digital assistant devices. A physical location may contain or include multiple computing devices. Computing devices may run different programs to form a fragmented, silo-like environment in which it is challenging or impossible for one computing device to communicate with another computing device. For example, each computing device may have different types of hardware and operating systems customized or configured for a particular computing device. In addition, it may be challenging to track or determine which computing device has which capability, resulting in a poor user experience or inability to effectively perform certain functions due to a lack of device capabilities. For example, a smart TV or monitor configured with a digital assistant may be controlled by a physical remote control. However, if the remote control is unavailable, lost, or inoperative, it may be challenging to control the smart TV or monitor. In addition, the smart TV or monitor may have a limited interface or be configured with a poor user interface.

[0004] The system and method of the present technical solution provide virtual remote control between digital assistant devices. In an illustrative example, a wearable computing device such as a smart watch can be brought into close range of a smart TV computing device that is broadcasting a presence signal. The wearable computing device can detect the presence signal of the smart TV and then determine the capabilities of the smart TV. The smart watch can display an icon indicating the ability to control the smart TV from the smart watch. In response to input from the user, the smart watch can perform a handshake process to establish a communication channel with the smart TV and call a virtual controller on the smart watch. The smart watch can then receive voice queries or other inputs from the user of the smart watch, which can be forwarded to the smart TV for processing. The smart TV can receive queries from the smart watch and process the query as if the query was received directly by the smart TV, thereby providing seamless virtual control of the smart TV via the smart watch. When the smart watch is paired with the smart TV, the smart watch can display a persistent on the smart watch to indicate the activated communication channel and virtual controller.

[0005] At least one aspect is directed to a system for dynamic remote control of a device. The system may include a first computing device having one or more processors and a memory. The first computing device may detect a second computing device located within range of the first computing device. The first computing device may determine capabilities of the second computing device in response to detecting the second computing device. The first computing device may generate a prompt indicating that the first computing device is capable of controlling the second computing device based on the capabilities of the second computing device. The first computing device may receive an instruction for controlling the second computing device in response to the prompt. The first computing device may establish a communication channel with the second computing device in response to the instruction. The first computing device may invoke a virtual controller on the first computing device. The virtual controller may forward a query received by the first computing device to the second computing device via the communication channel to control the second computing device.

[0006] At least one aspect is directed to a method for dynamic remote control of a device. The method may be performed by a first computing device having one or more processors and a memory. The method may include the first computing device detecting a second computing device located within range of the first computing device. The method may include the first computing device determining capabilities of the second computing device in response to detecting the second computing device. The method may include the first computing device generating a prompt indicating that the first computing device is capable of controlling the second computing device based on the capabilities of the second computing device. The method may include the first computing device receiving an instruction for controlling the second computing device in response to the prompt. The method may include the first computing device establishing a communication channel with the second computing device in response to the instruction. The method may include the first computing device invoking a virtual controller on the first computing device. The virtual controller may forward a query received by the first computing device to the second computing device via the communication channel to control the second computing device.

[0007] These and other aspects and embodiments are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of the various aspects and embodiments, and provide an overview or framework for understanding the nature and character of the claimed aspects and embodiments. The accompanying drawings provide illustration and further understanding of the various aspects and embodiments, and are incorporated into and constitute a part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings are not intended to be drawn to scale. Like reference numbers and names in the various drawings indicate like elements. For clarity, not every component is labeled in every drawing. In the drawings:

[0009] Figure 1 is an illustration of an example system for virtual remote control between digital assistant devices, according to an implementation;

[0010] Figure 2 is an illustration of an example operation of a system for virtual remote control between digital assistant devices, according to an embodiment;

[0011] Figure 3 is an illustration of an example method of virtual remote control between digital assistant devices according to an embodiment;

[0012] Figure 4 is to illustrate that one can use to implement the systems and methods described and illustrated herein (including, for example Figure 1 and Figure 2 The system described in Figure 3 A block diagram of the architecture of a computer system that implements the elements of the method depicted in FIG. DETAILED DESCRIPTION

[0013] The following is a more detailed description of various concepts and implementations thereof related to methods, apparatuses, and systems for virtual remote control between digital assistant devices. The various concepts introduced above and discussed in more detail below can be implemented in any of a variety of ways.

[0014] The present technical solution is generally directed to virtual remote control between digital assistant devices. For example, digital assistants can be used in more than 1 billion computing devices in the world. However, these computing devices contain various hardware and software, resulting in a large amount of fragmentation between devices and the inability to effectively communicate between devices to perform functions that the devices can perform. Therefore, the present technical solution can provide a virtual controller that is designed, constructed and operated to seamlessly control other devices (e.g., TVs, smart displays, cars or Internet of Things (IoT) devices) using a user's personal device (e.g., a mobile device, phone, watch, speaker or headset) in a manner that provides the same user interface and user experience, as if the user had physically interacted directly with the target device using the target device's physical remote control, touch or other default interface.

[0015] Providing cross-device control to perform various action sequences or key user journeys can be challenging. There may be a predefined limited set of key user journeys configured for cross-device control. A target device may be configured to perform a set of actions, while a source device different from the target device can only perform a smaller subset of these actions on the target device. For example, the target device may be able to complete the following queries: show me action movies; show me today's weather; show me my photos; play movie_name; pause / resume; turn on my TV. However, when the query is first received by a different device (e.g., a source device) and then the source device attempts to control the target device, the target device may only be able to complete a subset of the query. For example, the source device may not be able to perform the following functions on the target device: show me action movies on my TV; show me today's weather on my TV; and show me my photos on my TV. In addition, such cross-device control may require additional or excessive input to work properly. For example, the user may be required to enter the name of the target device along with each query. This may not only require the user to provide additional input, but also require the source computing device or other computing devices to perform additional natural language processing or other processing to parse the query and identify the name of the target device. Furthermore, because cross-devices and target devices are built as silos, they can have duplicate and redundant stacks that perform the same or similar functions, unnecessarily complicating the stack and accumulating technical debt without providing or detracting from a good user experience.

[0016] Therefore, the system and method of the present technical solution can provide a virtual controller that can provide seamless cross-device control. The virtual controller of the present technical solution can provide the same user experience on the target device regardless of which device is the source device on which the virtual controller is called. The virtual controller can be easily used by providing an active user interface to guide the user to access other target devices without requiring the user to learn a new user interface. The virtual controller of the present technical solution can utilize a single unified horizontal stack that allows the same experience on multiple digital surfaces and target devices.

[0017] Figure 1An example system 100 for virtual remote control between digital assistant devices according to an embodiment is shown. System 100 may include a content selection infrastructure. System 100 may include a data processing system 102. Data processing system 102 may communicate with one or more of source computing device 112, target computing device 132, or supplementary digital content provider device 144 via network 105. Network 105 may include a computer network such as the Internet, a local area network, a wide area network, a metropolitan area network, or other regional network, an intranet, a satellite network, and other communication networks (such as voice or data mobile phone networks). Network 105 may be used to access information resources such as web pages, websites, domain names, or uniform resource locators, which may be provided, output, presented, or displayed on at least one source computing device 112 or target computing device 132. Source computing device 112 may include, for example, a laptop, a desktop computer, a tablet computer, a digital assistant device, a smart phone, a mobile telecommunications device, a portable computer, a smart watch, a headset, or a speaker. Target computing device 132 may include, for example, a television, a smart display, a car unit, or a networked device (e.g., an Internet of Things (IoT) device). For example, via network 105, a user of source computing device 112 or target computing device 132 may access information or data provided by supplemental digital content provider device 144. In some cases, source computing device 112 or target computing device 132 may or may not include a display; for example, the computing device may include a limited type of user interface, such as a microphone and speaker. In some cases, the primary user interface of computing device 112 or 132 may be a microphone and speaker, or a voice interface. In some cases, computing device 112 includes a display device 122 coupled to source computing device 112, and the primary user interface of computing device 112 may utilize display device 122. Target computing device 132 may include one or more output interfaces, such as a display device or a speaker.

[0018] The source computing device 112 may refer to a computing device being used by a user or owned by a user. The source computing device 112 may be a wearable device or a mobile device. The source computing device 112 may be with the user when the user travels or moves from one location to another. The source computing device 112 may be a fixed device or a device that remains in the same location for a period of time. The target computing device 132 may refer to a computing device or client device located in a private place (such as a residence, a community house, a home, an apartment, a condominium). The target computing device 132 may be located in a public place, such as a hotel, an office, a restaurant, a retail store, a shopping mall, or a park.

[0019] The source computing device 112 may be referred to as a crossover device. The source computing device 112 may refer to a device with which a user interacts in order to control the target computing device 132. The target computing device 132 may refer to a device that is being controlled by the source computing device 112.

[0020] Network 105 may include or constitute a display network, for example, a subset of information resources available on the Internet that is associated with a content placement or search engine result system or that is eligible to include a third-party digital component as part of a digital component placement activity. Data processing system 102 may use network 105 to access information resources, such as web pages, websites, domain names, or uniform resource locators that may be provided, output, rendered, or displayed by source client computing device 112 or target computing device 132. For example, via network 105, a user of source computing device 112 or target computing device 132 may access information or data provided by supplemental digital content provider device 144.

[0021] The network 105 may be any type or form of network and may include any of the following: a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communications network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, an SDH (Synchronous Digital Hierarchy) network, a wireless network, and a wired network. The network 105 may include a wireless link, such as an infrared channel or a satellite band. The topology of the network 105 may include a bus, a star, or a ring network topology. The network may include a mobile phone network using any one or more protocols for communicating between mobile devices, including an advanced mobile phone protocol (AMPS), time division multiple access (TDMA), code division multiple access (CDMA), a global system for mobile communication (GSM), a general packet radio service (GPRS), or a universal mobile telecommunications system (UMTS). Different types of data may be transmitted via different protocols, or the same type of data may be transmitted via different protocols.

[0022] System 100 may include at least one data processing system 102. Data processing system 102 may include at least one logical device (such as a computing device with a processor) to communicate with, for example, source computing device 112, target computing device 132, or supplementary digital content provider device 144 (or third-party content provider device, content provider device) via network 105. Data processing system 102 may include at least one computing resource, server, processor, or memory. For example, data processing system 102 may include multiple computing resources or servers located in at least one data center. Data processing system 102 may include multiple logically grouped servers and facilitate distributed computing technology. Logical groups of servers may be referred to as data centers, server farms, or machine farms. Servers may also be geographically dispersed. A data center or machine farm may be managed as a single entity, or a machine farm may include multiple machine farms. The servers within each machine farm may be heterogeneous-one or more servers or machines may run according to one or more types of operating system platforms.

[0023] The servers in the machine cluster can be stored in a high-density rack system along with the associated storage systems and located in an enterprise data center. For example, consolidating servers in this manner can improve system manageability, data security, physical security, and system performance of the system by locating the servers and high-performance storage systems on a local high-performance network. Centralization of all or some of the data processing system 102 components, including servers and storage systems, and coupling them with advanced system management tools allows for more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage.

[0024] The system 100 may include, access, or otherwise interact with at least one third-party device, such as a supplemental digital content provider device 144 or a service provider device. The supplemental digital content provider device 144 or other service provider device may include at least one logic device, such as a computing device with a processor, to communicate with, for example, a source computing device 112, a data processing system 102, or a target computing device 132 via a network 105.

[0025] The supplementary digital content provider device 144 can provide an audio-based digital component for display as an audio output digital component by the source computing device 112 or the target computing device 132. The digital component can be called a sponsored digital component because it is provided by a third-party sponsor. The digital component can include an offer for a product or service, such as a voice-based message: "Do you want me to book a taxi for you?" For example, the supplementary digital content provider device 144 can include a memory for storing a series of audio digital components that can be provided in response to voice-based queries. The supplementary digital content provider device 144 can also provide an audio-based digital component (or other digital components) to the data processing system 102, wherein these digital components can be stored in the data repository of the data processing system 102. The data processing system 102 can select the audio digital component and provide (or instruct the supplementary digital content provider device 144 to provide) the audio digital component to the source computing device 112. The audio-based digital component can be audio only, or can be combined with text, image or video data.

[0026] The data processing system 102 may include a content placement system having at least one computing resource or server. The data processing system 102 may include, interface with, or communicate with at least one interface 104. The data processing system 102 may include, interface with, or communicate with at least one natural language processor 106 (or natural language processor component). The interface 104 or natural language processor 106 may form or be referred to as a server digital assistant component. The data processing system 102 may include, interface with, or communicate with at least one server digital assistant 108 (or server digital assistant component). The server digital assistant 108 may communicate or interface with one or more voice-based interfaces or various digital assistant devices or surfaces to provide data or receive data or perform other functions. The data processing system 102 may include at least one content selector 110 (or content selector component).

[0027] Data processing system 102, interface 104, NLP 106, or content selector 110 may each include at least one processing unit or other logic device, such as a programmable logic array engine, or a module configured to communicate with a data repository or database of data processing system 102. Interface 104, NLP 106, or content selector 110 may be separate components, a single component, or part of data processing system 102. System 100 and its components, such as data processing system 102, may include hardware elements, such as one or more processors, logic devices, or circuits.

[0028] The data processing system 102 may obtain anonymous computer network activity information associated with multiple source computing devices 112 (or computing devices or digital assistant devices) or target computing devices 132. The user of the source computing device 112 or mobile computing device may affirmatively authorize the data processing system 102 to obtain network activity information corresponding to the source computing device 112 or mobile computing device. For example, the data processing system 102 may prompt the user of the source computing device 112 to agree to obtain one or more types of network activity information. The source computing device 112 may include a mobile computing device such as a smart phone, a tablet computer, a smart watch, or a wearable device. The identity of the user of the source computing device 112 may remain anonymous, and the source computing device 112 may be associated with a unique identifier (e.g., a unique identifier of the user or computing device provided by the user of the data processing system or computing device). The data processing system 102 may associate each observation with a corresponding unique identifier.

[0029] The data processing system 102 may include an interface 104 (or interface component), which is designed, configured, constructed or operated to receive and send information using, for example, data packets. The interface 104 may use one or more protocols (such as network protocols) to receive and send information. The interface 104 may include a hardware interface, a software interface, a wired interface, or a wireless interface. The interface 104 may facilitate converting or formatting data from one format to another format. For example, the interface 104 may include an application programming interface, which includes definitions for communication between various components (such as software components). The interface 104 may communicate with one or more of the source computing device 112, the supplementary digital content provider device 144, or the target computing device 132 via the network 105.

[0030] The data processing system 102 may interface with an application, script, or program (such as an app) installed at the source computing device 112 or the target computing device 132 to transmit the input audio signal to the interface 104 of the data processing system 102 and drive the components of the source client computing device 112 or the target computing device 132 to present the output visual signal or audio signal. The data processing system 102 may receive a data packet or other signal that includes or identifies an audio input signal.

[0031] The data processing system 102 may include a natural language processor (NLP) 106. For example, the data processing system 102 may execute or run the NLP 106 to parse a received input audio signal or query. For example, the NLP 106 may provide for interaction between humans and computers. The NLP 106 may be configured with techniques for understanding natural language and allowing the data processing system 102 to deduce meaning based on human or natural language input. The NLP 106 may include or be configured with techniques based on machine learning, such as statistical machine learning. The NLP 106 may utilize a decision tree, a statistical model, or a probabilistic model to parse the input audio signal. NLP 106 can perform, for example, functions such as named entity recognition (e.g., given a stream of text, determining which items in the text map to appropriate names (such as people or places), and what type of each such name is (such as person, place, or organization)), natural language generation (e.g., converting information or semantic intent from a computer database into understandable human language), natural language understanding (e.g., converting text into more formal representations, such as first-order logic structures that can be manipulated by computer modules), machine translation (e.g., automatically translating text from one human language to another), morphological segmentation (e.g., separating words into individual morphemes and identifying categories of morphemes, which can be challenging based on the complexity of the morphology or structure of the words of the language under consideration), question answering (e.g., determining answers to human language questions, which can be specific or open-ended), semantic processing (e.g., processing that may occur after a word has been identified and its meaning has been encoded in order to associate the identified word with other words of similar meaning).

[0032] NLP 106 can convert the audio input signal into recognized text by comparing the input signal with a representative audio waveform set stored and selecting the closest match. The audio waveform set can be stored in a data repository or other database accessible to the data processing system 102. The representative waveform is generated across a larger user set, and then it can be expanded using a voice sample from the user. After the audio signal is converted into recognized text, NLP 106 (e.g., via training across users or by manual specification) matches the associated text to the action that the data processing system 102 can provide. Aspects or functions of NLP 106 can be performed by the data processing system 102, the source computing device 112, or the target computing device 132. For example, the NLP component can be executed on the source computing device 112 or the target computing device 132 to perform the conversion of the input audio signal into text and send the text to the data processing system 102 via data packets for further natural language processing.

[0033] The audio input signal may be detected by a sensor or transducer (e.g., a microphone) of the source client computing device 112. Via the transducer, audio driver, or other component, the source computing device 112 may provide the audio input signal to the data processing system 102 (e.g., via the network 105), where the audio input signal may be received (e.g., by the interface 104) and provided to the NLP 106 or stored in a data repository.

[0034] The data processing system 102 may receive, via the interface 104, a data packet including an input audio signal detected by a microphone of the source computing device 112 or a microphone of the target computing device 132. The data processing system 102 may receive a data packet generated based on the input audio signal detected by the microphone. The data packet may be filtered or unfiltered. The data packet may be a digitized version of the detected input audio signal. The data packet may include text generated by the source computing device 112 or the target computing device 132 based on the detected input audio signal. For example, the source digital assistant 114 of the source computing device 112 or the target digital assistant 142 of the target computing device 132 may process the detected input audio signal and send the data packet to the server digital assistant 108 for further processing or execution of an action based on the processed input audio signal.

[0035] The data processing system 102 may include a server digital assistant 108. The server digital assistant 108 and the NLP 106 may be a single component, or the server digital assistant 108 may include one or more components or functions of the NLP 106. The server digital assistant 108 may be docked with the NLP 106. The data processing system 102 (e.g., the server digital assistant 108) may process data packets to perform actions or otherwise respond to voice input. In some cases, the data processing system 102 may identify acoustic tags from input audio signals. The data processing system 102 may identify an electronic account corresponding to the acoustic tag based on a search in a data repository (e.g., a query database). In response to the identification of the electronic account, the data processing system 102 may establish a session and an account used in the session. An account may include a profile with one or more policies. The data processing system 102 may parse the input audio signal to identify a request and a trigger keyword corresponding to the request.

[0036] The NLP 106 may obtain an input audio signal. In response to the digital assistant detecting a trigger keyword, the NLP 106 of the data processing system 102 may receive a data packet with a speech input or an input audio signal. The trigger keyword may be a wake-up signal or a hot word that instructs the source computing device 112 to convert subsequent audio input into text and send the text to the data processing system 102 for further processing.

[0037] When receiving an input audio signal, NLP 106 can identify at least one request or at least one keyword corresponding to the request. The request can indicate the intention or theme of the input audio signal. The keyword can indicate the type of action that is likely to be taken. For example, NLP 106 can parse the input audio signal to identify at least one request to leave home at night to attend dinner and movies. The trigger keyword can include at least one word, phrase, root or part of a word, or a derivative indicating the action to be taken. For example, the trigger keyword "go" or "to go to" from the input audio signal can indicate the need for transmission. In this example, the input audio signal (or the identified request) does not directly indicate the intention for transmission, but the trigger keyword indicates that the transmission is an auxiliary action of at least one other action indicated by the request. In another example, the voice input can include a search query, such as "find jobs near me".

[0038] NLP 106 can parse the input audio signal to identify, determine, retrieve or otherwise obtain a request and one or more keywords associated with the request. For example, NLP 106 can apply semantic processing techniques to the input audio signal to identify keywords or requests. NLP 106 can apply semantic processing techniques to the input audio signal to identify keywords or phrases including one or more keywords (such as a first keyword and a second keyword). For example, the input audio signal may include the sentence "I want to buy an audio book". NLP 106 can apply semantic processing techniques or other natural language processing techniques to data packets including sentences to identify keywords or phrases "want to buy" and "audio books". NLP 106 can further identify multiple keywords, such as buy and audio books. For example, NLP 106 can determine that a phrase includes a first keyword and a second keyword.

[0039] NLP 106 can filter the input audio signal to identify trigger keywords. For example, the data packet carrying the input audio signal may include "It would be great if I could get someone that could help me goto the airport", in which case NLP 106 can filter out one or more of the following words: "it", "would", "be", "great", "if", "I", "could", "get", "someone", "that", "could", or "help". By filtering out these words, NLP 106 can more accurately and reliably identify trigger keywords (such as "goto the airport") and determine that this is a request for a taxi or ride-sharing service.

[0040] In some cases, NLP 106 can determine that the data packet carrying the input audio signal includes one or more requests. For example, the input audio signal can include the sentence "show me action movies". NLP 106 can determine that this is a request to play action movies. Server digital assistant 108 can send a request for content to content selector 110 based on the input audio signal. Server digital assistant 108 can send a request for supplementary or sponsored content from a third-party content provider. Content selector 110 can perform a content selection process to select supplementary content items or sponsored content items based on the action in the voice query. Content items can be sponsored or supplementary digital component objects. Content items can be provided by third-party content providers (such as supplementary digital content provider device 144). Supplementary content items can include advertisements for goods or services. In response to receiving a request for content from server digital assistant 108, content selector 110 can select content items using content selection criteria.

[0041] The server digital assistant 108 may receive supplemental or sponsored content items from the content selector 110. The server digital assistant 108 may receive content items in response to the request. The server digital assistant 108 may receive content items from the content selector 110 and present the content items via an audio output or a visual output. The server digital assistant 108 may provide the content items for presentation via a target computing device 132 or a source computing device 112 communicatively coupled to the target computing device 132.

[0042] The data processing system 102 may include a content selector 110, which is designed, constructed or operated to select supplementary content items (or sponsored content items or digital component objects). In order to select sponsored content items or digital components, the content selector 110 may use the generated content selection criteria to select matching sponsored content items based on broad matching, exact matching or phrase matching. For example, the content selector 110 may analyze, parse or otherwise process the subject of the candidate sponsored content items to determine whether the subject of the candidate sponsored content items corresponds to the subject of the keywords or phrases of the content selection criteria (e.g., actions or intentions). The content selector 110 may use image processing technology, character recognition technology, natural language processing technology or database search to identify, analyze or recognize the voice, audio, words, characters, text, symbols or images of the candidate digital components. The candidate sponsored content items may include metadata indicating the subject of the candidate digital components, in which case the content selector 110 may process the metadata to determine whether the subject of the candidate digital components corresponds to the input audio signal. The content campaign provided by the supplemental digital content provider device 144 may include content selection criteria that the data processing system 102 may match with the criteria indicated in the second profile layer or the first profile layer.

[0043] When establishing a content campaign that includes a digital component, the supplemental digital content provider may provide additional indicators. The supplemental digital content provider device 144 may provide information at the content campaign or content group level that the content selector 110 may identify by performing a search using information about candidate digital components. For example, a candidate digital component may include a unique identifier that may be mapped to a content group, content campaign, or content provider.

[0044] In response to the request, the content selector 110 may select a digital component object associated with a supplementary digital content provider device 144. The supplementary digital content may be provided by a supplementary digital content provider. The supplementary digital content may correspond to a service type different from the service type of the action data structure (e.g., a taxi service versus a food delivery service). The computing device 112 or 132 may interact with the supplementary digital content. The computing device 112 or 132 may receive an audio response to the digital component. The computing device 112 or 132 may receive an indication for selecting a hyperlink or other button associated with the digital component object, which enables or allows the computing device 112 or 132 to identify the supplementary digital content provider device 144, request a service from the supplementary digital content provider device 144, indicate that the supplementary digital content provider device 144 performs a service, send information to the supplementary digital content provider device 144, or otherwise query the supplementary digital content provider device 144.

[0045] The supplementary digital content provider device 144 can establish an electronic content activity. An electronic content activity can refer to one or more content groups corresponding to a common theme. A content activity can include a hierarchical data structure, which includes content groups, digital component data objects, and content selection criteria provided by a content provider. The content selection criteria provided by the content provider device 144 can include content types, such as digital assistant content types, search content types, streaming video content types, streamlined audio content types, or contextual content types. In order to create a content activity, the supplementary digital content provider device 144 can specify the value of the activity level parameter of the content activity. The activity level parameter can include, for example, an activity name, a preferred content network for placing digital component objects, resource values ​​to be used for content activities, the start and end dates of content activities, the duration of content activities, the timetable for the placement of digital component objects, language, geographic location, and the type of computing device on which digital component objects are provided. In some cases, an impression can refer to when a digital component object is obtained from its source (e.g., a data processing system 102 or a supplementary digital content provider device 144), and is countable. In some cases, robot activities can be filtered and excluded as impressions due to the possibility of click fraud. Thus, in some cases, an impression may refer to a measurement of a response from a web server to a page request from a browser, filtered from robot activity and error codes, and recorded at a point as close as possible to the opportunity of presenting a digital component object for display on a source computing device 112 or a target computing device 132. In some cases, an impression may refer to a visible or audible impression; for example, a digital component object is at least partially (e.g., 20%, 30%, 30%, 40%, 50%, 60%, 70%, or more) visible on a display device of a computing device, or audible via a speaker of a source computing device 112 or a target computing device 132. A click or selection may refer to a user interaction with a digital component object, such as a voice response to an audible impression, a mouse click, a touch interaction, a gesture, a shake, an audio interaction, or a keyboard click. A conversion may refer to a user taking a desired action with respect to a digital component object; for example, purchasing a product or service, completing a survey, visiting a physical store corresponding to the digital component, or completing an electronic transaction.

[0046] The supplemental digital content provider device 144 may also establish one or more content groups for a content campaign. A content group includes one or more digital component objects and corresponding content selection criteria, such as keywords, words, phrases, geographic locations, computing device types, time of day, interests, topics, or verticals. Content groups under the same content campaign may share the same campaign level parameters, but may have customized specifications for specific content group level parameters, such as keywords, negative keywords (e.g., blocking the placement of digital components in the presence of negative keywords on the primary content), bids on keywords, or parameters associated with bids or content campaigns.

[0047] In order to create a new content group, the content provider can provide the value of the content group level parameter of the content group. The content group level parameters include, for example, the content group name or content group theme, and the bid for different content placement opportunities (e.g., automatic placement or managed placement) or results (e.g., clicks, impressions, or conversions). The content group name or content group theme can be one or more words, and the supplementary digital content provider device 144 can use these one or more words to capture the topic or theme that the digital component object of the content group will be selected for display. For example, a car dealer can create different content groups for each brand of vehicle it sells, and can also create different content groups for each model of vehicle it sells. Examples of content group themes that car dealers can use include, for example, "A-type sports car", "B-type sports car", "C-type sedan", "C-type truck", "C-type hybrid" or "D-type hybrid". For example, the example content activity theme can be "hybrid", and includes content groups for both "C-type hybrid" and "D-type hybrid".

[0048] The supplementary digital content provider device 144 can provide one or more keywords and digital component objects to each content group. Keywords can include words related to products or services, associated with digital component objects, or identified by digital component objects. Keywords can include one or more words or phrases. For example, a car dealer can include "sports car", "V-6 engine", "four-wheel drive", "fuel efficiency" as keywords for content groups or content activities. In some cases, content providers can specify negative keywords to avoid, prevent, block or disable content placement on certain words or keywords. Content providers can specify matching types for selecting digital component objects, such as exact matching, phrase matching or broad matching.

[0049] The supplemental digital content provider device 144 may provide one or more keywords for the data processing system 102 to use to select digital component objects provided by the supplemental digital content provider device 144. The supplemental digital content provider device 144 may identify one or more keywords to bid on, and further provide bid amounts for various keywords. The supplemental digital content provider device 144 may provide additional content selection criteria used by the data processing system 102 to select digital component objects. Multiple supplemental digital content provider devices 144 may bid on the same or different keywords, and the data processing system 102 may run a content selection process or an advertising auction in response to receiving an indication of the keywords of the electronic message.

[0050] The supplementary digital content provider device 144 can provide one or more digital component objects for the data processing system 102 to select. The data processing system 102 (e.g., via the content selector 110) can select the digital component objects that match resource allocation, content scheduling, maximum bidding, keywords and other selection criteria specified for the content group when the content placement opportunity becomes available.Different types of digital component objects can be included in the content group, such as voice digital components, audio digital components, text digital components, image digital components, video digital components, multimedia digital components, digital component links or assistant application components.Digital component objects (or digital components, supplementary content items or sponsored content items) can include, for example, content items, online documents, audio, images, videos, multimedia content, sponsored content or assistant applications.When selecting a digital component, the data processing system 102 can send a digital component object to present (rendering) on ​​a source computing device 112 or a target computing device 132 or its display device.Presentation can include displaying a digital component on a display device, thereby executing an application such as a chatbot or a conversational robot, or playing a digital component via a speaker of a source computing device 112 or a target computing device 132. The data processing system 102 may provide instructions for rendering the digital component object to the target computing device 132. The data processing system 102 may instruct the target computing device 132 to generate an audio signal or sound wave.

[0051] In response to the request, the content selector 110 may perform a real-time content selection process. Real-time content selection may refer to or include performing content selection in response to the request. Real-time may refer to or include selecting content within 0.2 seconds, 0.3 seconds, 0.4 seconds, 0.5 seconds, 0.6 seconds, or 1 second of receiving the request. Real-time may refer to selecting content in response to receiving an input audio signal from the target computing device 132.

[0052] The content selector 110 may identify a plurality of candidate supplemental content items. The content selector 110 may determine a score or ranking for each of the plurality of candidate supplemental content items in order to select the highest ranked supplemental content item to provide to the computing device.

[0053] The system 100 may include, interface with, or otherwise communicate with a target computing device 132. The target computing device 132 may include a target digital assistant 142. The target digital assistant 142 may interface with the server digital assistant 108. The target digital assistant 142 may include one or more components or functions of the server digital assistant 108. The target digital assistant 142 may, for example, receive an input voice query and complete the query or request by performing an action. For example, the target computing device 132 may be a television or a smart display. The target digital assistant 142 may receive an input audio signal with the following inquiries or requests: "Turn up the volume", "Turn down the volume", "Change the channel", "Play an action movie", "Pause", "Resume", or "Turn on TV". When an input query is received, the target digital assistant 142 may interface with one or more of the controller 138 or the query processor 134 to parse the query and perform a corresponding action. The target digital assistant 142 may include one or more components or functions of the query processor 134.

[0054] The target computing device 132 may include a platform 136 that is designed, constructed, and operated to provide one or more functions of the target computing device 132. The platform 136 may refer to an operating system of the target computing device 132. The platform 136 may include a graphical user interface provided or presented by the target computing device 132. The platform 136 may be of any type and interface with the hardware of the target computing device 132. The platform 136 may interface with the query processor 134 to provide requests. The platform 136 may include, invoke, or launch an application on the target computing device 132, such as a streaming multimedia application, a video game, a music application, or a weather application.

[0055] The target computing device 132 may include a controller 138 that is designed, configured, and operated to control functions or aspects of the target computing device 132. The controller 138 may provide remote control of the target computing device 132. The controller 138 may receive signals or instructions from a hardware, physical remote control configured for use with the target computing device 132. The controller 138 may establish a communication channel with the virtual controller 120 of the source computing device 112 and receive queries from the virtual controller 120.

[0056] The controller 138 may include a gRPC service that receives input via a network 105 (such as a local WIFI network). The controller 138 may receive queries or interactions and forward the queries or interactions to the query processor 134 for further processing or completion. The query processor 134 may parse the query, determine the action to be performed, and then perform the action. For example, the query processor 134 may interface with the platform 136 to control what is displayed on the target computing device 132. The query processor 134 may interface with the data processing system 102 to determine what action to take. The query processor 134 may interface with the target digital assistant 142 to perform the action.

[0057] The query processor 134 may include one or more components or functions of the target digital assistant 142 or the NLP 106 to parse and complete the query. The query processor 134 may receive the query from the controller 138, the platform 136, or the target digital assistant 142.

[0058] The target computing device 132 may include a signal broadcaster 140 that is designed, constructed, and operated to broadcast wireless signals. The signal broadcaster 140 may broadcast wireless signals using one or more wireless protocols. The signal broadcaster 140 may broadcast signals using a short-range wireless protocol such as Bluetooth. The signal broadcaster 140 may broadcast signals using a near field communication protocol. The signal broadcaster 140 may broadcast signals using a WIFI protocol. The signal broadcaster 140 may broadcast signals using ZigBee or other wireless protocols.

[0059] Signal broadcaster 140 may broadcast or send signals continuously or based on time intervals. For example, signal broadcaster 140 may broadcast signals every 0.1 seconds, 0.2 seconds, 0.3 seconds, 0.5 seconds, 1 second, 2 seconds, 3 seconds, 4 seconds, or other time intervals.

[0060] Signal broadcaster 140 may broadcast a signal in response to a trigger condition or event. For example, signal broadcaster 140 may broadcast a signal in response to detecting motion or proximity to target computing device 132. For example, target computing device 132 may include or be coupled to a proximity sensor. The proximity sensor may use an electromagnetic field or electromagnetic radiation (e.g., infrared) beam to detect the presence of a nearby object without any physical contact and identify changes in the field or return signal. In response to detecting a change in the field or return, the proximity sensor may provide an indication to signal broadcaster 140 to broadcast a signal.

[0061] Signal broadcaster 140 may broadcast a signal with information. The information may identify target computing device 132 and the capabilities of target computing device 132. The information may identify the name (e.g., identifier) ​​or type of target computing device 132. For example, the name of target computing device 132 may be "Living Room TV," "Bedroom TV," "Study TV," "Conference Room A Display," or other names or labels established by an administrator, user, or owner of target computing device 132. The information may include or indicate the capabilities of target computing device 132. Capabilities may refer to what types of functions target computing device 132 is capable of performing. Capabilities may refer to what type of device target computing device 132 is. For example, a capability may be a smart display, a smart TV, a car unit, a networked device, or an IoT device. Capabilities may indicate functions such as multimedia streaming, movie services, music services, appliance functions, or car functions. Signal broadcaster 140 may broadcast additional information that may facilitate source computing device 112 to identify target computing device 132.

[0062] The system 100 may include, interface with, or communicate with a source computing device 112. The source computing device 112 may include or refer to a phone, a watch, a wearable device, a speaker, a headset, or other type of device. The source computing device 112 may include a source digital assistant 114. The source digital assistant 114 may include one or more components or functions of the server digital assistant 108 or the target digital assistant 142 or the NLP 106. The source digital assistant 114 may interface with or communicate with the server digital assistant 108 to complete or perform an action in response to a voice-based query received or detected by the source computing device 112.

[0063] The source computing device 112 or the target computing device 132 may include, dock or otherwise access one or more of a sensor, microphone, speaker, display device, transducer or audio driver. For example, the source computing device 112 may include a display device 122 and a microphone 124. The display device 122 may include, for example, a light indicator, a light emitting diode (LED), an organic light emitting diode (OLED), or other visual indicators configured to provide a visual or optical output. In some cases, the target computing device 132 may include a display device, such as a television display. The sensor may include, for example, an ambient light sensor, a proximity sensor, a temperature sensor, an accelerometer, a gyroscope, a motion detector, a GPS sensor, a position sensor, a microphone or a touch sensor. The transducer may include a speaker or a microphone. The audio driver may provide a software interface to the hardware converter. The audio driver may execute an audio file or other instructions to control the transducer to generate a corresponding sound wave or sound wave.

[0064] The sensor may receive or detect an input audio signal (e.g., a voice input). A digital assistant (e.g., a source digital assistant 114 or a target digital assistant 142) may be coupled to an audio driver, a transducer, and a sensor. The digital assistant may filter the input audio signal to create a filtered input audio signal (e.g., by removing certain frequencies or suppressing noise). The digital assistant may convert the filtered input audio signal into data packets (e.g., using a software or hardware digital-to-analog converter). In some cases, the digital assistant may convert the unfiltered input audio signal into data packets and send the data packets to the data processing system 102. The digital assistant may send the data packets to the data processing system 102 including one or more processors and memory that execute a natural language processor component. One or more components of the source computing device 112 or the target computing device 132 may each include at least one processing unit or other logic device, such as a programmable logic array engine, component, or module. The system 100 and its components may include hardware elements, such as one or more processors, logic devices, or circuits.

[0065] The source computing device 112 or the target computing device 132 may include at least one application, dock with it, or otherwise communicate with it. The application may be installed on the source computing device 112 or the target computing device 132. The application may be downloaded from an online application market (such as an online market provided by the data processing system 102). The application may include a local application installed on the operating system of the source computing device 112 or the target computing device 132 by the manufacturer of the source computing device 112 or the target computing device 132. The application may include any type of application that can provide resources or services. For example, the application may be a daily exercise application, a food ordering application, a taxi application, a weather application, a document processing application, a navigation application, a messaging application, a phone application, a streaming media application, a social networking application, a calendar application, a camera application, a ticket purchase application, an e-commerce application, a banking application, a financial service application, etc.

[0066] The source computing device 112 may include a data repository 126. The data repository 126 may include one or more local or distributed databases. The data repository 126 may include a computer data storage or memory and may store one or more profiles 128 or UI templates 130. The profile 128 may include a user profile, preferences, credentials, tokens, or other information that facilitates the execution of a function or the completion of a query or response. The UI template 130 may include visual or graphical user interface elements that may be used to generate prompts displayed on the display device 122.

[0067] The source computing device 112 may include a source digital assistant 114. The source computing device 112 may include a digital assistant 114, which is designed, constructed and operated to receive voice queries or other audio inputs detected by sensors of the source computing device 112, determine intent or actions based on the voice input, and cause the corresponding actions to be completed. The digital assistant 114 may include one or more components or functions of the NLP 106 or dock with the NLP 106. For example, the source digital assistant 114 may dock or communicate with the NLP 106 of the data processing system 102 to parse or process the voice input or audio input. In some cases, the source digital assistant 114 may be configured to process or parse the voice input without docking or communicating with the NLP 106 of the data processing system 102. For example, the source computing device 112 can perform digital assistant functions without communicating with the data processing system 102 via the network 105. Source computing device 112 may receive the voice query, parse the voice query, recognize the action, and invoke an application on source computing device 112 to complete the action without communicating with data processing system 102 via network 105 .

[0068] Source computing device 112 may include a presence application programming interface (API) 116 that is designed, constructed, and operated to detect a target computing device 132. Presence API 116 may detect or sense a signal broadcasted by a signal broadcaster 140 of target computing device 132. For example, source computing device 112 may be referred to as a first computing device, and target computing device 132 may be referred to as a second computing device.

[0069] The presence API 116 may be configured to listen for signals broadcast by the signal broadcaster 140 on a continuous basis, on a periodic basis, based on a time interval, or in response to an event, condition, or trigger. For example, the presence API 116 may wake up or open a communication port or search for a signal every 0.5 seconds, 1 second, 2 seconds, or other time interval. In another example, the presence API 116 may listen for broadcast signals in response to a gesture or action made by a user or wearer of the source computing device 112. For example, the source computing device 112 may be a watch, and the user may raise their wrist wearing the watch or lower their wrist in a manner that causes or triggers the presence API 116 to wake up or listen for broadcast signals.

[0070] The presence API 116 may detect a signal broadcast by the signal broadcaster 140. In response to detecting the signal, the presence API 116 may determine that the source computing device 112 is within the range of the target computing device 132. The presence API 116 may determine that the source computing device 112 is within a desired range, a threshold range, or another predetermined range of the target computing device 132. The presence API 116 may determine the range or distance between the source computing device 112 and the target computing device based on various techniques. For example, the presence API 116 may determine the distance or range from the target computing device 132 based on the signal strength broadcast by the signal broadcaster 140. The stronger the signal strength received or detected by the source computing device 112, the closer the source computing device 112 may be to the target computing device 132. If the signal strength detected at the source computing device 112 satisfies a strength threshold (e.g., amplitude or power), the presence API 116 may determine that the source computing device 112 is within the range of the target computing device 132. In some cases, source computing device 112 and target computing device 132 may exchange signals to calculate a distance between the two devices, such as based on the amount of time it takes for the signal to propagate from one device to the other (e.g., by comparing timestamps associated with the transmission and reception of the signal). In some cases, presence API 116 may determine that source computing device 112 is within range of target computing device 132 based on the type of wireless protocol being used and whether source computing device 112 is able to resolve the signal. For example, if the wireless protocol used by signal broadcaster 140 to transmit the signal is a near field communication protocol and source computing device 112 is able to process or resolve the signal, source computing device 112 may determine that target computing device 132 is within a satisfactory range.

[0071] The presence API 116 may be configured with or access one or more communication ports or sensors of the source computing device 112 to identify the signal broadcasted by the signal broadcaster 140. The presence API 116 may parse, process, or otherwise analyze the signal. The presence API 116 may parse or process the signal to identify the target computing device 132. For example, the presence API 116 may parse the signal to determine an identifier of the target computing device 132, such as the name of the target computing device 132. The signal may include one or more data packets having a header and a payload. The payload may include information about the target computing device 132 broadcasting the signal. The signal broadcaster 140 may be configured to embed or include information in the broadcasted signal. The presence API 116 may analyze the signal to determine one or more of the name of the target computing device 132, the type of the target computing device 132, or the capabilities of the target computing device 132.

[0072] The presence API 116 may determine to generate a prompt for display on the source computing device 112. The prompt may include one or more of a visual prompt, a tactile prompt, or an audio prompt. The prompt may include a display icon accompanied by one or more of an audio prompt (e.g., a beep or voice output) or tactile feedback. The presence API 116 may select the prompt type based on information in the signal broadcast by the signal broadcaster 140. The presence API 116 may access the UI template 130 data structure to identify a design or template for the prompt, and then generate the prompt based on the template. The UI template 130 may include different icons for different types of devices or capabilities. For example, if the target computing device 132 has a target digital assistant 142, the UI template 130 may include a microphone icon with a label or metadata indicating the use of the microphone icon of the target computing device 132 with digital assistant capabilities.

[0073] Signal broadcaster 140 may include in the signal information about the type of prompt to be generated. For example, the signal may include data for a visual icon corresponding to the type of target computing device 132. In some cases, presence API 116 may determine the type of prompt or visual icon to display based on the type of computing device or other information associated with the signal.

[0074] For example, presence API 116 may determine the capabilities of target computing device 132. Capabilities may include, for example, that target computing device 132 is a television that can play streaming content from one or more streaming services, can provide weather, has access to data processing system 102, is configured with target digital assistant 142, controller 138, query processor 134, includes speakers, or other capability information. Presence API 116 may determine whether to generate a prompt based on information in the signal or information otherwise obtained from target computing device 132. For example, a signal broadcast by signal broadcaster 140 may indicate the capabilities of target computing device 132 by including information in the signal indicating that target computing device 132 includes one or more of controller 138, target digital assistant 142, or query processor 134. Presence API 116 may determine to generate a prompt in response to determining that target computing device 132 includes one or more of controller 138, target digital assistant 142, or query processor 134. For example, in response to determining that the target computing device 132 includes the controller 138 and the query processor 134 , the presence API 116 may determine to generate an icon with a prompt requesting that the user whether to connect or pair with the target computing device 132 .

[0075] The presence API 116 can generate a prompt indicating that the source computing device 112 is capable of controlling the target computing device 132 based on the capabilities of the target computing device 132. If the target computing device 132 includes a controller 138 and a query processor 134, the presence API 116 can determine that the source computing device 112 can invoke a virtual controller 120 that can forward the query to the controller 138, which can be processed by the query processor 134 as if the query was detected by the target digital assistant 142 and the microphone of the target computing device 132.

[0076] In response to the prompt, source computing device 112 may receive instructions for controlling target computing device 132. The instructions may be received via a user interface of source computing device 112. The instructions may be received via a user interaction with the prompt. The user may interact with the prompt via a touch interface of source computing device 112. The user may interact with the prompt via voice input. The user may interact with the prompt via gestures. For example, the prompt may be a button or other user interface element that the user may select or click using the touch interface of source computing device 112.

[0077] If the user does not interact with the prompt, or source computing device 112 does not receive instructions to control target computing device 132, source computing device 112 may determine to remove or hide the prompt. For example, source computing device 112 may display the prompt for a duration, and if the user does not provide instructions to control target computing device 132 within the duration or time interval, source computing device 112 may determine that the user is not interested in controlling target computing device 132, and remove the icon accordingly. By removing the icon if the user does not want to control target computing device 132, source computing device 112 may reduce resource consumption, such as battery or processor consumption.

[0078] The source computing device 112 may include a pairing component 118 that is designed, constructed, and operated to pair with the target computing device 132. The pairing component 118 may perform a handshake process to pair with the target computing device 132. The pairing component 118 may pair with the target computing device 132 in response to receiving an instruction to control the target computing device 132, which may be received in response to a prompt generated by the presence API 116. The pairing component 118 may establish a communication channel with the target computing device 132 in response to the instruction.

[0079] The pairing component 118 may use one or more technologies to establish a communication channel. The pairing component 118 may establish a communication channel with one or more components of the target computing device 132. The pairing component 118 may establish a communication channel with the controller 138 of the target computing device 132. For example, the controller 138 may provide a websocket protocol. The websocket protocol may refer to or include a communication protocol configured to provide a full-duplex communication channel through a transmission control protocol (TCP). The websocket protocol may be located at layer 7 of the open systems interconnection (OSI) model. The controller 138 may be configured to communicate with the source computing device 112 using the websocket protocol and a communication port (such as HTTP ports 443 and 80). The websocket protocol may facilitate real-time data transmission.

[0080] In some cases, the controller 138 can be configured to use remote procedure calls, such as gRPC remote procedure calls. gRPC can refer to open source remote procedure calls, which can use HTTP / 2 for transport and protocol buffers as an interface description language for communication. gRPC can provide authentication, bidirectional flow and flow control, blocking or non-blocking binding, cancellation, and timeout. The controller 138 using the gRPC protocol can generate cross-platform client and server bindings for different programming languages. Therefore, the pairing component 118 can establish a communication channel as a bidirectional full-duplex communication layer over the wireless protocol.

[0081] The pairing component 118 can perform a handshake process with the controller 138 to establish a communication channel. During the handshake process, the pairing component 118 can provide authentication credentials, such as a username, password, security token, digital certificate, or other authentication information. The controller 138 can use the credentials to establish a connection with the data processing system 102 as a proxy. For example, the target digital assistant 142 can use the authentication certificate to establish a connection with the server digital assistant 108 on behalf of the user of the source computing device 112, thereby converting the target digital assistant 142 to the source digital assistant 114 of the source computing device 112. By doing so, the target digital assistant 142 can respond to and complete queries from the user of the source computing device 112. For example, the target digital assistant 142 using the authentication certificate can respond to queries such as "show me my photos" by accessing the user's photos stored in a cloud storage system in the data processing system 102 and presenting the photos via the target computing device 132.

[0082] The target computing device 132 may use the credentials to load the profile 128 of the user of the source computing device 112 in order to customize the target digital assistant 142 or configure the target digital assistant 142 to respond to queries from the user. The profile 128 information may include any profile information established by the user of the source computing device 112, including, for example, music information, media streaming information, photos, preferences, application preferences, settings, or contact information.

[0083] After the source computing device 112 has paired with the target computing device 132 or has established a communication channel with the target computing device 132 , the source computing device 112 may generate and present an icon on the display device 122 indicating the activated communication channel or communication session.

[0084] The source computing device 112 may maintain an icon on the display device 122 indicating the activated session. The icon may be a persistent icon that remains visible on the display device 122 as long as the communication session is active. The icon may be overlaid on other graphical user interface elements that may be presented on the display device 122. The source computing device 112 may maintain an icon on the display device 122 indicating that the virtual controller 120 is active to forward queries via the communication channel.

[0085] In response to establishing the communication channel or session, the source computing device 112 or the pairing component 118 may invoke the virtual controller 120 on the source computing device 112 to forward the query received by the source computing device 112 to the target computing device 132 via the communication channel to control the target computing device 132. The source computing device 112 may include a virtual controller 120 designed, constructed, and operated to forward the query to the controller 138 of the target computing device 132 via the communication session or channel established by the pairing component 118. The virtual controller 120 may receive or detect a query from a user of the source computing device 112. The query may be a voice input query detected by the microphone 124.

[0086] In some cases, the source digital assistant 114 can detect voice input and parse the voice input. The source digital assistant 114 can parse the voice input to identify a request or query. The source digital assistant 114 can query the virtual controller 120 to determine whether there is an activated communication session with the target computing device 132. The source digital assistant 114 can determine the capabilities of the target computing device 132 (e.g., based on a signal previously broadcast by the signal broadcaster 140). The source digital assistant 114 can determine that the target computing device 132 is capable of completing the request or query. The source digital assistant 114 can forward the voice input query to the virtual controller 120 and instruct the virtual controller 120 to forward the query to the controller 138. When receiving the query from the virtual controller 120, the controller 138 can forward the query to the query processor 134 of the target computing device 132. The query processor 134 can process the query as if the query was detected by a component of the target computing device 132. Thus, target computing device 132 may seamlessly process speech input detected by source computing device 112 as if target computing device 132 itself had originally detected the query.

[0087] In some cases, when there is an active communication session between the virtual controller 120 and the controller 138, the virtual controller 120 may automatically or by default forward all queries received by the source computing device 112 to the controller 138. In some cases, the virtual controller 120 may provide a prompt asking the user whether to forward the query in response to detecting the query. In some cases, the virtual controller 120 may generate a prompt with a timer that allows the user to stop or block forwarding the query to the controller 138, but forwards the query by default. Therefore, the virtual controller 120 may provide seamless forwarding of queries during an active communication session, and the query processor 134 may seamlessly process the query as if the query was originally detected or received by the target computing device 132 itself. The target computing device 132 may receive the query forwarded from the source computing device 112, and process the query to control the function of the target computing device 132.

[0088] The source computing device 112 may detect an audio input including a query via a microphone 124. The source computing device 112 may forward the audio input to the target computing device 132 via a communication channel. When there is an activated communication session with the target computing device 132, the source computing device 112 may determine to automatically forward all queries. The target computing device 132 may receive the audio input, parse the audio input to identify the query, and process the query to control the function of the target computing device 132. For example, the query may be to play an action movie, play a specific movie, display a user's photo, or provide weather. The query processor 134 may receive the input audio, parse the input audio to identify the query, and complete the query based on the profile information 128 received or accessed by the target computing device 132 in response to the authentication certificate received during the handshake process. In some cases, in order to complete the query, the query processor 134 may communicate with the data processing system 102 using the profile information. The data processing system 102 may provide, for example, a photo to the target computing device 132 for presentation.

[0089] The data processing system 102 may select and provide content items from the supplemental digital content provider 144. The data processing system 102 may generate a request for a supplemental content item. For example, the data processing system 102 may receive a request for content from the target computing device 132. The request for content may be for photos, weather, travel information, ride-sharing requests, streaming multimedia, or other requests. The data processing system 102 may receive the request from the query processor 134 or the target digital assistant 142. The target computing device 132 may generate and provide a request for content in response to receiving a query forwarded from the virtual controller 120 during an activated communication session.

[0090] Data processing system 102 may generate a second or new request for a supplemental content item. Data processing system 102 may generate a new request in response to receiving a request from target computing device 132. For example, in addition to providing the content requested by target computing device 132 (e.g., a user's photo), data processing system 102 may generate a new supplemental request for supplemental content (such as an advertising digital object). Data processing system 102 may provide the new supplemental request to content selector 110. Content selector 110 may use information associated with the original request received from target computing device 132 or profile information associated with profile 128 of source computing device 112 to perform real-time content selection profile. In response to the second request for supplemental digital content, content selector 110 may select a supplemental content item in response to the second request and based on profile information associated with source computing device 112 in response to a communication channel between source computing device 112 and target computing device 132 being activated. Data processing system 102 may provide a supplemental content item different from the requested content for display on target computing device 132. Thus, target computing device 132 may present content (eg, weather information) in response to the request forwarded by virtual controller 120 as well as supplemental content items (eg, advertisements). The supplemental content items may be related to the original request, or profile information or other information.

[0091] The communication session or channel between the source computing device 112 and the target computing device 132 may be terminated or disconnected. Terminating or disconnecting a communication channel or session may refer to interrupting the channel, ending the communication session, blocking further communications, preventing further communications, or otherwise stopping the communication of data between the source computing device 112 and the target computing device 132. However, in some cases, ending the communication session may not stop or prevent the signal broadcaster 140 from broadcasting signals that can be received or detected by the presence API 116. For example, ending the communication session may stop forwarding queries from the source computing device 112 to the target computing device 132.

[0092] One or more of the source computing device 112, the target computing device 132, or the data processing system 102 may terminate or end the communication channel. The communication channel may be terminated in response to a request to terminate the communication channel. The communication channel may be terminated based on or in response to an event, condition, or trigger. For example, the communication channel may be terminated based on a time interval or duration (e.g., 10 minutes, 15 minutes, 20 minutes, 30 minutes, or 1 hour). The communication channel may be terminated when the source computing device 112 is away from the target computing device 132 or is no longer within the range of the target computing device 132. For example, if the source computing device 112 is no longer able to detect a signal broadcasted by the signal broadcaster 140 with a satisfactory signal strength (or amplitude or power), the source computing device 112 may terminate the communication channel. The communication channel may be terminated after the query is completed. For example, the target computing device 132 may send a signal to the source computing device 112 that the query has been completed, and the source computing device 112 may determine to disconnect from the target computing device 132. In some cases, the source computing device 112 may remain connected to the target computing device 132 or maintain a persistent communication channel until a user provides instructions to terminate the communication channel, or the source computing device 112 moves out of range of the target computing device 132. In some cases, the target computing device 132 or the source computing device 112 may determine to terminate the communication channel in response to an idle timeout condition. For example, if the virtual controller 120 does not forward any queries to the controller 138 within a time interval (e.g., 10 minutes, 15 minutes, 20 minutes, 30 minutes, 1 hour, or other time interval), the source computing device 112 or the target computing device 132 may determine that the communication channel has timed out. Terminating the communication channel when not in use can reduce security vulnerabilities and resource utilization (e.g., processor, memory, or battery utilization).

[0093] Upon termination of the communication channel or the communication channel otherwise timing out, target computing device 132 may remove any authentication credentials or profile information received from source computing device 112. Target computing device 132 may erase, scrub, or delete any profile information or other information received from source computing device 112 during the communication channel. Target computing device 132 may further erase, scrub, or delete any information received from data processing system 102 to complete a query received from source computing device 112. For example, in response to a usage query for displaying photos, target computing device 132 may delete photos that may have been received from data processing system 102. Thus, target computing device 132 may remove any user information received pursuant to establishing the communication channel or received after the communication channel is established.

[0094] Figure 2 is an illustration of example operations of a system for dynamic remote control of a device according to an embodiment. Operation 200 may be performed by Figure 1 The example embodiment of the present invention may be executed by one or more systems or components depicted in the embodiment of the present invention, including, for example, source computing device 112, target computing device 132, or data processing system 102. Source computing device 112 may be a mobile device, such as a smart phone. Target computing device 132 may be a smart display or a television.

[0095] At 202, source computing device 112 may be in an initial state. The initial state may refer to a state in which source computing device 112 is not proximate to or within range of target computing device 132. During the initial state, there may be no communication channel between source computing device 112 and target computing device 132. During initial state 202, source computing device 112 may not be receiving a signal broadcasted by a signal broadcaster of target computing device 132. For example, during initial state 202, source computing device 112 may be far away from target computing device 132, such as in a different room, floor, or building.

[0096] Source computing device 112 may move toward target computing device 132. For example, a user of source computing device 112 may carry source computing device 112 close to target computing device 132, or within range of a signal being broadcast by target computing device 132. At 204, source computing device 112 may detect a signal broadcast by target computing device 132. For example, source computing device 112 may detect the signal via a presence API. In response to detecting the signal and corresponding information associated with the signal, source computing device 112 may generate a prompt 206. The prompt may indicate that source computing device 112 may be used to control target computing device 132. The prompt may be, for example, "Control your TV," where TV may refer to a television or target computing device 132.

[0097] The user of source computing device 112 may respond to prompt 206 by selecting the prompt or otherwise providing input. The input may cause source computing device 112 to establish a communication channel at 208. For example, the user may click prompt 206, and in response to the selection, source computing device 112 may perform a handshake process with target computing device 132 to establish communication channel 208.

[0098] When the communication channel is established, and in some cases, source computing device 112 may generate and present a persistent icon 210 indicating that source computing device 112 has an activated communication session with target computing device 132. If target computing device 132 has the capability to receive voice input and process voice input, the icon may be a microphone. In some cases, source computing device 112 may provide suggestions 212 about the types of functions that target computing device 132 may perform. Suggestions 212 may be generated based on the capabilities or functions of target computing device 132. Source computing device 112 may generate suggestions based on UI templates stored in a memory of source computing device 112. Source computing device 112 may receive suggestions from target computing device 132 or data processing system 102.

[0099] At 214, source computing device 112 may receive a query from a microphone of source computing device 112. The query may be "show me my pictures." At 216, source computing device 112 may determine to forward the query to target computing device 132. For example, source computing device 112 may include a virtual controller that may detect an input query and an activated communication session and then determine to forward the query to a controller of target computing device 132.

[0100] The target computing device 132 may receive the query forwarded from the source computing device 112. Due to the established communication channel, the target computing device 132 may process the query as if the target computing device 132 had received the voice query directly from the user rather than from the virtual controller. The target computing device 132 may change the default or current display 218 (e.g., a screen saver, television program, or other media currently being presented on the television) with the display of the photo 220 associated with the account or profile of the source computing device 112.

[0101] Figure 3 is a diagram of an example method of dynamic remote control of a device according to an embodiment. Method 300 may be performed by Figure 1 , a source computing device, or a target computing device. At 302, a source device may detect the presence of a target device. The source device may include a mobile computing device (such as a smart phone or smart watch) or other computing device. The source device may detect the presence by detecting a signal sent or broadcast by the target device. The target device may include, for example, a smart TV or display, or a speaker, or a networked device (such as an IoT device (e.g., a smart appliance, a speaker, or a car)).

[0102] At 304, the source device may determine the capabilities of the target device. The source device may determine that the target device includes a controller to which the query may be forwarded by the source device. The source device may determine that the target device is configured with a digital assistant. The source device may determine the capabilities based on information present in the broadcast signal or using other techniques, such as performing a lookup or web search using a device identifier or type.

[0103] At 306, the source device may determine whether the target device is compatible with the source device. The source device may determine a compatibility match based on the conditions or specifications of the source device, and the compatibility match allows the source device to forward the query to the target device for completion. For example, if the target device includes a controller configured to receive queries from a virtual controller of the source device, the source device may determine that the target device is compatible with the source device. In some cases, a compatibility match may refer to or be based on a version of a software application or program installed on the source device or the target device. In some cases, a compatibility match may be based on user preferences. For example, a profile stored on the source device may include settings or preferences for the type of target device with which a communication session is established. The user may indicate in the profile that only a communication session with a smart display or smart TV is established. Accordingly, if the target device is a smart display or smart TV, the source device may determine at 306 that there is a compatibility match. If there is no compatibility match (e.g., the device type does not match the setting or the target device does not have a controller or digital assistant that is compatible with the source device), the method may proceed to box 308 to block further presence detection of the target device.

[0104] However, if the source device determines at 306 that the target device is compatible with the source device, the source device may proceed to 310 to generate and provide a prompt. The source device may provide the prompt based on the compatibility or capability of the target device. For example, if the target device includes a digital assistant or query processor that can handle voice queries, the prompt may include a microphone. The prompt may require user authorization to establish a communication channel or communication session with the target device. The source device may present the prompt via a display of the source device. In some cases, the prompt may include or be associated with an audio prompt, a visual prompt, a tactile prompt, or other types of prompts that gain the user's attention.

[0105] At 312, the source device may receive instructions for controlling the target device. The instructions may be received via user input from the source device and in response to a prompt. The instructions may be touch input or voice input indicating that the target device is desired to be controlled. Controlling the target device may include, for example, playing a video or music on the target device, accessing information via the target device, playing a photo on the target device, or otherwise using the functions or capabilities of the target device. If the user does not input instructions for controlling the device, or provides instructions for not controlling the target device, the method may proceed to 314 to block the establishment of a communication channel. In some cases, the source device may modify the prompt. Modifying the prompt may refer to or include removing the prompt, hiding the prompt, or changing the prompt. For example, if the user indicates that a communication channel is not established, or does not provide any input within a predetermined amount of time (e.g., 1 minute, 2 minutes, 3 minutes, 4 minutes, etc.) after the prompt is generated, the source device may make the prompt smaller or faded, or move it to the background on the display.

[0106] However, if the user indicates in response to the prompt to establish a communication channel with the target device, the method can proceed to 316 to establish a communication channel with the target device. Establishing the communication channel can include performing a handshake process, sharing authentication credentials, or sharing a profile. The communication channel can be a gRPC session or other type of protocol.

[0107] At 318, the source device may invoke a virtual controller. The source device may invoke or configure the virtual controller in response to the communication session established with the target device. The virtual controller may act as a proxy input interface for the target device. For example, the virtual controller may detect voice input or queries received by the source device and then forward the queries to the target device. Upon receiving the forwarded queries, the target device may process the queries as if they were received directly from the user at the target device, thereby providing seamless virtual control of the target device via the source device.

[0108] Figure 44 is a block diagram of an example computer system 400. The computer system or computing device 400 may include or be used to implement the system 100 or its components, such as the data processing system 102, the source computing device 112, or the target computing device 132. The data processing system 102, the source computing device 112, or the target computing device 132 may include an intelligent personal assistant or a voice-based digital assistant. The computing system 400 includes a bus 405 or other communication component for transmitting information and a processor 410 or processing circuit coupled to the bus 405 for processing information. The computing system 400 may also include one or more processors 410 or processing circuits coupled to the bus for processing information. The computing system 400 also includes a main memory 415 (such as a random access memory (RAM) or other dynamic storage device) coupled to the bus 405 for storing information and instructions to be executed by the processor 410. The main memory 415 may be or include a data repository. The main memory 415 may also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 410. The computing system 400 may also include a read-only memory (ROM) 420 or other static storage device coupled to the bus 405 for storing static information and instructions for the processor 410. A storage device 425, such as a solid-state device, a magnetic disk, or an optical disk, may be coupled to the bus 405 to store information and instructions persistently. The storage device 425 may include or be part of a data repository.

[0109] The computing system 400 may be coupled to a display 435 (such as a liquid crystal display or an active matrix display) via the bus 405 for displaying information to a user. An input device 430 (such as a keyboard including alphanumeric and other keys) may be coupled to the bus 405 for communicating information and command selections to the processor 410. The input device 430 may include a touch screen display 435. The input device 430 may also include a cursor control (such as a mouse, trackball, or cursor direction keys) for communicating direction information and command selections to the processor 410 and for controlling cursor movement on the display 435. The display 435 may be, for example, a Figure 1 The data processing system 102, the source computing device 112, the target computing device 132, or other components may be part of the data processing system 102, the source computing device 112, the target computing device 132, or other components.

[0110] The processes, systems, and methods described herein can be implemented by the computing system 400 in response to the processor 410 executing the instruction arrangement contained in the main memory 415. Such instructions can be read into the main memory 415 from another computer-readable medium (such as storage device 425). The execution of the instruction arrangement contained in the main memory 415 causes the computing system 400 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement can also be used to execute the instructions contained in the main memory 415. Hard-wired circuits can be used in place of software instructions or in combination with software instructions and the systems and methods described herein. The systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0111] Despite Figure 4 An example computing system is described in the specification, but the subject matter including the operations described in this specification may be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.

[0112] For situations where the systems discussed herein collect personal information about users or can make use of personal information, users may be provided with an opportunity to control whether a program or feature can collect personal information (e.g., information about a user's social network, social actions or activities, a user's preferences, or a user's location), or to control whether or how to receive content that may be more relevant to the user from a content server or other data processing system. In addition, before storing or using certain data, it may be anonymized in one or more ways so that personally identifiable information is removed when generating parameters. For example, a user's identity may be anonymized so that personally identifiable information about the user cannot be determined, or a user's geographic location may be summarized as a location where location information is obtained (such as a city, zip code, or state level) so that the user's specific location cannot be determined. Therefore, a user may control how a content server collects and uses information about him or her.

[0113] The subject matter and operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The subject matter described in this specification may be implemented as one or more computer programs (e.g., one or more circuits of computer program instructions), which are encoded on one or more computer storage media for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Although a computer storage medium is not a propagation signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagation signal. A computer storage medium may also be or be included in one or more separate components or media (e.g., multiple CDs, disks or other storage devices). The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0114] The terms "data processing system," "computing device," "component," or "data processing apparatus" encompass various apparatuses, devices, and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or multiples or combinations of the foregoing. The apparatus may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. For example, there is an API 116, a pairing component 118, a virtual controller 120, or other components that may include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0115] A computer program (also referred to as a program, software, software application, app, script, or code) may be written in any form of programming language (including compiled or interpreted languages, declarative or procedural languages) and may be deployed in any form (including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment). A computer program may correspond to a file in a file system. A computer program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files that store one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer, or on multiple computers located at one location or distributed at multiple locations and interconnected by a communications network.

[0116] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs (e.g., components of data processing system 102) to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, and the apparatus may also be implemented as special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, by way of example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

[0117] The subject matter described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or includes a combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0118] A computing system (such as system 100 or system 400) may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network (e.g., network 105). The relationship of client and server is due to computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., data packets representing digital components) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from it). Data generated at the client device (e.g., the result of a user interaction) can be received from the client device at the server (e.g., received by the data processing system 102 from the source computing device 112 or the supplementary digital content provider device 144).

[0119] Although operations are depicted in the drawings in a particular order, such operations need not be performed in the particular order shown or in sequential order, and not all illustrated operations need be performed. The actions described herein may be performed in a different order.

[0120] The separation of various system components need not be separated in all embodiments, and the described program components may be included in a single hardware or software product. For example, the natural language processor 106 and the interface 104 may be a single component, app or program, or a logic device having one or more processing circuits, or part of one or more servers of the data processing system 102.

[0121] Now that some illustrative embodiments have been described, it is clear that the foregoing is illustrative rather than restrictive and is provided by way of example. Specifically, although many of the examples presented herein relate to specific combinations of method actions or system elements, those actions and those elements may be combined in other ways to achieve the same goals. The actions, elements, and features discussed in conjunction with one embodiment are not intended to be excluded from similar roles in other one or more embodiments.

[0122] The phraseology and words used herein are for descriptive purposes and should not be considered limiting. As used herein, "including," "containing," "having," "comprising," "involving," "characterized by," "characterized by," and variations thereof are meant to encompass the items listed thereafter, their equivalents and additional items, and alternative embodiments consisting of the items specifically listed thereafter. In one embodiment, the systems and methods described herein include one, each combination of more than one, or all of the described elements, actions, or components.

[0123] Any reference to the embodiments or elements or actions of the systems and methods mentioned herein in the singular may also include embodiments comprising a plurality of these elements, and any reference to any embodiment or element or action in the plural form herein may also include embodiments comprising only a single element. References in the singular or plural form are not intended to limit the currently disclosed systems or methods, their components, actions or elements to single or multiple configurations. References to any actions or elements based on any information, action or element may include embodiments wherein the action or element is based at least in part on any information, action or element.

[0124] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment," "some embodiments," "an embodiment," etc. are not necessarily mutually exclusive, but are intended to indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment or example. These words used herein do not necessarily all refer to the same embodiment. Any embodiment may be combined with any other embodiment, either inclusively or exclusively, in any manner consistent with the aspects and embodiments disclosed herein.

[0125] References to "or" may be interpreted as inclusive, such that any term described using "or" may indicate any of a single, more than one, and all of the described terms. References to at least one of a combined list of terms may be interpreted as an inclusive "or" to indicate any of a single, more than one, and all of the described terms. For example, references to "at least one of A and B" may include only A, only B, and both A and B. Such references used in conjunction with "including" or other public terms may include additional items.

[0126] Where a technical feature in the drawings, detailed description or any claim is followed by a reference sign, the reference sign has been included to increase the intelligibility of the drawings, detailed description and claims. Accordingly, the presence or absence of a reference sign has no limiting effect on the scope of any claim element.

[0127] The systems and methods described herein may be implemented in other specific forms without departing from their characteristics. The foregoing embodiments are illustrative rather than limiting of the systems and methods described. Accordingly, the scope of the systems and methods described herein is indicated by the appended claims rather than the preceding description, and changes within the meaning and range of equivalents of the claims are included therein.

Claims

1. A system for remote control of a device, comprising: A first computing device, including one or more processors and a memory, the first computing device configured to: Detect the second computing device located within the wireless communication range of the first computing device by using a presence application programming interface (API) based on the strength of one or more signals exchanged between the first computing device and the second computing device, wherein the strength of the one or more signals indicates the distance between the first computing device and the second computing device: Identify, by the presence API, the signal broadcast by the signal broadcaster of the second computing device; Analyze, by the presence API, the signal; And Based on the analysis of the signal, determine at least one of the name of the second computing device, the device type of the second computing device, or the capabilities of the second computing device; In response to detecting the second computing device and based on the one or more signals exchanged between the first computing device and the second computing device, determine the capabilities of the second computing device, wherein the capabilities of the second computing device are used to complete a request or query based on the second computing device including at least (i) a controller and (ii) a query processor; Based on the capabilities of the second computing device to complete the request or query, generate a prompt indicating that the first computing device is capable of controlling the second computing device; In response to the prompt, receive an instruction for controlling the second computing device based on the input received by the first computing device; In response to the instruction, automatically establish a communication channel with the second computing device; Invoke a virtual controller on the first computing device to forward the query received by the first computing device to the second computing device via the communication channel to control the second computing device; Receive an input including a natural language query from a sensor of the first computing device, wherein the sensor includes a microphone; and Send the natural language query to the controller of the second computing device, wherein the query is processed by at least one of the controller or the query processor of the second computing device to perform an action associated with the natural language query.

2. The system according to claim 1, wherein The second computing device broadcasts information about the second computing device, including that the first computing device is configured to: Receive the information broadcast by the second computing device; And Based on the information, determine the capabilities of the second computing device.

3. The system according to claim 1, comprising: The first computing device is configured to use one or more security certificates to perform a handshake process with the second computing device to establish the communication channel.

4. The system according to claim 1, comprising: The first computing device is configured to maintain an icon on the display device of the first computing device, the icon indicating that the virtual controller is activated to forward queries via the communication channel.

5. The system according to claim 1, comprising: The second computing device is configured to access a profile associated with the first computing device in response to the establishment of the communication channel.

6. The system according to claim 1, comprising: The first computing device is configured to disconnect from the second computing device and terminate the communication channel, wherein after the establishment of the communication channel, the second computing device removes any profile information received from the first computing device.

7. The system according to claim 1, comprising: The first computing device is configured to establish the communication channel as a two-way communication layer via a wireless protocol.

8. The system according to claim 1, comprising: The first computing device is configured to establish the communication channel with the second computing device using the websocket protocol.

9. The system according to claim 1, comprising: The second computing device is configured to receive a query forwarded from the first computing device and process the query to control the functions of the second computing device.

10. The system according to claim 1, comprising: The first computing device is configured to detect an audio input including a query via a microphone of the first computing device and forward the audio input to the second computing device via the communication channel; and The second computing device is configured to receive the audio input, parse the audio input to identify the query, and process the query to control the functions of the second computing device.

11. The system according to claim 1, comprising: The second computing device is configured to send a request for content to a data processing system remote from the first computing device and the second computing device; and The data processing system is configured to: generate a second request for supplementary digital content different from the request in response to the request for content; select, in response to the second request for supplementary digital content, a supplementary content item in response to the second request and based on profile information associated with the first computing device in response to the communication channel between the first computing device and the second computing device being active; and provide the supplementary content item different from the request for display on the second computing device.

12. A method for remote control of a device, comprising: detecting, by a first computing device including one or more processors and a memory, a second computing device within a wireless communication range of the first computing device using a presence application programming interface (API) based on intensities of one or more signals exchanged between the first computing device and the second computing device, the intensities of the one or more signals indicating a distance between the first computing device and the second computing device, by: identifying, by the presence API, a signal broadcast by a signal broadcaster of the second computing device; analyzing, by the presence API, the signal; and determining, based on analyzing the signal, at least one of a name of the second computing device, a device type of the second computing device, or a capability of the second computing device; The first computing device determines the capabilities of the second computing device in response to detecting the second computing device and based on the one or more signals exchanged between the first computing device and the second computing device, where the capabilities of the second computing device are used to complete a request or query based on the second computing device including at least (i) a controller and (ii) a query processor; The first computing device generates a prompt indicating that the first computing device is capable of controlling the second computing device based on the capabilities of the second computing device to complete the request or query; The first computing device receives an instruction for controlling the second computing device in response to the prompt and based on an input received by the first computing device; The first computing device automatically establishes a communication channel with the second computing device in response to the instruction; The first computing device invokes a virtual controller on the first computing device to forward a natural language query received by the first computing device to the second computing device via the communication channel to control the second computing device; Receiving an input including a natural language query from a sensor of the first computing device, where the natural language input includes a natural language query obtained from a sensor of the first computing device including a microphone; and Sending the natural language query to the controller of the second computing device, where the query is processed by at least one of the controller or the query processor of the second computing device to perform an action associated with the natural language query.

13. The method according to claim 12, wherein, the second computing device broadcasts information about the second computing device, including: The first computing device receives the information broadcast by the second computing device; and The first computing device determines the capabilities of the second computing device based on the information.

14. The method according to claim 12, including: The first computing device performs a handshake process with the second computing device using one or more security certificates to establish the communication channel.

15. The method according to claim 12, including: The first computing device maintains an icon on a display device of the first computing device, where the icon indicates that the virtual controller is activated to forward queries via the communication channel.

16. The method according to claim 12, including: The second computing device accesses a profile associated with the first computing device in response to the establishment of the communication channel.

17. The method according to claim 12, including: The first computing device disconnects from the second computing device and terminates the communication channel, where after the establishment of the communication channel, the second computing device removes any profile information received from the first computing device.

18. The method according to claim 12, including: The first computing device establishes the communication channel as a two-way communication layer via a wireless protocol.

19. The method according to claim 12, including: Receive a query forwarded from the first computing device by the second computing device, and process the query to control the functions of the second computing device.

Images