Aggregating information from different data feed services

Abstract

Implementations are described herein for aggregating information responsive to a query from a plurality of different data feed services using machine learning. In various implementations, NLP may be performed on a natural language input that includes a query for information to generate a data feed independent aggregator embedding (FAAE). A plurality of data feed services may be selected, each data feed service having its own data feed service action space including actions executable to access data through the data feed service. The FAAE may be processed based on a domain-specific machine learning model corresponding to the selected data feed service. Each domain-specific machine learning model may translate between a respective data feed service action space and a data feed independent semantic embedding space. Using these models, actions may be selected from a data feed service action space and performed to aggregate data responsive to queries from a plurality of data feed services.

Description

Background Art

[0001] Individuals often seek semantically related information from multiple different sources in a habitual manner. For example, a particular individual may tend to check the same data feed service, such as a news website and / or a social network data feed, to obtain different perspectives on a particular topic, such as a developing event, a new movie, a sporting event, etc. However, in addition to the specific information that the individual seeks, various data feed services can also transmit countless other information that is irrelevant to the individual at the moment. Therefore, it may be troublesome and / or time-consuming for individuals to screen multiple different data feed services to obtain the specific information they are looking for. This problem may be amplified when interacting with a voice user interface (VUI) provided by a computing device such as a smart speaker or an in-vehicle voice command system. In addition, personal data feed services routinely change how to present data and / or make data accessible, which may further hinder the individual's efforts. Summary of the invention

[0002] This article describes an implementation for using machine learning to aggregate information responsive to a query from multiple different data feed services. More specifically, but not exclusively, this article describes an implementation for using a domain-specific machine learning model to embed query information from different data feed services using a data feed-agnostic aggregator. The embodiments described herein produce various technical advantages. Instead of navigating multiple different data feed services to obtain information about a topic, an individual can simply issue a free-form natural language query that describes what information the individual is looking for, and, where applicable, describes where the individual is looking for the information. Response information from multiple different data feed services can then be obtained and presented to the individual together, for example, as part of an aggregated data feed. Therefore, an individual can obtain different perspectives on a topic and / or multiple views on a topic from another person across multiple different data feed services.

[0003] In some implementations, a method may be implemented using one or more processors and may include: obtaining a natural language input comprising a query for information; performing natural language processing (NLP) on the natural language input to generate a data feed-independent aggregator embedding; selecting a plurality of data feed services, wherein each of the selected plurality of data feed services includes its own data feed service action space, wherein each data feed service action space includes actions executable to access data transmitted through the corresponding data feed service; processing the feed-independent aggregator embedding based on a plurality of domain-specific machine learning models corresponding to the plurality of data feed services, wherein each domain-specific machine learning model is trained to convert between the corresponding data feed service action space and a data feed-independent semantic embedding space including the data feed-independent aggregator embedding; based on the above processing, selecting and executing one or more actions from each of the data feed service action spaces to aggregate data responsive to the query from the plurality of data feed services; and causing the aggregated response data to be presented as output.

[0004] In various implementations, multiple data feed services may be selected based on entity identifiers included in a query for information. In various implementations, the query for information may include a request for social media posts from a specific individual, and the selected multiple data feed services may include two or more social media services, which are selected based on the membership of the specific individual with the two or more social media services. In various implementations, for a given social media service among the two or more social media services, one or more actions selected from the data feed service action space of the given social media service may include accessing one or more posts of the specific individual from the posting history of the specific individual. In various implementations, for a given social media service among the two or more social media services, one or more actions selected from the data feed service action space of the given social media service may include: filtering one or more posts of the specific individual from a general data feed on the given social media service provided to the user issuing the natural language input.

[0005] In various implementations, the plurality of data feed services may be selected based on a lookup table controlled by the user issuing the natural language input. In various implementations, the lookup table may include a contact list of the user. In various implementations, the plurality of data feed services may be selected based on that the user issuing the natural language input has previously provided the aggregator agent with permission to access the plurality of data feed services. In various implementations, at least one of the data feed services may include a virtual space that forms part of a larger metaverse that includes the plurality of virtual spaces.

[0006] In various implementations, the aggregated response data can be presented to the user issuing the natural language input as part of a metaverse graphical user interface. In various implementations, multiple data feed services can be selected based on the browsing history of the user issuing the natural language input.

[0007] In addition, some implementations include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause performance of any of the foregoing methods. Some implementations include at least one non-transitory computer-readable storage medium storing computer instructions executable by the one or more processors to perform any of the foregoing methods.

[0008] It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a schematic diagram of an example environment in which implementations disclosed herein may be implemented.

[0010] Figure 2 Examples of how data may be exchanged and / or processed to perform selected aspects of the present disclosure are schematically illustrated according to various implementations.

[0011] Figure 3 Another example of how data may be processed to query multiple domains using data feed-agnostic aggregator embeddings according to various implementations is schematically illustrated.

[0012] Figure 4 is a flow chart illustrating an example method of practicing selected aspects of the present disclosure according to implementations disclosed herein.

[0013] Figure 5 An example architecture for a computing device is shown. DETAILED DESCRIPTION

[0014] This document describes an implementation for aggregating information responsive to a query from multiple different data feed services using machine learning. More specifically, but not exclusively, this document describes an implementation for utilizing a domain-specific machine learning model to query information from different data feed services using a data feed-independent aggregator embedding. The embodiments described herein produce various technical advantages. Instead of navigating multiple different data feed services to obtain information about a topic, an individual can simply issue a free-form natural language query that describes what information the individual is looking for, and, where applicable, where the individual is looking for the information. Response information from multiple different data feed services can then be obtained and presented to the individual together, for example, as part of an aggregated data feed. Therefore, an individual is able to obtain different perspectives on a topic and / or multiple views on a topic from another person across multiple different data feed services.

[0015] As used herein, a "data feed service" may be any computer-implemented service, such as a web service, that can be used by an individual to publish, broadcast, push, and / or multicast content to others. A data feed service is typically accessible by multiple individuals, and is typically updated frequently or at least periodically with new content, which is typically presented in reverse chronological order. A common example of a data feed service is a social networking service on which individual users can post information (e.g., facts, opinions, pictures, statuses) to their "friends" or "followers," or anyone else who accesses the social networking service. Another example of a data feed service is a news website (or more generally, a news service that can publish to both websites and proprietary applications) that continuously publishes news stories, opinions, columns, and the like.

[0016] These examples of data feed services are not meant to be limiting. Other examples of data feed services include, but are not limited to, weblogs (which are more commonly referred to as "blogs"), web feeds (e.g., Really Simple Syndication or "RSS" feeds), comment aggregation sites, social news sites featuring user-posted content, and the like. And data feed services are not limited to services that contribute content to websites. For example, a data feed service may take the form of a virtual space (e.g., a massively multiplayer online role-playing game ("MMORPG") or a portion thereof, a virtual coffee shop or forum, etc.) that forms part of a larger metaverse that includes multiple virtual spaces.

[0017] In order to obtain information in response to a single natural language query from multiple data feed services at once, natural language processing (NLP) may be performed on the provided natural language query to generate a semantic representation referred to herein as a "data feed-independent aggregator embedding." The data feed-independent aggregator embedding may represent the semantics contained in the natural language query in the abstract. In some instances, the data feed-independent aggregator embedding may be a dense numeric representation, such as a vector of real numbers, which serves as an embedding in a continuous vector space.

[0018] In order to utilize data feed agnostic aggregator embedding to obtain information from multiple sources at once, multiple data feed services that potentially contain information responsive to a personal query may be selected. The data feed services may be selected on a query-by-query basis and / or across multiple different queries. Potentially responsive data feed services may be selected in various ways based on various signals.

[0019] In some implementations, a data feed service can be selected based on a lookup table associated with the individual who issues the query. As an example, an individual may have a contact list (e.g., a phone book, friends on social media, etc.) that specifies which data feed services the individual's contacts use, and therefore specifies which data feed services to select. As another example, an individual may provide explicit permission to what will be referred to as an "aggregator agent" in this article to obtain response information from an enumeration or a selected list of data feed services. As another example, in general situations and / or in specific situations, an individual's browsing history may be consulted to determine which data feed services the individual tends to consult with or without input from the individual. For example, if an individual issues a sports-related query, a sports-related data feed service that the individual has historically visited may be selected. In some implementations, a data feed service may be selected based on an entity (person, place, or thing) identified in the query. For example, an individual may request information about an event that is developing that is transmitted (e.g., published, released, created) by a specific media expert. A data feed service used by the specific media expert may be selected.

[0020] Each data feed service may include its own data feed service action space, which includes actions that can be executed to access data transmitted by the data feed service. The actions of such action space may include, for example, actions that can be performed using input devices (such as keyboards and pointer devices) to navigate a graphical user interface (GUI) provided by the data feed service. For example, if the GUI takes the form of an interactive web page, the action space may include actions that can be performed using graphical inputs (such as fields, drop-down menus, buttons, etc.) that are presented as part of the interactive web page. In addition or alternatively, the action may include commands, queries, and / or parameters that can be used to navigate the GUI or VUI provided by the data feed service to obtain specific information. For example, the interface (GUI or VUI) of a particular data feed service may include a search field that facilitates the submission of natural language queries to quickly obtain responsive content, and / or a filter field that narrows down search results.

[0021] Domain-specific machine learning models configured with selected aspects of the present disclosure can be trained to convert between these action spaces and data feed-independent semantic embedding spaces including the above-mentioned data feed-independent aggregator embeddings. These domain-specific machine learning models (or referred to as "domain models" elsewhere in this article) can take various forms, such as various types of neural networks, transformers, RNNs, graph-based neural networks, etc. In various implementations, domain-specific machine learning models can be used to process data feed-independent aggregator embeddings to generate one or more probability distributions over actions in the action space of the data feed service. Based on these probability distributions, actions can be selected and executed to obtain content from the data feed service in response to queries semantically represented by the data feed-independent aggregator embeddings. Response content from the data feed service can be aggregated with response content obtained from other data feed services in a similar manner. The aggregated response content can then be presented to the individual who issued the original natural language input.

[0022] As used herein, "domain" may refer to a target subject area in which a computing component is intended to operate, for example, the scope of knowledge, influence, and / or activity around which the logic of a computing component operates. In some implementations, the domain to which a query is to be submitted may be identified by matching the keywords in the input provided by the user with the domain keywords heuristically. In other implementations, for example, NLP techniques (such as word2vec, bidirectional encoder representation (BidirectionalEncoder Representation from Transformers, BERT) transformers, various types of recurrent neural networks ("RNN", such as long short-term memory or "LSTM", gated recurrent unit or "GRU"), etc.) may be used to process the input provided by the user to generate a semantic embedding representing a natural language input. In some implementations, the natural language input semantic embedding, which as described above may be referred to as "data feed-independent aggregator embedding", may be used, for example, to identify one or more domains based on the distance in the embedding space (or vector space) between the data feed-independent aggregator embedding and other embeddings associated with various domains. These distances in the embedding space may be calculated using techniques such as Euclidean distance, dot product, cosine similarity, etc.

[0023] In various implementations, one or more domain models may have been previously generated for each domain. For example, one or more machine learning models—such as RNNs (e.g., LSTM, GRU), BERT transformers, various types of neural networks, reinforcement learning strategies, etc.—may be trained based on a corpus of documents associated with the domain. As a result of this training, one or more of the domain models may be at least guided so that they can be used to process what will be referred to herein as "domain-independent aggregator embeddings" to generate one or more probability distributions over an action space associated with a target domain. Based on these probability distributions, multiple actions may be selected and executed to execute a user-submitted query in the target domain.

[0024] Figure 1 Example environments are schematically described in which selected aspects of the present disclosure may be implemented according to various implementations. Figure 1 Any computing device depicted in or elsewhere in the figures may include logic components, such as one or more microprocessors (e.g., central processing units or “CPUs,” graphics processing units or “GPUs,” tensor processing units or “TPUs”) that execute computer-readable instructions stored in memory, or other types of logic components, such as application specific integrated circuits (“ASICs”), field programmable gate arrays (“FPGAs”), etc. Figure 1Some systems depicted in the present invention, such as the inter-domain knowledge system 102, can be implemented using one or more server computing devices that form what is sometimes referred to as a "cloud infrastructure," although this is not required. In other implementations, for example, various aspects of the inter-domain knowledge system 102 can be implemented on a client device 120 for purposes of protecting privacy, reducing latency, etc.

[0025] The inter-domain knowledge system 102 may include a number of different components configured with selected aspects of the present disclosure, such as a domain module 104, an interface module 106, and a machine learning ( Figure 1 The inter-domain knowledge system 102 may also include any number of databases for storing machine learning model weights and / or other data for performing selected aspects of the present disclosure. Figure 1 In, for example, the inter-domain knowledge system 102 includes a database 110 storing a global domain model and another database 112 storing data indicating global action embeddings.

[0026] The inter-domain knowledge system 102 may be operatively coupled to any number of client computing devices operable by any number of users via one or more computer networks (114). Figure 1 In the example, a first user 118-1 operates one or more client devices 120-1, e.g., a coordinated ecosystem of client devices controlled by user 118-1 and / or associated with an online profile of user 118-1. A first user 118-P operates one or more client devices 120-P. As used herein, client devices 120 may include, for example, one or more of the following: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (which in some cases may include a visual sensor and / or a touch screen display), a smart appliance such as a smart TV (or a standard TV equipped with a networked dongle with automated assistant capabilities), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or optional client computing devices may be provided.

[0027] The domain module 104 may be configured to determine various different information about domains (e.g., data feed services) that are relevant to a given user 118 at a given point in time, such as data feed services that the user 118 is currently participating in, data feed services that the user would like to query for information, etc. To this end, the domain module 104 may collect contextual information about, for example, foreground and / or background applications executing on a client device 120 operated by the user 118, web pages currently / recently visited by the user 118, domains that the user 118 has access rights to and / or frequently visits, etc.

[0028] Using the collected context information, in some implementations, the domain module 104 can be configured to identify one or more domains (e.g., data feed services) related to the natural language input provided by the user. For example, a user-written query seeking response information from multiple different data feed services can be processed by the domain module 104 to identify the data feed service that the user 118 intended to query.

[0029] In some implementations, the domain module 104 may also be configured to retrieve domain knowledge from a variety of different sources associated with the identified domain. In some such implementations, for example, in addition to the previously mentioned natural language input or context information, the retrieved domain knowledge (and / or embeddings generated therefrom) may be provided to downstream components. This additional domain knowledge may allow downstream components (particularly machine learning models) to make predictions that are more likely to be satisfactory (e.g., aggregating response information from multiple different domains).

[0030] In some implementations, the domain module 104 may apply the collected contextual information (e.g., current state) across one or more “domain selection” machine learning models 105 that are distinct from the domain models described herein. These domain selection machine learning models 105 may take various forms, such as various types of neural networks, support vector machines, random forests, BERT transformers, and the like. In various implementations, the domain selection machine learning models 105 may be trained to select applicable domains based on properties (or “context signals”) of the current context or state of the user 118 and / or client device 120. For example, if the user 118 is operating an input form of a particular website to obtain goods or services, the uniform resource locator (URL) of the website or properties of the underlying web page (such as keywords, tags, document object model (DOM) elements, and the like) may be applied across the model inputs in their native form or as reduced dimensionality embeddings. Other contextual signals that may be considered include, but are not limited to, the user's IP address (e.g., work vs. home vs. mobile IP address), time-of-day, social media status, calendar, email / text messaging content, etc.

[0031] The interface module 106 may provide one or more GUIs or VUIs that may be operated by different individuals, such as users 118-1 through 118-P, to perform various actions provided by the semantic task automation system. In various implementations, the user 118 may operate a GUI (e.g., a standalone application or web page) provided by the interface module 106 to opt-in or opt-out of using the various techniques described herein. For example, the users 118-1 through 118-P may be required to provide explicit permission for each data feed service (more generally, domain) that they wish to query before their search requests are used to retrieve response information from those data feed services.

[0032] ML module 108 has access to data indicative of various global domain / machine learning models / strategies in database 110. These trained global domain / machine learning models / strategies can take various forms, including but not limited to graph-based networks (such as graph neural networks (GNNs), graph attention neural networks (GANNs), or graph convolutional neural networks (GCNs)), sequence-to-sequence models (such as encoder-decoders), various forms of recurrent neural networks (e.g., LSTMs, GRUs, etc.), BERT transformer networks, reinforcement learning strategies, and any other type of machine learning model that can be applied to facilitate selected aspects of the present disclosure. ML module 108 can process various data based on these machine learning models at the request or command of other components such as domain module 104 and / or interface module 106.

[0033] Each client device 120 may operate at least a portion of what will be referred to herein as an "aggregator proxy" 122. Aggregator proxy 122 may be a computer application that may be operated by user 118 to perform selected aspects of the present disclosure to facilitate inter-domain data aggregation as described herein. For example, aggregator proxy 122 may receive requests from user 118 and / or may aggregate query response data from multiple different domains. In some implementations, aggregator proxy 122 may be operable to grant access rights to domains controlled by the user to others (e.g., other aggregator proxies 122). For example, user 118-1 may interact with aggregator proxy 122-1 to allow specific other individuals to access a portion of a domain controlled by user 118-1, such as the user's own social network profile feed. Without such explicit permission, other aggregator proxies 122 may be unable to retrieve response information from the user's social network profile feed.

[0034] In some implementations, the aggregator agent 122 may take the form of what is commonly referred to as a "virtual assistant" or "automated assistant" that is configured to participate in a human-computer interaction natural language conversation with the user 118. For example, the aggregator agent 122 may be configured to semantically process natural language input provided by the user 118 to identify one or more intents. Based on these intents, the aggregator agent 122 may perform various tasks, such as operating smart appliances, retrieving information, performing tasks, and the like. In some implementations, the conversation between the user 118 and the aggregator agent 122 (or a separate automated assistant accessible to / by the aggregator agent 122) may constitute an action sequence, which may be captured, abstracted into a domain-independent embedding, and then extended to other domains, as described herein.

[0035] exist Figure 1 , each of the client devices 120-1 may include an aggregator agent 122-1 that serves the first user 118-1. The first user 118-1 and its aggregator agent 122-1 have access to and / or may be associated with a "profile" that includes various data related to performing selected aspects of the present disclosure on behalf of the first user 118-1. For example, the aggregator agent 122 has access to one or more edge databases or data stores associated with the first user 118-1, including an edge database 124-1 that stores a local domain model and / or another edge database 126-1 that stores recorded actions. Other users 118 may have similar arrangements. Any data stored in the edge databases 124-1 and 126-1 may be stored in part or in whole on the client device 120-1, for example, to protect the privacy of the first user 118-1. For example, the recorded actions 126 - 1 may be stored locally on the client device 120 - 1 in its original form, and the recorded actions 126 - 1 may include sensitive and / or personal information of the first user 118 - 1 , such as payment information, address, phone number, etc.

[0036] The local domain models stored in the edge database 124-1 may include, for example, local versions of global models stored in the global domain model database 110. In some implementations, the global models may be propagated to the edges in order to guide the aggregator agent 122 to expand tasks into new domains associated with those propagated models; thereafter, the local models at the edges may or may not be trained locally based on the activities and / or feedback of the users 118. In some such implementations, the local models (also referred to as "local gradients" in the edge database 124) may be periodically used to train the global models (in the database 110), for example, as part of a federated learning framework. As the global model is trained based on the local models, the global model may be propagated back to other edge databases (124) in some cases, thereby keeping the local models up to date.

[0037] However, it is not required to adopt federated learning in all implementations. In some implementations, the aggregator agent 122 can provide cleaned data to the inter-domain knowledge system 102, and the ML module 108 can remotely apply the model to the cleaned data. In some embodiments, the "cleaned" data can be data from which sensitive and / or personal information has been removed and / or obfuscated. In some implementations, personal information can be cleaned by the aggregator agent 122, for example, at the edge, based on various rules. In other implementations, the cleaned data provided to the inter-domain knowledge system 102 by the aggregator agent 122 can be in the form of a dimensionality reduction embedding generated from the original data at the client device 120.

[0038] As previously described, the edge database 126-1 can store actions logged by the aggregator agent 122-1. The aggregator agent 122-1 can log actions in a variety of different ways, depending on the level of access the aggregator agent 122-1 has to computer applications executing on the client device 120-1 and the permissions granted by the user 118. For example, most smartphones include an operating system (OS) interface for providing or revoking permissions (e.g., location, access to the camera, etc.) to various computer applications. In various implementations, such an OS interface can be operated to provide / revoke access rights to the aggregator agent 122, and / or select a specific level of access that the aggregator agent 122 will have to specific computer applications and / or domains to which those applications provide access rights.

[0039] Aggregator agent 122-1 can have various levels of access to how computer applications work, depending on the permissions granted by user 118 and the cooperation of the software developer providing the computer application. For example, with the permission of user 118, some computer applications can provide aggregator agent 122 with "behind-the-scenes" access to the application's API or scripts written in a programming language (e.g., macros) embedded in the computer application. Other computer applications may not provide as much access. In such cases, aggregator agent 122 can record actions in other ways, such as by capturing screenshots, performing optical character recognition (OCR) on those screenshots to identify menu items, and / or monitoring user input (e.g., interrupts captured by the OS) to determine which graphical elements are operated by the user 118 in which order. In some implementations, aggregator agent 122 can capture actions performed using a computer application from data exchanged between the computer application and the underlying OS (e.g., through system calls). In some implementations, the aggregator agent 122 may capture and / or have access to data exchanged between or used by window managers and / or window systems.

[0040] Figure 2 Schematically depicting examples of how data may be processed by and / or using various components across domains. Starting from the top left, a user 118 operates a client device 120 to provide a typed or spoken natural language query NL query 1. In the latter case, the spoken utterance may first be processed using a speech-to-text (STT) engine (not depicted) to generate a speech recognition output. In either case, NL query 1 may be provided to an aggregator agent 122.

[0041] The aggregator agent 122 may process or cause the ML module 108 to process data indicating the NL query 1 to generate a data feed-agnostic aggregator embedding (DAAE) Q1'. In some implementations, the aggregator agent 122 and / or the ML module 108 may generate the DAAE Q1' using a machine learning model such as a transformer network or a recurrent neural network (e.g., LSTM, GRU). The DAAE Q1' may then be processed, for example, by the aggregator agent 122 or the ML module 108 using a domain model B associated with a first data feed service and a domain model C associated with a second data feed service. Processing of the DAAE Q1' using the domain model B generates a probability distribution, which in turn may be used to select a plurality of actions {B1, B2, ...} from a first data feed service action space associated with the first data feed service. Similarly, processing of the DAAE Q1' using the domain model C results in selecting a plurality of actions {C1, C2, ...} from a second data feed service action space associated with the second data feed service.

[0042] These selected actions {B1, B2, ...} and {C1, C2, ...} may be performed by various components to retrieve information responsive to the original query NL Query 1 from the respective domains of the first and second data feed services. Figure 2 , for example, the selected actions {B1, B2, . . .} and {C1, C2, . . .} are provided to the client device 120 by the aggregator proxy 122. The client device 120 may then perform the selected actions, for example, through corresponding applications installed and running on the client device 120.

[0043] For example, if a first data feed service associated with domain model B is a social networking service, a compatible social networking application (e.g., client) operating on client device 120 may automatically perform actions {B1, B2, ...} to retrieve response data from the social networking service. Similarly, if a second data feed service associated with domain model C is a blog curated by a second person, a compatible client application (e.g., web browser) operating on client device 120 may automatically perform actions {C1, C2, ...} to retrieve response data from the blog. User 118 may or may not be able to see these actions automatically performed in the GUI rendered by the client application.

[0044] The quality and / or responsiveness of the aggregate information returned in response to a user's initial query may depend, at least in part, on the specificity of the query. Ambiguous queries may result in actions that do not actually perform the user's intent, for example, because the resulting aggregate information may have limited value and / or because the actions selected and performed with respect to different data feed services may be inconsistent, which may result in inconsistent response information retrieved from different data feed services. Figure 2 For example, if NL query 1 is vague and / or ambiguous, the selected actions {B1, B2, ...} and {C1, C2, ...} may obtain significantly different results from each data feed service. On the other hand, queries that are more detailed and clearly state the user's intent may produce better results. However, if the burden on the user to be specific and clear is too high, the user may prefer to manually aggregate information from multiple different data feed services.

[0045] Thus, in some implementations, users may be able to record actions they perform with respect to a particular data feed service to obtain response information, and associate those actions with custom natural language sentences they provide. In other words, the recorded actions, rather than sentences, contain contextual information that can be used to perform similar actions using other data feed services. In this way, it is possible to associate natural language sentences that would otherwise lack detail (e.g., "long tail" natural language sentences) with specific actions that can be used to aggregate information from multiple different data feed services.

[0046] Figure 2 An example of this is depicted in . Below the dashed line, the user 118 operates the client device 120 to request and / or allow recording of actions performed by the user 118 using the client device 120 that are associated with another natural language query NL query 2. In various implementations, the aggregator agent 122 cannot record the actions without receiving this permission. In some implementations, this permission may be granted on an individual data feed service basis and / or on an application-by-application basis, in much the same way that applications are granted permission to access GPS coordinates, local files, use a built-in camera, and the like. In other implementations, this permission may be granted only before the user 118 says otherwise, for example, by pressing a "stop recording" button similar to recording a macro or by providing voice input such as "stop recording" or "that's it."

[0047] Once the request / permission is received, in some implementations, the aggregator proxy 122 may acknowledge (ACK) the request / permission. Next, a sequence of actions {B3, B1, ...} and {C5, C2, ...} performed by the user 118 using the client device 120 may be captured and stored in the edge database 126. These actions {B3, B1, ...} and {C5, C2, ...} may take various forms or combinations of forms, such as command line input, and interaction with one or more VUIs or GUIs using various types of input, such as pointer device (e.g., mouse) input, keyboard input, voice input, gaze input, speech input, and any other type of input capable of interacting with graphical elements of a GUI or with a VUI.

[0048] In various implementations, the domain in which these actions are performed may be identified, for example, by domain module 104 using any combination of NL queries 2, computer applications operated by user 118 to perform these actions, remote data feed services accessed by the user (e.g., email, text messaging, social media), projects the user is working on, etc. In some implementations, the domain may be identified, at least in part, by an area of ​​a simulated digital world, sometimes referred to as a "metaverse," in which user 118 virtually operates or accesses. For example, user 118 may record actions {B3, B1, ...} that retrieve scores of particular other users (e.g., their online gaming friends), short video replays of those other users' performances, etc. from a first metaverse game associated with domain model B. Similarly, user 118 may record actions {C5, C2, ...} that result in retrieval of scores of the same other users, short video replays of those other users' performances, etc. from a second metaverse game associated with domain model C.

[0049] The aggregator agent 122 may process the actions {B3, B1, ...} using the domain model B (or another domain model associated with the same domain) to generate the action embedding B'. Likewise, the aggregator agent 122 may process the actions {C5, C2, ...} using the domain model C (or another domain model associated with the same domain) to generate the action embedding C'. And as previously described, the aggregator agent 122 (or another component, such as the ML module 108) may process the NL query 2 to generate another data feed-independent aggregator embedding Q2'.

[0050] The aggregator agent 122 (or another component, such as the ML module 108) may then associate the embeddings Q2', B', and C' in and / or across one or more embedding spaces using a variety of different techniques, such as triplet loss, etc. In some implementations, the embeddings Q2', B', and C' may be combined into a single data feed-independent aggregator embedding, for example, by concatenation, averaging, etc. However, with the embeddings Q2', B', and C' associated with each other or combined into a unified embedding, the user 118 or other users may then be able to issue semantically similar natural language queries in the future. In some instances, these queries may be mapped to action embeddings B' and / or C', which in turn may be utilized to aggregate data from multiple different data feed services (including data feed services other than those associated with the domain models B and C). Notably, the NL query 2 need not be loaded with details, nor do other semantically similar queries subsequently issued.

[0051] In various implementations, simulations may be performed, for example, by components of the aggregator agent 122 and / or the inter-domain knowledge system 102 to further train the domain model. More specifically, various permutations of actions may be simulated to determine synthetic results. For example, these synthetic results may be compared to natural language input associated with the original set of actions from which the simulated permutations were selected. The success or failure of these synthetic results may be used as positive training examples and / or negative training examples for the domain model. In this way, the domain model may be trained based on far more actions and accompanying natural language input than those recorded by users.

[0052] Figure 3 From Figure 2 Different angles schematically depict another example of how the techniques described herein may be used to aggregate information from multiple different data feed services in response to a query issued by a user. Starting from the bottom left, user 118 operates client device 120 (in this example, a standalone interactive speaker) and speaks a natural language command "What do the experts and my colleagues in the office think about last night's playoff game?"

[0053] The STT module 330 can perform STT processing to generate speech recognition output. The speech recognition output can be processed by a natural language processing (NLP) module 332, for example, using a machine learning model such as a transformer, RNN, etc., to generate a data feed-independent aggregator embedding 334. Embedding Finder ( Figure 3The STT module 330, the NLP module 332, and / or the EF module 336 may be implemented as part of the aggregator agent 122, as part of the inter-domain knowledge system 102, or any combination thereof.

[0054] Embedding space 338 may be a continuous space including multiple data feed independent aggregator embeddings, each aggregator embedding in Figure 3 , represented by block dots in . These data feed-independent aggregator embeddings can be abstractions of previous natural language queries and, where applicable, domain-specific actions recorded in action spaces of various domains. For purposes of illustration and understanding only, embedding space 338 is depicted as two-dimensional. It should be understood that embedding space 338 will actually have as many dimensions as individual embeddings, which can be hundreds or even thousands of dimensions.

[0055] The white stars represent coordinates in the embedding space 338 associated with the aggregator embedding 334 that is independent of the data feed. Figure 3 , the white star is actually located between the two data feed independent aggregator embeddings enclosed by ellipse 340. In some implementations, multiple embeddings may match a single natural language input, for example, because those multiple embeddings are semantically similar to each other. In some implementations, multiple matching data feed independent aggregator embeddings (such as the two within ellipse 340) may be combined into a unified representation, for example, by concatenation or averaging, and the unified data feed independent aggregator embedding may be processed by downstream components.

[0056] The aggregator proxy 122 may then process or have processed data feed-independent aggregator embedding using multiple domain models AC, each domain model being associated with a different domain from which the user 118 wishes to aggregate information. Domain A may represent, for example, a social media data feed service provided by one or more social media servers 342A. Domain B may represent, for example, a sports data feed service (e.g., a sports-centric website) served by one or more servers 342B. Domain C may represent, for example, a microblog data feed service served by one or more servers 342C. One or more of the servers 342A-C may or may not be part of a cloud infrastructure and, therefore, may not necessarily be tied to a specific server instance.

[0057] Processing the selected action embeddings based on domain model A may generate probability distributions over actions in applicable action spaces. Based on those probability distributions, actions {A1, A2, ...} may be selected, similar to what was previously described. Similarly, processing the selected action embeddings based on domain models B and C may result in actions {B1, B2, ...} and {C1, C2, ...} being selected, respectively. These actions may be executed, for example, by servers 342A-C and / or by compatible client applications executing on client device 120 in their respective domains.

[0058] As a result, the social media server 342A can retrieve recent social media posts from anyone included on the list of media experts and / or office colleagues maintained by the user 118 and return them to, for example, the client device 120 (e.g., via the aggregator agent 122). In some implementations, the user's own default social media feed can be searched, where posts from the user's friends and / or people the user follows can appear in reverse chronological order to find responsive content posted by the expert and / or the user's office mates. Alternatively, the personal feeds of each of the expert and / or the user's office mates can be searched to find responsive content. In various implementations, the manner in which the social media data feed (or any other data feed) is searched can depend on the permissions granted by the entity providing the data feed. For example, a social media data feed service can provide an API that can be accessed to obtain responsive information. Alternatively, if the social media data feed service wishes, it can prevent the aggregator agent 122 from obtaining responsive content, for example, by using terms and conditions and / or using technical means, such as a completely automated public Turing test to tell computers and humans apart (CAPTCHA).

[0059] The sports data feed server 342B can retrieve the latest content contributed by any of the experts and return it to, for example, the client device 120 (e.g., via the aggregator agent 122). Although it may be the case that no one in the office posts content to such a sports website, those office members may be able to post in the comment section of the sports website, such as at the bottom of an article. In some such instances, the comments of those office members can be searched using the techniques described herein to obtain response data. The microblog server 342C can retrieve the most recent microblog post of any of the identified experts or office members and return it to, for example, the client device 120 (e.g., via the aggregator agent 122).

[0060] In some implementations, all or some of these returned messages may be collated and / or aggregated and presented audibly or visually to the user 118. In other implementations, the returned messages may be compared to identify the most recent message, and that message may be presented individually to the user 118. For example, in the case where the client device 120 is a stand-alone interactive speaker without display capabilities, such as Figure 3 In situations where the user 118 is in a vehicle (or, for example, may be in a vehicle), minimizing the amount of output may be beneficial to avoid overwhelming or distracting the user 118, in which case the latest content for any domain may be read aloud. Additionally or alternatively, the returned content may be processed, for example, by the ML module 108 using a sequence-to-sequence machine learning model trained to rephrase and / or summarize longer text content, so that the content ultimately presented to the user 118 is shorter and / or more concise.

[0061] Figure 4 4 is a flowchart illustrating an example method 400 for practicing selected aspects of the present disclosure according to implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of the inter-domain knowledge system 102. In addition, although the operations of the method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0062] At box 402, the system can obtain natural language input including a query for information. The user (118) can type such natural language and / or can provide spoken utterances, which can be processed, for example, by the STT module 330 to generate speech recognition output. In some cases, the natural language input can identify one or more entities related to the search. These entities can include, for example, individuals who publish content to data feed services. These individuals can be friends, experts, scholars, journalists, politicians, celebrities, or other public or private figures. Additionally or alternatively, the natural language input can identify one or more data feed services to be searched. For example, an individual can request content from all subscribed social media feeds of the individual.

[0063] Regardless of the form of the natural language input, at box 404, natural language processing (NLP) can be performed on the natural language input, for example by the ML module 108, to generate a data feed-independent aggregator embedding. For example, the NLP can be performed using a machine learning model such as an RNN or a transformer. As previously described, when the natural language input contains sufficient detail, the data feed-independent aggregator embedding itself can be sufficient to be processed using a domain model to generate a probability distribution. Optionally, the data feed-independent aggregator embedding can be relatively fuzzy as long as it is semantically similar to a previous data feed-independent aggregator embedding generated from a "long tail" or other low-detail natural language query that is mapped to (or combined with) an action embedding generated by the same user or a different user.

[0064] At block 406, the system may select, for example, via the aggregator agent 122 or the domain module 104, a plurality of data feed services from which to aggregate content. The data feed services may be selected based on various signals or factors, such as the content of the natural language input, synonyms of tokens in the natural language input, the context of the user (e.g., at work, in a driving vehicle, time of day), permissions granted by the user (e.g., the user may have previously opted the aggregation agent 122 into various selected data feed services), the user's contact list (which may indicate which data feed services the user's friends push content on), etc. In various implementations, each selected data feed service may include its own data feed service action space of actions that may be executed to access data transmitted by the corresponding data feed service. These actions may include, for example, actions that may be executed at a client application, e.g., using voice, keyboard, pointer device, etc., or actions that may be executed on the server side, e.g., through API calls.

[0065] At block 408, the system, e.g., via the ML module 108 or the aggregator agent 122, may process the feed-independent aggregator embeddings based on a plurality of domain-specific machine learning models corresponding to the plurality of data feed services selected at block 406. Each domain-specific machine learning model may be trained to map data feed service action spaces and a data feed-independent semantic embedding space (e.g., Figure 3 338) in the conversion.

[0066] Based on the processing of box 408, at box 410, the system can select and execute one or more actions from each of the data feed service action spaces to aggregate data responsive to the query from multiple data feed services. For example, the processing of box 408 can generate a probability distribution over the actions in the space for each applicable action space. Those actions with the highest probability can be executed first, then actions with lower probabilities (but still exceeding a certain minimum threshold) can be executed next, and so on. In some cases, the same domain model can be iteratively applied to a sequence of states. The sequence of states can represent, for example, the evolving state of a client application communicating with a data feed service, the evolving state of an interaction or exchange with a data feed service, the changing context of a user or a client device they operate, and so on. In some implementations, reinforcement learning can be used to train the iteratively applied domain model, although this is not required.

[0067] At block 412 , the system (eg, via the interface module 106 ) may cause the aggregated response data to be presented as output, audibly, on a screen, or the like.

[0068] Figure 5 is a block diagram of an example computing device 510 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing devices 120-1 through 120-P, the inter-domain knowledge system 102, and / or other components can include one or more components of the example computing device 510.

[0069] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0070] The user interface input device 522 may include a keyboard, a pointer device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen integrated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 510 or over a communication network.

[0071] User interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other machines for creating visible images. The display subsystem may also provide a non-visual display such as through an audio output device. Typically, the use of the term "output device" is intended to include devices and methods of all possible types of information output from computing device 510 to a user or another machine or computing device.

[0072] The storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include a Figure 4 The logic of selected aspects of method 400.

[0073] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random-access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 526 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations can be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible by the processor 514.

[0074] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0075] Computing device 510 may be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of computing device 510 described in the foregoing is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of computing device 510 may have more Figure 5 The computing device described in more or less components.

[0076] Although several implementations have been described and illustrated herein, various other devices and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be utilized, and each of these changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which the teachings are used. Those skilled in the art will recognize or be able to determine many equivalents of the specific implementations herein using experiments that do not exceed routine. Therefore, it should be understood that the aforementioned implementations are presented only as examples, and within the scope of the appended claims and their equivalents, implementations may be practiced in a manner different from that specifically described and claimed. The implementations of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented using one or more processors, comprising: obtaining natural language input comprising a query for information; performing natural language processing (NLP) on the natural language input to generate a data feed-agnostic aggregator embeddings; selecting a plurality of data feed services, wherein each data feed service of the selected plurality of data feed services comprises its own data feed service action space, wherein each data feed service action space comprises actions executable to access data communicated through the respective data feed service; processing the feed-independent aggregator embeddings based on a plurality of domain-specific machine learning models corresponding to the plurality of data feed services, wherein each domain-specific machine learning model is trained to transform between a corresponding data feed service action space and a data feed-independent semantic embedding space including the data feed-independent aggregator embeddings; Based on the processing, selecting and executing one or more actions from each of the data feed service action spaces to aggregate data responsive to the query from the plurality of data feed services; and Causes the aggregated response data to be presented as output. 2 . The method of claim 1 , wherein the plurality of data feed services are selected based on an entity identifier contained in the query for information.

3. The method according to claim 1 or 2, wherein: The query for information includes a request for social media posts from a particular individual, and the selected plurality of data feed services includes two or more social media services selected based on membership of the particular individual in the two or more social media services.

4. The method according to claim 3, wherein: For a given social media service of the two or more social media services, the one or more actions selected from a data feed service action space of the given social media service include accessing one or more posts of the particular individual from a posting history of the particular individual.

5. The method according to claim 3 or 4, wherein: For a given social media service of the two or more social media services, the one or more actions selected from a data feed service action space of the given social media service include filtering one or more posts of the specific individual from a general data feed on the given social media service provided to a user issuing the natural language input.

6. The method of any one of claims 1 to 5, wherein the plurality of data feed services are selected based on a lookup table controlled by a user issuing the natural language input.

7. The method according to claim 6, wherein: The lookup table includes a contact list for the user.

8. The method of any one of claims 1 to 7, wherein the plurality of data feed services are selected based on that a user issuing the natural language input has previously provided an aggregator agent with permission to access the plurality of data feed services.

9. A method according to any one of claims 1 to 8, wherein at least one of the data feed services comprises a virtual space, the virtual space forming part of a larger metaverse comprising a plurality of virtual spaces.

10. The method according to any one of claims 1 to 9, wherein: The aggregated response data is presented as part of a Metaverse graphical user interface to the user who issued the natural language input.

11. The method of any one of claims 1 to 10, wherein the plurality of data feed services are selected based on a browsing history of a user issuing the natural language input.

12. A system comprising one or more processors and a memory storing instructions, wherein in response to execution of the instructions, the one or more processors: obtaining natural language input comprising a query for information; performing natural language processing (NLP) on the natural language input to generate a data feed-agnostic aggregator embeddings; selecting a plurality of data feed services, wherein each data feed service of the selected plurality of data feed services comprises its own data feed service action space, wherein each data feed service action space comprises actions executable to access data communicated through the respective data feed service; processing the feed-independent aggregator embeddings based on a plurality of domain-specific machine learning models corresponding to the plurality of data feed services, wherein each domain-specific machine learning model is trained to transform between a corresponding data feed service action space and a data feed-independent semantic embedding space including the data feed-independent aggregator embeddings; selecting and executing one or more actions from each of the data feed service action spaces to aggregate data responsive to the query from the plurality of data feed services; and Causes the aggregated response data to be presented as output.

13. The system of claim 12, wherein the plurality of data feed services are selected based on an entity identifier contained in the query for information.

14. The system according to claim 12 or 13, wherein: The query for information includes a request for social media posts from a particular individual, and the selected plurality of data feed services includes two or more social media services selected based on membership of the particular individual in the two or more social media services.

15. The system of claim 14, wherein: For a given social media service of the two or more social media services, the one or more actions selected from a data feed service action space of the given social media service include accessing one or more posts of the particular individual from a posting history of the particular individual.

16. The system according to claim 14 or 15, wherein: For a given social media service of the two or more social media services, the one or more actions selected from a data feed service action space of the given social media service include filtering one or more posts of the specific individual from a general data feed on the given social media service provided to a user issuing the natural language input.

17. The system of any one of claims 12 to 16, wherein the plurality of data feed services are selected based on a lookup table controlled by a user issuing the natural language input.

18. The system of claim 17, wherein the lookup table comprises a contact list of the user.

19. The system of any of claims 12-16, wherein the plurality of data feed services are selected based on that a user issuing the natural language input has previously provided an aggregator agent with permission to access the plurality of data feed services.

20. A non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by a processor, cause the processor to: obtaining natural language input comprising a query for information; performing natural language processing (NLP) on the natural language input to generate a data feed-agnostic aggregator embeddings; selecting a plurality of data feed services, wherein each data feed service of the selected plurality of data feed services comprises its own data feed service action space, wherein each data feed service action space comprises actions executable to access data communicated through the respective data feed service; processing the feed-independent aggregator embeddings based on a plurality of domain-specific machine learning models corresponding to the plurality of data feed services, wherein each domain-specific machine learning model is trained to transform between a corresponding data feed service action space and a data feed-independent semantic embedding space including the data feed-independent aggregator embeddings; selecting and executing one or more actions from each of the data feed service action spaces to aggregate data responsive to the query from the plurality of data feed services; and Causes the aggregated response data to be presented as output.

Images