Aggregation of information from different data feed services

Abstract

This system and medium uses machine learning to aggregate information in response to queries from multiple different data feed services. [Solution] A natural language processing (NLP) method is performed on a natural language input containing a query seeking information to generate a data feed-independent aggregator embedding (FAAE), the method selecting multiple data feed services. Each data feed service has its own data feed service action space, which contains actions that can be performed to access data through the data feed service. The FAAE processes based on domain-specific machine learning models corresponding to the selected data feed services, using these models to select an action(s) from the data feed service action space and aggregate data from the multiple data feed services in response to the query.

Description

Technical Field

[0001] Individuals often habitually seek semantically related information from multiple different sources. For example, a particular individual may check the same data feed service such as a news website and / or a social networking data feed to obtain different perspectives on a particular topic such as developing events, new movies, sports events, etc. However, in addition to the specific information sought by an individual, various data feed services may communicate countless other information that is not relevant to that individual at that time. As a result, it can be cumbersome and / or time-consuming for an individual to sift through multiple different data feed services to obtain the specific information they seek. This problem can be amplified when interacting with a voice user interface (VUI) provided by a computing device such as a smart speaker or an in-vehicle voice command system. Additionally, individual data feed services routinely change how data is presented and / or made accessible, which can further impede an individual's efforts.

Summary of the Invention

[0002] This specification describes implementations for using machine learning to aggregate information in response to queries from multiple different data feed services. More specifically, this specification describes implementations for leveraging domain-specific machine learning models to query information from heterogeneous data feed services using data feed-independent aggregator embeddings, although these are not mutually exclusive. The embodiments described herein offer several technical advantages. Rather than navigating multiple different data feed services to obtain information on a given topic, an individual can simply issue a free-form natural language query describing what information they are seeking and, where applicable, from where they are seeking this information. Response information from multiple different data feed services can then be retrieved and presented to the individual, for example, as part of an aggregated data feed. Thus, the individual can obtain different perspectives and / or multiple perceptions of a given topic from different people across multiple different data feed services.

[0003] In some implementations, the method can be performed using one or more processors and includes: obtaining natural language input containing a query seeking information; performing natural language processing (NLP) on the natural language input to generate a data feed-independent aggregator embedding; selecting multiple data feed services, each of which will be selected, containing its own data feed service action space, each data feed service action space containing actions that can be performed to access data communicated through each data feed service; processing the feed-independent aggregator embedding based on multiple domain-specific machine learning models corresponding to the multiple data feed services, each domain-specific machine learning model trained to translate between its respective data feed service action space and a data feed-independent semantic embedding space containing the data feed-independent aggregator embedding; and, based on the processing, selecting and executing one or more actions from each of the data feed service action spaces to aggregate data from the multiple data feed services in response to the query; and presenting the aggregated response data as output.

[0004] In various implementations, multiple data feed services may be selected based on entity identifiers included in the query requesting information. In various implementations, the query requesting information may include a request for social media posts from a specific individual, and the selected multiple data feed services may include two or more social media services, and the two or more social media services may be selected based on the specific individual's membership in those two or more social media services. In various implementations, for a given social media service among two or more social media services, one or more actions selected from the data feed service action space of a given social media service may include accessing one or more posts by a specific individual from that individual's posting history. In various implementations, for a given social media service among two or more social media services, one or more actions selected from the data feed service action space of a given social media service may include filtering one or more posts by a specific individual from a general data feed on a given social media service provided to the user who issued the natural language input.

[0005] In various implementations, multiple data feed services may be selected based on a lookup table controlled by the user who issued the natural language input. In various implementations, the lookup table may include the user's contact list. In various implementations, multiple data feed services may be selected based on the user who issued the natural language input and has previously granted the aggregator agent permission to access multiple data feed services. In various implementations, at least one of the data feed services may include a virtual space that forms part of a larger metaverse comprising multiple virtual spaces.

[0006] In various implementations, aggregated response data may be presented to the user who issued the natural language input as part of a metaverse graphical user interface. In various implementations, multiple data feed services may be selected based on the browsing history of the user who issued the natural language input.

[0007] Furthermore, some implementations include one or more processors in one or more computing devices, the one or more processors being operable to execute instructions stored in associated memory, and the instructions being configured to perform one of the methods described above. Some implementations include at least one non-temporary computer-readable storage medium storing computer instructions that can be executed by one or more processors to perform one of the methods described above.

[0008] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are intended to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are intended to be part of the subject matter disclosed herein. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of an exemplary environment in which the implementations disclosed herein can be realized. [Figure 2] This outlines some examples of how data may be exchanged and / or processed in order to implement a selected aspect of this disclosure in various implementation forms. [Figure 3] This outlines another example of how data can be processed to query multiple domains using data feed-independent aggregator embedding in various implementation forms. [Figure 4] This flowchart shows an exemplary method for practicing a selected aspect of the disclosure, as disclosed herein. [Figure 5]This shows an exemplary architecture of a computing device. [Modes for carrying out the invention]

[0010] This specification describes implementations for using machine learning to aggregate information in response to queries from multiple different data feed services. More specifically, this specification describes implementations for leveraging domain-specific machine learning models to query information from heterogeneous data feed services using data feed-independent aggregator embeddings, although these are not mutually exclusive. The embodiments described herein offer several technical advantages. Rather than navigating multiple different data feed services to obtain information on a given topic, an individual can simply issue a free-form natural language query describing what information they are seeking and, where applicable, from where they are seeking this information. Response information from multiple different data feed services can then be retrieved and presented to the individual, for example, as part of an aggregated data feed. Thus, the individual can obtain different perspectives and / or multiple perceptions of a given topic from different people across multiple different data feed services.

[0011] As used herein, “data feed service” may be any computer implementation service, such as a web service, that can be used by an individual to publish, broadcast, push, and / or multicast content to others. A data feed service is typically accessible to multiple individuals and is generally updated frequently, or at least periodically, with new content, often presented in reverse chronological order. One common example of a data feed service is a social networking service, where individual users can publish information, such as facts, opinions, photos, and status, to their “friends” or “followers,” or to anyone who accesses that social networking service. Another example of a data feed service is a news website (or more generally, a news service that can publish on both a website and its own application) that serially publishes news articles, opinions, columns, etc.

[0012] These examples of data feed services are not intended to be limiting. Other examples of data feed services include, but are not limited to, weblogs (more commonly called "blogs"), web feeds (e.g., really simply syndication or "RSS" feeds), review aggregation websites, and social news websites featuring user-generated content. Furthermore, data feed services are not limited to services provided to websites. For example, a data feed service can take the form of a virtual space that forms part of a larger metaverse containing multiple virtual spaces (e.g., a massively multiplayer online role-playing game ("MMORPG") or a part thereof, a virtual coffee shop, or a forum).

[0013] To obtain information in response to a single natural language query from multiple data feed services simultaneously, natural language processing (NLP) can be performed on the natural language query to generate a semantic representation referred to herein as a "data feed-independent aggregator embedding." A data feed-independent aggregator embedding can abstractly represent the meaning contained in the natural language query. In some cases, a data feed-independent aggregator embedding may be a dense numerical representation, such as a vector of real numbers, functioning as an embedding in a continuous vector space.

[0014] By leveraging data feed-independent aggregator embedding, multiple data feed services that potentially contain information responding to an individual's query may be selected to retrieve information from multiple sources simultaneously. Data feed services may be selected per query and / or across multiple different queries. Potentially responding data feed services may be selected in various ways based on various signals.

[0015] In some implementations, data feed services may be selected based on a lookup table associated with the individual who issued the query. For example, an individual may have a contact list (e.g., phone book, friends on social media, etc.) that specifies which data feed services are used by their contacts, and therefore which data feed services should be selected. Another example is that an individual may grant explicit permission to what is referred to herein as an "aggregator agent" to retrieve response information from an enumerated or aggregated list of data feed services. Yet another example is that an individual's browsing history can be examined to determine which data feed services they tend to look at, generally and / or in a specific context, with or without input from the individual. For example, if an individual issues a sports-related query, the sports-related data feed services that the individual has visited in the past can be selected. In some implementations, data feed services may be selected based on entities (people, places, or things) identified within the query. For example, an individual may request information about an ongoing event communicated (e.g., published, posted, or organized) by a particular media commentator. The data feed services used by that particular media commentator can then be selected.

[0016] Each data feed service may include its own data feed service action space, which includes actions that can be performed to access data communicated through that data feed service. Actions in such an action space may include actions that can be performed using input devices, such as keyboards and pointer devices, to navigate a graphical user interface (GUI) provided by the data feed service. For example, if the GUI takes the form of an interactive web page, the action space may include actions that can be performed using graphical inputs such as fields, pull-down menus, and buttons presented as part of the interactive web page. In addition to or instead of this, actions may include commands, queries, and / or parameters that can be used to navigate the GUI or VUI provided by the data feed service to retrieve specific information. For example, the interface (GUI or VUI) of a particular data feed service may include search fields to facilitate the submission of natural language queries to quickly retrieve response content, and / or filter fields to refine search results.

[0017] Domain-specific machine learning models configured in selected aspects of this disclosure may be trained to translate between these action spaces and data-feed-independent semantic embedding spaces, including the aforementioned data-feed-independent aggregator embeddings. These domain-specific machine learning models (or simply “domain models” elsewhere herein) may take various forms, such as various types of neural networks, transformers, RNNs, graph-based neural networks, etc. In various implementations, domain-specific machine learning models may be used to process data-feed-independent aggregator embeddings to generate one or more probability distributions across actions in the action space of a data-feed service. Based on these probability distributions, actions may be selected and performed to retrieve content from the data-feed service in response to queries semantically represented by the data-feed-independent aggregator embeddings. Response content from the data-feed service may be aggregated with response content similarly retrieved from other data-feed services. The aggregated response content may then be presented to the individual who issued the original natural language input.

[0018] As used herein, “domain” can refer to a target area in which a computing component is intended to operate, e.g., the scope of knowledge, influence, and / or activity around which the logic of the computing component revolves. In some implementations, the domain to which a query is submitted may be identified by heuristically matching keywords in user-provided input with domain keywords. In other implementations, user-provided input may be processed using NLP techniques such as word2vec, Bidirectional Encoder Representation with Transformers (BERT) transformers, and various types of recurrent neural networks ("RNNs," e.g., Long Short-Term Memory, i.e., "LSTM," Gated Recurrent Units, i.e., "GRU") to generate semantic embeddings representing the natural language input. In some implementations, these natural language input semantic embeddings (which may be referred to as “data-feed-independent aggregator embeddings” as previously stated) may be used to identify one or more domains based on the distance(s) in the embedding space(s) between the data-feed-independent aggregator embeddings and other embeddings associated with various domains. These distances within the embedding space can be calculated using techniques such as Euclidean distance, dot product, or cosine similarity.

[0019] In various implementations, one or more domain models may be pre-generated for each domain. For example, one or more machine learning models such as RNNs (e.g., LSTM, GRU), BERT transformers, various types of neural networks, and reinforcement learning policies may be trained on a corpus of documentation associated with the domain. As a result of this training, one or more domain models may be at least bootstrapped to be usable for processing what is referred to herein as “domain-independent aggregator embeddings,” and may generate one or more probability distributions across the action space associated with the target domain. Based on these probability distributions, multiple actions may be selected and executed to perform user-submitted queries in the target domain.

[0020] Figure 1 schematically illustrates exemplary environments in which selected embodiments of this disclosure may be realized in various implementation forms. Any computing devices shown in Figure 1 or elsewhere in the Figure may include logic such as one or more microprocessors (e.g., a central processing unit or "CPU", a graphical processing unit or "GPU", a tensor processing unit ("TPU")) that execute computer-readable instructions stored in memory, or other types of logic such as application-specific integrated circuits ("ASICs"), field-programmable gate arrays ("FPGAs"). Some of the systems shown in Figure 1, such as the interdomain knowledge system 102, may be realized using one or more server computing devices that form what is sometimes called a "cloud infrastructure," but this is not required. In other implementation forms, embodiments of the interdomain knowledge system 102 may be realized on a client device 120, for example, for purposes such as protecting privacy or reducing latency.

[0021] The interdomain knowledge system 102 may include several different components configured in a selected aspect of the disclosure, such as, for example, a domain module 104, an interface module 106, and a machine learning (ML in Figure 1) module 108. The interdomain knowledge system 102 may also include any number of databases for storing machine learning model weights and / or other data used to perform a selected aspect of the disclosure. In Figure 1, for example, the interdomain knowledge system 102 includes a database 110 for storing global domain models and another database 112 for storing data representing global action embeddings.

[0022] The interdomain knowledge system 102 can be operably coupled with any number of client computing devices operated by any number of users via one or more computer networks (114). In Figure 1, for example, the first user 118-1 operates a coordinated ecosystem of one or more client devices 120-1 (e.g., client devices controlled by user 118-1 and / or associated with user 118-1's online profile). The p-th user 118-P operates one or more client devices 120-P. As used herein, client device(s) 120 may include, for example, one or more of the following: desktop computing devices, laptop computing devices, tablet computing devices, mobile phone computing devices, computing devices in the user's vehicle (e.g., in-vehicle communication systems, in-vehicle entertainment systems, in-vehicle navigation systems), standalone interactive speakers (which may optionally include visual sensors and / or touchscreen displays), smart devices such as smart televisions (or standard televisions with networked dongles having automation assistant functions), and / or wearable devices for the user including computing devices (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0023] Domain module 104 can be configured to determine various different information regarding a domain (e.g., a data feed service) related to a given user 118 at a given point in time (e.g., the data feed service that the user 118 is currently involved in, the data feed service(s) that the user wants to query information about, etc.). For this purpose, the domain module 104 can collect context information regarding, for example, foreground applications and / or background applications executed on client device(s) 120 operated by the user 118, web pages currently / most recently visited by the user 118, domains that the user 118 has access to and / or frequently accesses, etc.

[0024] Using this collected context information, in some implementations, the domain module 104 may be configured to identify one or more domains (e.g., data feed services) related to natural language input provided by the user. For example, a user-configured query requesting response information from multiple different data feed services can be processed by the domain module 104 to identify the data feed service(s) that the user 118 is trying to query.

[0025] In some implementations, the domain module 104 may also be configured to extract domain knowledge from various different sources related to the identified domain. In some such implementations, this extracted domain knowledge (and / or embeddings generated therefrom) can be provided to downstream component(s), for example, in addition to the aforementioned natural language input or context information. This additional domain knowledge can enable the downstream component(s), particularly a machine learning model, to make more likely satisfactory predictions (e.g., aggregating response information across multiple different domains).

[0026] In some implementations, the domain module 104 can apply the collected context information (e.g., the current state) across one or more “domain selection” machine learning models 105 different from the domain model described herein. These domain selection machine learning models 105 can take various forms, such as various types of neural networks, support vector machines, random forests, BERT transformers, etc. In various implementations, the domain selection machine learning models 105 may be trained to select an applicable domain based on the attributes (or “context signals”) of the current context or state of the user 118 and / or the client device 120. For example, when the user 118 is operating an input form on a particular website to procure a good or service, the attributes of the underlying web page(s), such as the uniform resource locator (URL) of that website, or keywords, tags, document object model (DOM) elements (if any), can be applied as input across the models, either in their native form or a dimensionality-reduced embedding. Other context signals that can be considered include, but are not limited to, the user's IP address (e.g., work vs. home vs. mobile IP address), time, social media status, calendar, email / text messaging content, etc.

[0027] Interface module 106 may provide one or more GUIs or VUIs that can be operated by various individuals, such as users 118-1 to 118-P, to perform various actions made available by the semantic task automation system. In various implementations, user 118 may operate the GUI provided by interface module 106 (e.g., a standalone application or a web page) and may or may not select and utilize the various techniques described herein. For example, users 118-1 to 118-P may provide explicit permission for each data feed service (more generally, a domain) they wish to query, and then request that those search requests be used to retrieve response information from those data feed services.

[0028] ML module 108 can access data representing various global domains / machine learning models / policies in database 110. These trained global domains / machine learning models / policies can take various forms, including, but are not limited to, graph-based networks such as graph neural networks (GNNs), graph attention neural networks (GANNs), or graph convolutional neural networks (GCNs), sequence-to-sequence models such as encoder-decoders, various recurrent neural networks (e.g., LSTM, GRU, etc.), BERT transformer networks, reinforcement learning policies, and any other types of machine learning models that may be applied to facilitate selected aspects of this disclosure. ML module 108 may process various data based on these machine learning models in response to requests or commands from other components such as domain module 104 and / or interface module 106.

[0029] Each client device 120 can operate at least a portion of what is referred to herein as “Aggregator Agent” 122. The Aggregator Agent 122 may be a computer application operable by User 118 to perform selected embodiments of the Disclosure to facilitate the cross-domain data aggregation described herein. For example, the Aggregator Agent 122 may receive requests and / or permissions from User 118 to aggregate query response data from several different domains. In some implementations, the Aggregator Agent 122 may be operable to grant access to other parties, such as other Aggregator Agents 122, to domains controlled by the user. For example, User 118-1 may interact with Aggregator Agent 122-1 to allow a specific other individual to access a portion of a domain controlled by User 118-1, such as the user's own social networking profile feed. Without such explicit permission, other Aggregator Agents 122 may not be able to retrieve response information from that user's social networking profile feed.

[0030] In some implementations, the aggregator agent 122 may take the form of a “virtual assistant” or often referred to as an “automation assistant” configured to engage in human-computer natural language interaction with the user 118. For example, the aggregator agent 122 may be configured to semantically process natural language input provided by the user 118 to identify one or more intentions. Based on these intentions, the aggregator agent 122 can perform a variety of tasks, such as operating smart devices, retrieving information, and executing tasks. In some implementations, the interaction between the user 118 and the aggregator agent 122 (or a separate automation assistant accessible by / through the aggregator agent 122) may constitute a set of actions that can be captured, abstracted into domain-independent embeddings, and then extended to other domains, as described herein.

[0031] In Figure 1, each of the client devices 120-1 may include an aggregator agent 122-1 that provides services to a first user 118-1. The first user 118-1 and its aggregator agent 122-1 can access and / or associate with a “profile” containing various data related to performing selected aspects of the Disclosure on behalf of the first user 118-1. For example, an aggregator agent 122 can access one or more edge databases or datastores associated with the first user 118-1, including an edge database 124-1 that stores local domain models and / or another edge database 126-1 that stores recorded actions. Other users 118 may have a similar configuration. Any of the data stored in edge databases 124-1 and 126-1 may be stored partially or entirely on the client device 120-1, for example, to protect the privacy of the first user 118-1. For example, the recorded action 126-1 may include confidential and / or personal information of the first user 118-1, such as payment information, address, and telephone number, and may be stored locally in its raw form on the client device 120-1.

[0032] Local domain models (or multiple) stored in edge database 124-1 may include, for example, local versions of global models (or multiple) stored in global domain model database 110. For example, in some implementations, global models can be propagated to the edges for the purpose of bootstrapping the aggregator agent 122 to extend tasks to new domains associated with those propagated models, and the local models at the edges may or may not be trained locally based on user 118 activity and / or feedback. In some such implementations, local models (in edge database 124, substituted by "local gradients") may be periodically used to train global models (in database 110), for example, as part of a federated learning framework. Since global models are trained on local models, global models can, in some cases, be propagated back to other edge databases (124) to keep the local models up-to-date.

[0033] However, the adoption of federative learning is not a requirement in all implementations. In some implementations, the aggregator agent 122 can provide scrubbed data to the interdomain knowledge system 102, and the ML module 108 can remotely apply a model to the scrubbed data. In some implementations, the "scrubbed" data may be data from which sensitive and / or personal information has been removed and / or obscured. In some implementations, personal information may be scrubbed at the edge, for example, by the aggregator agent 122, based on various rules. In other implementations, the scrubbed data provided to the interdomain knowledge system 102 by the aggregator agent 122 may be in the form of dimensionality-reduced embeddings generated from the raw data at the client device 120.

[0034] As mentioned above, the edge database 126-1 can store actions recorded by the aggregator agent 122-1. The aggregator agent 122-1 can record actions in various different ways depending on the access level of the aggregator agent 122-1 to the computer applications running on the client device 120-1 and the permissions granted by the user 118. For example, most smartphones include an operating system (OS) interface for granting or revoking permissions to various computer applications (e.g., location, camera access, etc.). In various implementations, such an OS interface may be able to operate to grant / revok access to the aggregator agent 122, and / or allow the aggregator agent 122 to select a particular level of access it has to specific computer applications and / or domains to which those applications provide access.

[0035] The aggregator agent 122-1 may have varying levels of access to the computer application's operations, depending on the permissions granted by user 118 and the cooperation of the software developers providing the computer application. Some computer applications may, for example, with user 118's permission, provide the aggregator agent 122 with "hidden" access to the application's API or to scripts written using a programming language (e.g., macros) embedded in the computer application. Other computer applications may not provide as much access. In such cases, the aggregator agent 122 may record actions in other ways, such as by capturing screenshots, performing optical character recognition (OCR) on those screenshots to identify menu items, and / or by monitoring user input (e.g., interrupts caught by the OS) to determine which graphical elements were manipulated by user 118 and in what order. In some implementations, the aggregator agent 122 may intercept actions performed using the computer application from data exchanged between the computer application and the underlying OS (e.g., via system calls). In some implementations, the aggregator agent 122 can intercept and / or access data exchanged between the window manager and / or the window system, or data used by the window manager and / or the window system.

[0036] Figure 2 is a schematic diagram illustrating an example of how data may be processed by and / or using various components across domains. Starting from the top left, user 118 interacts with client device 120 to provide a typed or spoken natural language query NL QUERY1. In the latter case, the spoken utterance may first be processed using a speech-to-text (STT) engine (not shown) to generate speech recognition output. In either case, NL QUERY1 may be provided to aggregator agent 122.

[0037] The aggregator agent 122 may process data indicating NL QUERY 1, or have the ML module 108 process it, to generate a data feed-independent aggregator embedding (DAAE) Q1'. In some implementations, the aggregator agent 122 and / or the ML module 108 may use a transformer network or a recurrent neural network, such as a machine learning model like LSTM or GRU, to generate the DAAE Q1'. The DAAE Q1' may then be processed, for example, by the aggregator agent 122 or the ML module 108, using a domain model B associated with a first data feed service and a domain model C associated with a second data feed service. Processing the DAAE Q1' using domain model B generates a probability distribution(s), which may be used to select a set of actions {B1, B2, ...} from the first data feed service action space associated with the first data feed service. Similarly, processing DAAE Q1' using domain model C results in multiple actions {C1, C2, ...} being selected from a second data feed service action space associated with the second data feed service.

[0038] These selected actions {B1, B2, ...} and {C1, C2, ...} can be executed by various components to retrieve information in response to the original query NL QUERY 1 from the respective domains of the first and second data feed services. In Figure 2, for example, the selected actions {B1, B2, ...} and {C1, C2, ...} are provided to the client device 120 by the aggregator agent 122. The client device 120 can then execute the selected actions, for example, by the corresponding application installed and running on the client device 120.

[0039] For example, if the first data feed service associated with domain model B is a social networking service, a compatible social networking application (e.g., a client) running on client device 120 can automatically perform actions {B1, B2, ...} to retrieve response data from the social networking service. Similarly, if the second data feed service associated with domain model C is a blog compiled by a second individual, a compatible client application (e.g., a web browser) running on client device 120 can automatically perform actions {C1, C2, ...} to retrieve response data from the blog. User 118 may or may not be able to see these automatically performed actions in the GUI rendered by the client application.

[0040] The quality and / or responsiveness of the aggregated information returned in response to a user's initial query may depend, at least in part, on the specificity of the query. Vague queries may result in actions that do not truly fulfill the user's intent, for example, because the resulting aggregated information may have limited values, and / or the actions selected and performed for different data feed services may not match the user's intent, and the response information retrieved from different data feed services may also not match the user's intent. In Figure 2, for example, if NL QUERY1 is vague and / or ambiguous, the selected actions {B1, B2, ...} and {C1, C2, ...} may yield substantially different results from each data feed service. On the other hand, queries that are more detailed and clearly indicate the user's intent are more likely to yield better results. However, if the burden of being specific and clear is too great for the user, the user may prefer to manually aggregate information from multiple different data feed services.

[0041] Therefore, in some implementations, users may be able to record the actions they perform on a particular data feed service to obtain response information and associate those actions with user-defined natural language statements. In other words, the recorded actions, rather than the statements themselves, contain contextual information that can be used to perform similar actions on other data feed services. In this way, otherwise vague natural language statements (e.g., "long-tail" natural language statements) can be associated with specific actions that can be used to aggregate information from multiple different data feed services.

[0042] An example of this is illustrated in Figure 2. Below the dashed line, user 118 operates client device 120 to request and / or authorize the recording of actions performed by user 118 using client device 120 in relation to another natural language query NL QUERY2. In various implementations, the aggregator agent 122 cannot record actions without receiving this permission. In some implementations, this permission may be granted per individual data feed service and / or per application, much like how an application is granted permission to access the use of GPS coordinates, local files, onboard cameras, etc. In other implementations, this permission may only be granted, for example, by pressing a “Stop Recording” button, or by providing voice input such as “Stop Recording” or “End,” similar to recording a macro, unless user 118 states otherwise.

[0043] When a request / permission is received, in some implementations, the aggregator agent 122 may acknowledge (ACK) the request / permission. Then, a series of actions {B3, B1, ...} and a series of actions {C5, C2, ...} performed by the user 118 using the client device 120 may be captured and stored in the edge database 126. These actions {B3, B1, ...} and {C5, C2, ...} may take various forms or combinations of forms, such as command line input, as well as interaction with one or more VUI or GUI graphical elements using various types of input, such as command line input, as well as input from a pointer device (e.g., mouse), keyboard input, voice input, eye-tracking input, and any other type of input capable of interacting with VUI or GUI graphical elements.

[0044] In various implementations, the domain(s) in which these actions are performed may be identified by, for example, the domain module 104, using any combination of, for example, NL QUERY 2, the computer application(s) operated by user 118 to perform these actions, remote data feed services accessed by the user (e.g., email, text messaging, social media), and the project the user is working on. In some implementations, the domain(s) may be identified at least partially by an area of ​​a simulated digital world, sometimes referred to as the “metaverse,” which user 118 virtually operates on or visits. For example, user 118 may record actions {B3, B1, ...} from a first metaverse game associated with domain model B to retrieve the scores of specific other users (e.g., friends in those online games), a brief video playback of those other users' performance, etc. Similarly, user 118 may record actions {C5, C2, ...} from a second metaverse game associated with domain model C to retrieve the scores of the same other users, a brief video playback of those other users' performance, etc.

[0045] Aggregator agent 122 may process actions {B3, B1, ...} using domain model B (or another domain model associated with the same domain) to generate action embedding B'. Similarly, aggregator agent 122 may process actions {C5, C2, ...} using domain model C (or another domain model associated with the same domain) to generate action embedding C'. As previously stated, aggregator agent 122 (or another component such as ML module 108) may process NL QUERY2 to generate another data feed-independent aggregator that embeds Q2'.

[0046] Next, the aggregator agent 122 (or another component such as the ML module 108) may associate the embeddings Q2', B', and C' within and / or across one or more embedding spaces using various different techniques, such as triplet loss. In some implementations, the embeddings Q2', B', and C' may be combined into a single data feed-independent aggregator embedding via concatenation, averaging, etc. Regardless of how the embeddings Q2', B', and C' are associated with each other or combined into a unified embedding, user 118 or other users may issue semantically similar natural language queries in the future. In some cases, these queries may be mapped to action embeddings B' and / or C', which can be used to aggregate data from multiple different data feed services, including data feed services other than those associated with domain models B and C. In particular, NL QUERY2 does not need to contain details, nor do other semantically similar queries issued later need to contain details.

[0047] In various implementations, simulations can be performed, for example, by components of the aggregator agent 122 and / or the inter-domain knowledge system 102, to further train the domain model. More specifically, various permutations of actions can be simulated to determine the synthesis results. These synthesis results can be compared, for example, with the natural language input associated with the original set of actions from which the simulated permutations were selected. The success or failure of these synthesis results can be used as positive and / or negative training examples for the domain model. In this way, it is possible to train the domain model based on far more than just user-recorded actions and accompanying natural language input.

[0048] Figure 3 schematically illustrates, from a different perspective than Figure 2, another example of how the techniques described herein can be used to aggregate information from multiple different data feed services in response to a user-issued query. Starting from the bottom left, user 118 interacts with client device 120 (in this example, a standalone interactive speaker) and speaks the natural language command, "What are the commentators and my colleagues at work saying about last night's playoff game?"

[0049] The STT module 330 can perform STT processing to generate speech recognition output. The speech recognition output can be processed by the Natural Language Processing (NLP) module 332 using, for example, a machine learning model (multiple) (e.g., a converter, RNN, etc.) to generate a data feed-independent aggregator embedding 334. The embedding finder ("EF" in Figure 3) module 336 can map or project the data feed-independent aggregator embedding 334 onto existing data feed-independent aggregator embeddings (white stars) in the embedding space 338. In various implementations, the STT module 330, the NLP module 332, and / or the EF module 336 may be implemented as part of the aggregator agent 122, as part of the interdomain knowledge system 102, or as any combination thereof.

[0050] The embedding space 338 may be a contiguous space containing multiple data feed-independent aggregator embeddings, each represented by a black dot in Figure 3. These data feed-independent aggregator embeddings may be abstractions of previous natural language queries and, where applicable, domain-specific actions recorded in action spaces across various domains. The embedding space 338 is shown as two-dimensional for illustrative and comprehensible purposes only. It should be understood that the embedding space 338 actually has as many dimensions as the individual embeddings, and may have hundreds or thousands of dimensions.

[0051] The white star represents the coordinates in the action embedding space 338 associated with the data feed-independent aggregator embedding 334. As can be seen in Figure 3, this white star is actually located between two data feed-independent aggregator embeddings enclosed by an ellipse 340. In some implementations, multiple embeddings may match a single natural language input, for example, because those multiple embeddings are semantically similar to one another. In some implementations, multiple matching data feed-independent aggregator embeddings, such as the two within the ellipse 340, may be combined into an integrated representation, for example, via concatenation or averaging, and the integrated data feed-independent aggregator embedding may be processed by downstream components.

[0052] Next, the aggregator agent 122 may, or already is, process data feed-independent aggregator embeddings using multiple domain models A to C, each associated with a different domain from which user 118 wishes to aggregate information. Domain A may represent, for example, a social media data feed service provided by one or more social media servers 342A. Domain B may represent, for example, a sports data feed service (e.g., a sports-centric website) serviced by one or more servers 342B. Domain C may represent, for example, a microblogging data feed service serviced by one or more servers 342C. One or more of servers 342A to C may or may not be part of the cloud infrastructure and therefore do not necessarily have to be tied to a specific server instance.

[0053] By processing the action embedding(s) selected based on domain model A, it is possible to generate probability distributions(s) across actions in the applicable action space. Based on these probability distributions(s), actions {A1, A2, ...} may be selected in the same manner as previously described. Similarly, by processing the action embedding(s) selected based on domain models B and C, it is possible to result in the selection of actions {B1, B2, ...} and {C1, C2, ...}, respectively. These actions may be executed in their respective domains, for example, by servers 342A-C and / or by compatible client applications(s) running on client device 120.

[0054] As a result, the social media server 342A can retrieve the latest social media posts from anyone included in the list of media commentators and / or work colleagues compiled by user 118 and return them, for example, to client device 120 (e.g., by aggregator agent 122). In some implementations, the user's own default social media feed may appear in reverse chronological order with posts from the user's friends and / or people the user follows, and may be searched for response content posted by commentators and / or the user's work colleagues. Alternatively, each personal feed of a commentator and / or the user's work colleagues may be searched for response content. In various implementations, the manner in which social media data feeds (or any other data feeds) are searched may depend on permissions granted by the entity providing the data feed. For example, a social media data feed service may provide an API that can be tapped into to retrieve response information. Alternatively, if the social media data feed service so desires, the aggregator agent 122 may prevent the acquisition of response content by using technical means such as a fully automated public Turing test (CAPTCHA) to distinguish between computer and human users and to provide terms of use.

[0055] A sports data feed server(s) 342B may, for example, retrieve and return the most recent content provided by any of the commentators to a client device 120 (e.g., via an aggregator agent 122). While none of the workplace colleagues may publish content to such sports websites, those colleagues may, for example, post comments in the comment section of the sports website at the bottom of an article. In some such cases, the comments of those workplace colleagues may be retrieved for response data using the techniques described herein. A microblog server(s) 342C may, for example, retrieve and return the most recent microblogging posts by any of the identified commentators or workplace colleagues to a client device 120 (e.g., via an aggregator agent 122).

[0056] In some implementations, all of these returned messages may be matched and / or aggregated and presented to the user 118 audibly or visually. In other implementations, these returned messages may be compared to identify the most recent one, and only that message may be presented to the user 118. For example, if the client device 120 is a standalone interactive speaker without a display function, as in Figure 3 (or, for example, in a vehicle), it may be advantageous to minimize the amount of output to avoid overwhelming or distracting the user 118, in which case the most recent content from any of the domains may be read aloud. In addition or instead, the returned content may be processed by, for example, an ML module 108 using an inter-sequence machine learning model trained to paraphrase and / or summarize longer text content, so that the content ultimately presented to the user 118 is shorter and / or more concise.

[0057] Figure 4 is a flowchart illustrating an exemplary method 400 for implementing a selected aspect of the disclosure in an implementation described herein. For convenience, the operations in the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of an interdomain knowledge system 102. Furthermore, the operations of method 400 are shown in a particular order, but this is not limiting. One or more operations can be rearranged, omitted, or added.

[0058] In block 402, the system may acquire natural language input containing a query seeking information. The user (118) may type such natural language and / or provide spoken words, which can be processed, for example, by the STT module 330 to generate speech recognition output. In some cases, the natural language input may identify one or more entities related to the search. These entities may include, for example, individuals who post content to data feed services. These individuals may be friends, commentators, academics, journalists, politicians, celebrities, or other public or private figures. In addition to or instead of this, the natural language input may identify one or more data feed services to be searched. For example, an individual may request content from all of their subscribed social media feeds.

[0059] Regardless of the format of the natural language input, in block 404, for example, the ML module 108 can perform natural language processing (NLP) on the natural language input to generate data feed-independent aggregator embeddings. This NLP may be performed using a machine learning model such as an RNN or a transformer. As previously described, when the natural language input contains sufficient detail, the data feed-independent aggregator embeddings may be sufficient on their own to be processed using a domain model to generate a probability distribution. Alternatively, the data feed-independent aggregator embeddings may be relatively ambiguous, as long as they are semantically similar to previous data feed-independent aggregator embeddings generated from "long tail" or other low-detail natural language queries, which are mapped (or combined) with action embeddings generated by the same or different users.

[0060] In block 406, the system can select multiple data feed services on which content should be aggregated, for example, by the aggregator agent 122 or the domain module 104. The data feed services may be selected based on various signals or factors, such as the content of the natural language input, synonyms of tokens in the natural language input, the user's context (e.g., working, driving a vehicle, time), permissions granted by the user (e.g., the user may have already selected the aggregator agent 122 to various selected data feed services), or the user's contact list (which may indicate which data feed services the user's friends push content on). In various implementations, each selected data feed service may include its own data feed service action space of actions that can be performed to access the data communicated through that data feed service. These actions may include, for example, actions that can be performed by a client application (e.g., actions that can be performed using voice, keyboard, pointing device, etc.) or actions that can be performed on the server side, for example, actions that can be performed via API calls.

[0061] In block 408, the system may process feed-independent aggregator embeddings based on multiple domain-specific machine learning models corresponding to multiple data feed services selected in block 406, for example, by the ML module 108 or the aggregator agent 122. Each domain-specific machine learning model may be trained to translate between its respective data feed service action space and a data feed-independent semantic embedding space (e.g., reference numeral 338 in Figure 3), which includes data feed-independent aggregator embeddings.

[0062] Based on the processing in block 408, in block 410 the system may select and execute one or more actions from each of the data feed service action spaces to aggregate data in response to a query from multiple data feed services. For example, the processing in block 408 may generate a probability distribution across the actions in each applicable action space. The action with the highest probability may be executed first, followed by actions with lower probabilities (but still exceeding some minimum threshold). In some cases, the same domain model may be iteratively applied to a sequence of states. This sequence of states may represent, for example, the evolving state of a client application communicating with a data feed service, the evolving state of interaction or exchange with a data feed service, or the changing context of a user or the client device on which they operate. The iteratively applied domain model may, in some implementations, be trained using reinforcement learning, but this is not required.

[0063] In block 412, the system may, for example, use interface module 106 to audibly display the aggregated response data as output on a screen or the like.

[0064] Figure 5 is a block diagram of an exemplary computing device 510 that can be optionally used to perform one or more embodiments of the techniques described herein. In some implementations, one or more of the client computing devices 120-1 to 120-P, the interdomain knowledge system 102, and / or other components may comprise one or more components of the exemplary computing device 510.

[0065] The computing device 510 typically includes at least one processor 514 that communicates with several peripheral devices via a bus subsystem 512. These peripheral devices may include, for example, a storage subsystem 524 including a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable the computing device 510 to interact with a user. The network interface subsystem 516 provides an interface to an external network and is coupled to a corresponding interface device in another computing device.

[0066] The user interface input device 522 may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphic tablets, scanners, touchscreens integrated into displays, voice input devices such as voice recognition systems and microphones, and / or other types of input devices. Generally, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 510 or a communication network.

[0067] The user interface output device 520 may include non-visual displays such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include flat panel devices such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs), projection devices, or any other mechanism for creating visible images. The display subsystem may also provide non-visual displays via audio output devices, etc. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 510 to a user or another machine or computing device.

[0068] The storage subsystem 524 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 524 may include logic for performing a selected embodiment of method 400 shown in Figure 4.

[0069] These software modules are generally executed by processor 514 alone or in combination with other processors. The memory 525 used within the storage subsystem 524 may include several memories, including main random access memory (RAM) 530 for storing instructions and data during program execution, and read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 526 can provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules performing functions in a particular implementation form may be stored by the file storage subsystem 526 within the storage subsystem 524 or on other machines accessible by processor 514.

[0070] The bus subsystem 512 provides a mechanism for various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0071] The computing device 510 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly changing nature of computers and networks, the description of the computing device 510 shown in Figure 5 is intended only as a specific example to illustrate several implementation forms. Many other configurations of the computing device 510 are possible, having more or fewer components than the computing device shown in Figure 5.

[0072] While several implementations have been described and illustrated herein, various other means and / or structures can be utilized to perform the function and / or to obtain one or more of the results and / or benefits described herein, and each of such variations and / or modifications shall be considered within the scope of the implementations described herein. More generally, all parameters, dimensions, materials and configurations described herein are intended to be illustrative, and the actual parameters, dimensions, materials and / or configurations are intended to depend on one or more specific applications in which this teaching is used. A person skilled in the art will recognize or be able to investigate many equivalents to the specific implementations described herein using only routine experimentation. Therefore, it should be understood that the above implementations are presented only as examples, and that within the scope of the appended claims and their equivalents, implementations may be practiced in ways other than those specifically described and claimed. The implementations of this disclosure apply to each individual feature, system, article, material, kit and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not inconsistent with each other.

Claims

1. A method that is executed using one or more processors, Obtaining natural language input that includes information queries, The process involves performing natural language processing (NLP) on the aforementioned natural language input to generate a data feed-independent aggregator embedding, Selecting multiple data feed services, wherein each of the selected data feed services includes its own data feed service action space, and each data feed service action space includes actions that can be performed to access data communicated through each of the selected data feed services. Processing the feed-independent aggregator embeddings based on multiple domain-specific machine learning models corresponding to the multiple data feed services, wherein each domain-specific machine learning model is trained to translate between the respective data feed service action space and the data feed-independent semantic embedding space, which includes the data feed-independent aggregator embeddings. Based on the above processing, one or more actions are selected and executed from each of the data feed service action spaces to aggregate the data responding to the query from the multiple data feed services. The aggregated response data is presented as output, A method that includes this.

2. The method according to claim 1, wherein the plurality of data feed services are selected based on entity identifiers included in the query for obtaining the information.

3. The method according to claim 1 or 2, wherein the query for the aforementioned information includes a request for social media posts from a specific individual, the selected data feed services include two or more social media services, and the two or more social media services are selected based on the specific individual's membership in the two or more social media services.

4. The method according to claim 3, wherein, with respect to a given social media service among the two or more social media services, one or more actions selected from the data feed service action space of the given social media service include accessing one or more posts by the particular individual from the posting history of the particular individual.

5. The method according to claim 3 or 4, wherein, with respect to a given social media service among the two or more social media services, one or more actions selected from the data feed service action space of the given social media service include filtering one or more posts by the specific individual from a general data feed on the given social media service provided to the user who issued the natural language input.

6. The method according to any one of claims 1 to 5, wherein the plurality of data feed services are selected based on a lookup table controlled by the user who issued the natural language input.

7. The method according to claim 6, wherein the lookup table includes the user's contact list.

8. The method according to any one of claims 1 to 7, wherein the plurality of data feed services are selected based on the user who issued the natural language input and has previously provided the aggregator agent with permission to access the plurality of data feed services.

9. The method according to any one of claims 1 to 8, wherein at least one of the data feed services comprises a virtual space that forms part of a larger metaverse comprising multiple virtual spaces.

10. The method according to any one of claims 1 to 9, wherein the aggregated response data is presented to the user who issued the natural language input as part of a metaverse graphical user interface.

11. The method according to any one of claims 1 to 10, wherein the plurality of data feed services are selected based on the browsing history of the user who issued the natural language input.

12. A system comprising one or more processors and memory for storing instructions, wherein, in response to the execution of the instructions, the one or more processors, It obtains natural language input that includes a query seeking information, Natural language processing (NLP) is performed on the aforementioned natural language input to generate a data feed-independent aggregator embedding. The process involves selecting multiple data feed services, each of which is to be selected, including its own data feed service action space, and each data feed service action space including actions that can be performed to access data communicated through each of the selected data feed services. Processing the feed-independent aggregator embeddings based on multiple domain-specific machine learning models corresponding to the multiple data feed services, wherein each domain-specific machine learning model is trained to translate between the respective data feed service action space and the data feed-independent semantic embedding space, which includes the data feed-independent aggregator embeddings, and the feed-independent aggregator embeddings are processed based on these multiple domain-specific machine learning models. From the aforementioned multiple data feed services, in order to aggregate the data that responds to the query, one or more actions are selected and executed from each of the data feed service action spaces. The aggregated response data is presented as output. system.

13. The system according to claim 12, wherein the plurality of data feed services are selected based on entity identifiers included in the query for obtaining the information.

14. The system according to claim 12 or 13, wherein the query for the aforementioned information includes a request for social media posts from a specific individual, the selected data feed services include two or more social media services, and the two or more social media services are selected based on the specific individual's membership in the two or more social media services.

15. The system according to claim 14, wherein, for a given social media service among the two or more social media services, one or more actions selected from the data feed service action space of the given social media service include accessing one or more posts by the particular individual from the posting history of the particular individual.

16. The system according to claim 14 or 15, wherein, for a given social media service among the two or more social media services, one or more actions selected from the data feed service action space of the given social media service include filtering one or more posts by the specific individual from a general data feed on the given social media service provided to the user who issued the natural language input.

17. The system according to any one of claims 12 to 16, wherein the plurality of data feed services are selected based on a lookup table controlled by the user who issued the natural language input.

18. The system according to claim 17, wherein the lookup table includes the user's contact list.

19. The system according to any one of claims 12 to 16, wherein the plurality of data feed services are selected based on the user who issued the natural language input and has previously provided the aggregator agent with permission to access the plurality of data feed services.

20. A non-temporary computer-readable medium containing instructions, wherein the instructions are provided to the processor in response to the processor's execution of the instructions. It obtains natural language input that includes a query seeking information, Natural language processing (NLP) is performed on the aforementioned natural language input to generate a data feed-independent aggregator embedding. The process involves selecting multiple data feed services, each of which is to be selected, including its own data feed service action space, and each data feed service action space including actions that can be performed to access data communicated through each of the selected data feed services. Processing the feed-independent aggregator embeddings based on multiple domain-specific machine learning models corresponding to the multiple data feed services, wherein each domain-specific machine learning model is trained to translate between the respective data feed service action space and the data feed-independent semantic embedding space, which includes the data feed-independent aggregator embeddings, and the feed-independent aggregator embeddings are processed based on these multiple domain-specific machine learning models. From the aforementioned multiple data feed services, in order to aggregate the data that responds to the query, one or more actions are selected and executed from each of the data feed service action spaces. The aggregated response data is presented as output. Non-temporary computer-readable media.

Images