Method and system for facilitating secure exchange of data with self servers

Abstract

Implementations described herein utilize independent servers to facilitate secure exchange of data between multiple different parties. A standalone server receives client data to be transmitted to a given third-party application via an automated assistant application executing at least in part at a client device. The independent server processes the client data using the first encoder-decoder model to generate opaque client data, and transmits the opaque client data to a given third party application without transmitting any of the client data. Further, the standalone server receives response data via the given third party application, the response data being generated based on the opaque client data and to be transmitted back to the client device. The independent server processes the response data using a second encoder-decoder model to generate opaque response data, and transmits the opaque response data to the client device without transmitting any of the response data.

Description

Background Art

[0001] Humans may use interactive software applications, referred to herein as "automated assistants" (also referred to as "chatbots," "interactive personal assistants," "intelligent personal assistants," "personal voice assistants," "conversational agents," etc.) to conduct human-computer conversations. Automated assistants typically rely on a pipeline of components to interpret and respond to spoken utterances and / or other user inputs (e.g., typed inputs, touch inputs, etc.). For example, an automatic speech recognition (ASR) engine may process audio data corresponding to a user's spoken utterance to generate ASR outputs, such as phonetic hypotheses (i.e., sequences of terms and / or other tokens) for the spoken utterance. Further, a natural language understanding (NLU) engine may process the ASR outputs (or touch / typed inputs) to generate NLU outputs, such as one or more predicted intents expressed by a user when providing a spoken utterance (and / or other user input), and, optionally, slot values ​​for parameters associated with each of the one or more predicted intents. In addition, a fulfillment engine may be used to process the NLU outputs to obtain fulfillment outputs, such as content to be provided for presentation to the user in response to the spoken utterance and / or actions to be performed by an automated assistant or another computing device in response to the spoken utterance.

[0002] In many instances, the automated assistant may interact with various third parties when fulfilling a user's request and / or when proactively providing content for presentation to the user. For example, assume that a user provides the spoken utterance "Assistant, play Crazy using Example Music Streaming Platform". In this example, the automated assistant may generate an NLU output that includes at least a predicted music playback intent with a slot value of "Crazy" for a song parameter to be played using the "Example Music Streaming Platform". Although the user does not specify any slot value for the artist parameter, the automated assistant may infer various artist slot values ​​for the artist parameter based on the client device data to determine multiple interpretations of the spoken utterance. Further, the automated assistant may transmit one or more of the multiple interpretations to the "Example Music Streaming Platform". However, when one or more of the multiple interpretations of the spoken utterance are transmitted to the “Example Music Streaming Platform,” the user's client device data may be vulnerable to compromise by the “Example Music Streaming Platform” and / or malicious actors because the “Example Music Streaming Platform” and / or malicious actors may be able to infer some client device data (e.g., ASR output, NLU output, user preferences, user history, etc.) transmitted over the network that was used to determine one or more of the multiple interpretations of the spoken utterance, even when the underlying client device data itself is not transmitted over the network.

[0003] Continuing with the above example, assume that the automated assistant wants to proactively provide song recommendations for presentation to the user after providing "Crazy" for audible playback to the user. In this example, the automated assistant may determine group data for a given user group to which the user belongs based on each of the users of the given user group having the same or similar music preferences across multiple distinct user cohorts. Further, the automated assistant may transmit the group data to the "Example Music Streaming Platform" to obtain one or more song recommendations for the user based on the group data. However, when transmitting the group data for the given group to which the user belongs to the "Example Music Streaming Platform," not only may the user's client device data be vulnerable to compromise by the "Example Music Streaming Platform" and / or malicious actors, but corresponding client device data of other users may also be vulnerable because those other users also belong to the given user cohort.

[0004] However, if this client device data is not shared, the request may be suboptimally fulfilled (e.g., a song named "Crazy" by an artist that was not intended by the user is played), and / or the content that is proactively provided for presentation to the user may be suboptimal (e.g., song recommendations from a music genre that the user does not like). As a result, the user may resubmit various spoken utterances (and optionally include more specific slot values) to the automated assistant and / or manually interact with "Example StreamingPlatform", thereby increasing the amount of user input received at the client device and wasting computing and / or network resources unnecessarily. Further, the user may become frustrated not only with the automated assistant, but also with "Example Streaming Platform", thereby negatively impacting the user's user experience. Therefore, there is a need in the art for techniques to facilitate the secure exchange of such data. Summary of the invention

[0005] The implementation described herein relates to utilizing an independent server to facilitate the secure exchange of data between multiple disparate parties. For example, an independent server may facilitate the secure exchange of data between an automated assistant for a given client device of a given user and a given third-party application that is at least partially executed. An independent server may receive client device data to be transmitted to a given third-party application via an automated assistant. Further, an independent server may process client device data using a trained encoder-decoder model to generate opaque client device data that is an opaque version of the client device data, and may transmit the opaque client device data to a given third-party application without transmitting any underlying client device data in the underlying client device data. Transmitting the opaque client device data to a given third-party application may cause the given third-party application to process the opaque client device data to generate third-party response data, and transmit the third-party response data back to the independent server. An independent server may receive third-party response data via a given third-party application, which is generated based on the opaque client data and will be transmitted back to the client device. In addition, the independent server may process the third-party response data using the trained encoder-decoder model and / or the additionally trained encoder-decoder model to generate opaque third-party response data that is an opaque version of the third-party response data, and may transmit the opaque third-party response data back to the given client device without transmitting any of the underlying third-party response data. This enables the given client device and / or an automated assistant implemented at least in part on the given client device to generate content and cause the content to be provided for presentation to the user.

[0006] Although the above example is described with respect to a standalone server that facilitates the secure exchange of data between an automated assistant and a given third-party application, it should be understood that this is for purposes of example and is not meant to be limiting. As another example, a standalone server may additionally or alternatively be utilized to facilitate the secure exchange of data between multiple disparate third-party applications, such that the secure exchange of data via the standalone server may be provided as a service.

[0007] In various implementations, encoder-decoder models and / or additional encoder-decoder models for facilitating the secure exchange of data between an automated assistant and a given third-party application may be trained during a joint session at an independent server. When training the encoder-decoder model and / or additional encoder-decoder model, the independent server may obtain a corresponding word element set corresponding to the vocabulary, a corresponding client device data set, and a corresponding third-party response data set. The corresponding word element set corresponding to the vocabulary may include words, phrases, symbols, numbers, and / or any other representations that can encode the underlying meaning of the data. The corresponding word element set may be agreed upon by at least the automated assistant and the given third-party application to facilitate the secure transmission of data between the automated assistant and the given third-party application in a two-way manner. Each client device data set in the corresponding client device data set may include given user request data, given user group data, given user profile data, and / or other given client device data that can be transmitted to a given third-party application. The corresponding third-party response data set may include given third-party data, given third-party user group data, given third-party user profile data, and / or other given third-party data that can be transmitted back to a given client device and / or an automated assistant.

[0008] In some versions of these implementations, multiple automated assistant application training instances may be generated based on a corresponding word-gram set and a corresponding client device data set. The multiple automated assistant application training instances may then be used to train an encoder-decoder model to generate an opaque version of the client device data. For example, a given automated assistant application training instance may include a training instance input and a training instance output. The training instance input may include a given corresponding client device data set from among the corresponding client device data set, and the training instance output may include one or more word-grams from among the corresponding word-gram set. Therefore, when the encoder-decoder model is trained based on a given automated assistant application training instance, the encoder-decoder model learns to generate an opaque version of a given corresponding client device data set represented by one or more word-grams.

[0009] In some further versions of these implementations, multiple given third-party application training instances may be generated based on corresponding word-gram sets and corresponding client device data sets. Multiple given third-party application training instances may then be used to train encoder-decoder models and / or additional encoder-decoder models to generate opaque versions of third-party device data. For example, a given third-party application training instance may include training instance input and training instance output. The training instance input may include a given corresponding third-party data set from a corresponding third-party response data set, and the training instance output may include one or more additional word-grams from a corresponding word-gram set, the one or more additional word-grams including at least one word-gram unique to one or more word-grams of the training instance output of a given automated assistant application training instance. Therefore, when the encoder-decoder model and / or additional encoder-decoder model are trained based on a given third-party application training instance, the encoder-decoder model and / or additional encoder-decoder model learn to generate an opaque version of given third-party application data represented by one or more additional word-grams. It is noteworthy that a given third-party application training instance may include corresponding third-party response data previously generated in response to a corresponding client device data set used to generate a given automated assistant application training instance. Thus, the automated assistant application training instance and the given third-party application training instance can be considered a training instance pair. Further, since these training instances utilize corresponding word-gram sets mutually agreed upon by both the automated assistant and the given third-party application, the corresponding machine learning models implemented by these disparate parties can be able to handle the opaque version of the data generated by these encoder-decoder models.

[0010] In various implementations, and following a joint training session, the independent server may utilize the encoder-decoder model and / or the additional encoder-decoder model to facilitate the secure exchange of data between multiple disparate parties participating in the joint training session. Notably, the encoder-decoder model and / or the additional encoder-decoder model may be utilized in response to an explicit user request for data from a given third-party application (e.g., "Assistant, play a song using Example Music Streaming Platform," where "Example Streaming Platform" corresponds to a given third-party application) and / or an implicit user request (e.g., when the automated assistant identifies a need for a suggestion or recommendation to be provided to a given user of a given client device). Thus, the independent server may utilize the encoder-decoder model and / or the additional encoder-decoder model for the purposes of ranking, natural language understanding (NLU), trend analysis, recommendations, suggestions, and / or any other scenario requiring the transfer of data between multiple disparate parties.

[0011] In some versions of these implementations, the independent server may be utilized only in response to determining that the data to be transmitted between the multiple disparate parties is privacy-sensitive data (such as data including data personal to a given user of a given client device). Otherwise, the data may be deemed to be non-privacy-sensitive data and the data may be transmitted directly from one of the multiple disparate parties to another of the multiple disparate parties. Thus, the independent server may be selectively utilized to facilitate the secure transmission of data only in the event that the data being transmitted is deemed to be privacy-sensitive data.

[0012] By using the techniques described herein, one or more technical advantages may be achieved. As a non-limiting example, the techniques described herein enable secure transmission of data between multiple disparate parties (such as between an automated assistant and a given third-party application, between a given third-party application, or between additional given third-party applications). For example, instead of directly sharing data between the multiple disparate parties, an opaque version of the data may be generated using an independent server before the data is transmitted from one of the multiple disparate parties to another. Therefore, direct sharing of data between the multiple disparate parties may be reduced or eliminated, resulting in improved security of the user's data. Further, content generated based on the opaque version of the data provides the same or similar accuracy and / or recall, because the various machine learning models of the multiple disparate parties are able to handle the opaque version of the data due to the use of a commonly agreed upon set of word elements when generating the opaque version of the data during the joint training session, as described herein. Therefore, not only the security of the user's data may be improved, but also the user experience may be improved.

[0013] In some implementations, the term "first-party (1P) application" may refer to a computing agent, system, or software application developed and / or maintained by the same entity that develops and / or maintains the automated assistant described herein. In some implementations, the term "third-party (3P) application" may refer to a computing agent, system, or software application developed and / or maintained by an entity different from the entity that develops and / or maintains the automated assistant described herein. In additional or alternative implementations, the term "third-party (3P) application" refers to any computing agent, system, or software application developed and / or maintained by the same entity that develops and / or maintains the automated assistant described herein but is computationally distinct from the automated assistant (e.g., hosted at one or more servers that are physically distinct from the client device and / or any server that hosts the automated assistant). In some implementations, the term "stand-alone server" may refer to a computing agent or system developed and / or maintained to implement a secure data exchange system. A stand-alone server may be developed and / or maintained by an entity that is the same entity that develops and / or maintains the automated assistant described herein or by an entity that develops and / or maintains the automated assistant described herein.

[0014] Each of the software applications described herein (e.g., automated assistant applications, IP applications, 3P applications, and / or any other software applications described herein) may be implemented by a computing device or system that includes at least a memory, a processor for executing instructions stored in the memory, and one or more network interfaces for receiving and / or transmitting data via a wired or wireless local area network ("LAN", including Wi-Fi LAN, mesh network, Bluetooth, near field communication, etc.) and / or a wide area network ("WAN", including the Internet) and / or an application programming interface. Further, each of the software applications described herein may be implemented locally at a client device, remotely from the client device (e.g., at a remote server), or both (e.g., in a distributed manner).

[0015] The above description is provided as an overview of only some of the implementations disclosed herein. Those implementations and other implementations are described in more detail herein.

[0016] It should be appreciated that the techniques disclosed herein may be implemented locally on a client device, remotely by a server connected to the client device via one or more networks, or both. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Depicted is a block diagram of an example environment showing various aspects of the disclosure and in which implementations disclosed herein may be implemented.

[0018] Figure 2 Describes the use of various implementations Figure 1 Various components are used to demonstrate example process flows of various aspects of the present disclosure.

[0019] Figure 3 Depicted is a flow diagram illustrating an example method of facilitating secure exchange of data from the perspective of a client device in accordance with various implementations.

[0020] Figure 4 Depicted is a flow diagram illustrating an example method of facilitating secure exchange of data from the perspective of an independent server in accordance with various implementations.

[0021] Figure 5 Depicted are flow diagrams illustrating example methods of training various encoder-decoder models at a standalone server, which may subsequently be used to facilitate secure exchange of data, according to various implementations.

[0022] Figure 6 Depicted are example architectures for computing devices in accordance with various implementations. DETAILED DESCRIPTION

[0023] Now turn to Figure 1 , depicts a block diagram of an example environment that illustrates various aspects of the present disclosure and in which implementations disclosed herein may be implemented. The example environment includes a client device 110 and a secure data exchange system 180. In some implementations, the secure data exchange system 180 may be implemented locally at the client device 110. In additional or alternative implementations, the secure data exchange system 180 may be implemented remotely from the client device 110, such as Figure 1 In these implementations, client device 110 and secure data exchange system 180 may be communicatively coupled to each other via one or more networks 199, such as one or more wired or wireless local area networks ("LANs," including Wi-Fi LANs, mesh networks, Bluetooth, near field communications, etc.) or wide area networks ("WANs," including the Internet).

[0024] The client device 110 may be, for example, one or more of a desktop computer, a laptop computer, a tablet computer, a mobile phone, a vehicle's computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (optionally with a display), a smart home appliance (such as a smart TV), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0025] Client device 110 may execute automated assistant client 114. Instances of automated assistant client 114 may be applications that are separate from (e.g., installed “on top of”) an operating system of client device 110, or may alternatively be implemented directly by an operating system of client device 110. Automated assistant client 114 may interact with secure data exchange system 180 implemented locally at client device 110 and / or remotely via one or more of networks 199, such as Figure 1 Automated assistant client 114 (and optionally through its interaction with other remote systems (e.g., servers)) can form what appears to be a logical instance of automated assistant 115 from the user's perspective, with which the user can engage in a human-computer conversation. Figure 1 10 and is surrounded by a dashed line that includes an automated assistant client 114 of a client device 110 and a secure data exchange system 180. Thus, it should be understood that a user interacting with an automated assistant client 114 executing on a client device 110 may actually be interacting with his or her own logical instance of an automated assistant 115 (or a logical instance of an automated assistant 115 shared among a family or other group of users). For the sake of brevity and simplicity, automated assistant 115 as used herein will refer to an automated assistant client 114 executing locally on a client device 110 and / or one or more remote servers that may implement secure data exchange system 180.

[0026] In various implementations, the client device 110 may include a user input engine 111 configured to detect user input provided by a user of the client device 110 using one or more user interface input devices. For example, the client device 110 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to the user's spoken words or other sounds in the environment of the client device 110. Additionally or alternatively, the client device 110 may be equipped with one or more visual components that are configured to capture visual data corresponding to images and / or movements (e.g., gestures) detected in a field of view of one or more of the visual components. Additionally or alternatively, the client device 110 may be equipped with one or more touch-sensitive components (e.g., a keyboard and mouse, a stylus, a touch screen, a touch panel, one or more hardware buttons, etc.) that are configured to capture signals corresponding to touch input directed to the client device 110.

[0027] In various implementations, the client device 110 may include a rendering engine 112 configured to provide content for audible and / or visual presentation to a user of the client device 110 using one or more user interface output devices. For example, the client device 110 may be equipped with one or more speakers that enable content to be provided for audible presentation to a user via the client device 110. Additionally or alternatively, the client device 110 may be equipped with a display or projector that enables content to be provided for visual presentation to a user via the client device 110. Additionally or alternatively, the client device 110 may be equipped with one or more tactile sensors that enable content to be provided for presentation to a user via one or more of the tactile sensors of the client device 110.

[0028] In various implementations, client device 110 may include one or more presence sensors 113 configured to provide a signal indicating detected presence, particularly human presence, if approved by the corresponding user. In some of those implementations, automated assistant 115 may identify client device 110 (or another computing device associated with a user of client device 110) to satisfy the spoken utterance based at least in part on the user's presence at client device 110 (or at another computing device associated with a user of client device 110). The spoken utterance may be satisfied by rendering responsive content at client device 110 and / or the other computing device associated with the user of client device 110 (e.g., via rendering engine 112), causing client device 110 and / or the other computing device associated with the user of client device 110 to be controlled, and / or causing client device 110 and / or the other computing device associated with the user of client device 110 to perform any other action to satisfy the spoken utterance. In various implementations, automated assistant 115 may utilize data determined based on presence sensor 113 to determine client devices 110 (or other computing devices) to utilize when satisfying a spoken utterance based on where the user is or was recently, and provide corresponding commands only to client devices 110 (or those other computing devices).

[0029] In some additional or alternative implementations, automated assistant 115 may utilize data determined based on presence sensor 113 to determine whether any user (any user or a specific user) is currently proximate to client device 110 (or other computing device), and may optionally refrain from providing data to and / or from client device 110 (or other computing device) based on the user being proximate to client device 110 (or other computing device). In some additional or alternative implementations, automated assistant 115 may utilize data determined based on presence sensor 113 to determine whether any user (any user or a specific user) is currently proximate to client device 110 (or other computing device), and may optionally obtain recommendations and / or suggestions to be proactively provided at client device 110 (or other computing device) based on the user being proximate to client device 110 (or other computing device).

[0030] Presence sensor 113 may take various forms. For example, client device 110 may detect the presence of a user using one or more of the user interface input components described above with respect to user input engine 111. Additionally or alternatively, client device 110 may be equipped with other types of light-based presence sensors 113, such as a passive infrared ("PIR") sensor that measures infrared ("IR") light radiated from objects within its field of view.

[0031] Additionally or alternatively, in some implementations, presence sensor 113 may be configured to detect other phenomena associated with human presence or device presence. For example, in some embodiments, client device 110 may be equipped with presence sensor 113 that detects various types of wireless signals (e.g., waves, such as radio waves, ultrasound waves, electromagnetic waves, etc.) emitted, for example, by other computing devices carried / operated by a user (e.g., mobile devices, wearable computing devices, etc.) and / or other computing devices. For example, client device 110 may be configured to emit waves that are imperceptible to humans, such as ultrasound waves or infrared waves, which may be detected by other computing devices (e.g., via ultrasound / infrared receivers, such as microphones with ultrasound capabilities).

[0032] Additionally or alternatively, the client device 110 may emit other types of waves that are not perceptible to humans, such as radio waves (e.g., Wi-Fi, Bluetooth, cellular, etc.), which may be detected by other computing devices carried / operated by the user (e.g., mobile devices, wearable computing devices, etc.) and used to determine the specific location of the user. In some implementations, GPS and / or Wi-Fi triangulation may be used to detect the location of a person, for example, based on GPS and / or Wi-Fi signals to / from the client device 110. In other implementations, other wireless signal characteristics (such as flight time, signal strength, etc.) may be used by the client device 110, either alone or in combination, to determine the location of a specific person based on signals emitted by other computing devices carried / operated by the user.

[0033] Additionally or alternatively, in some implementations, client device 110 may perform speaker identification (SID) to recognize the user from the user's voice based on audio data captured via a microphone of client device 110, and / or may perform facial identification (FID) to recognize the user from his / her face captured based on visual data captured via a visual component of client device 110. In some implementations, the movement of the speaker may then be determined, for example, by presence sensor 113 of client device 110 (and optionally, a GPS sensor, Soli chip, and / or accelerometer of client device 110). In some implementations, based on such detected movement, the user's location may be predicted, and this location may be assumed to be the user's location when any content is to be rendered at client device 110 and / or other computing devices based at least in part on the proximity of client device 110 and / or other computing devices to the user's location. In some implementations, it may be simply assumed that the user is in the last location where he / she interacted with automated assistant 115, especially in situations where not much time has passed since the last interaction.

[0034] Further, client device 110 and / or secure data exchange system 180 may include one or more memories for storing data and / or software applications, one or more processors for accessing data and executing software applications, and / or other components that facilitate communication over one or more of networks 199. In some implementations, one or more of the software applications may be installed locally at client device 110, while in other implementations, one or more of the software applications may be remotely hosted (e.g., by one or more servers) and may be accessible by client device 110 over one or more of networks 199. Operations performed by client device 110, other computing devices, and / or automated assistant 115 may be distributed across multiple computer systems. Automated assistant 115 may be implemented, for example, as a distributed system on client device 110 and / or over a network (e.g., Figure 1 A computer program running on one or more computers located in one or more locations coupled to each other by a network 199).

[0035] In some implementations, the operations performed by automated assistant 115 may be implemented locally at client device 110 via automated assistant client 114. Figure 1 As shown, automated assistant client 114 may include automatic speech recognition (ASR) engine 120A1, natural language understanding (NLU) engine 130A1, fulfillment engine 140A1, and text-to-speech (TTS) engine 150A1. In some implementations, the operations performed by automated assistant 115 may be distributed across multiple computer systems, such as when secure data exchange system 180 is implemented remotely from client device 110 (e.g., Figure 1 In these implementations, automated assistant 115 may additionally or alternatively utilize ASR engine 120A2, NLU engine 130A2, fulfillment engine 140A2, and TTS engine 150A2 of secure data exchange system 180.

[0036] Each of these engines may be configured to perform one or more functions. For example, the ASR engine 120A1 and / or 120A2 may use an ASR model (e.g., a recurrent neural network (RNN) model, a transformer model, an RRN-transducer (RNN-T) model, and / or any other type of machine learning (ML) model capable of performing ASR) to process audio data that captures spoken speech and is generated by a microphone of the client device 110 to generate an ASR output. Further, the NLU engine 130A1 and / or 130A2 may use an NLU model (e.g., a long short-term memory (LSTM), a gated recurrent unit (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and / or grammatical rules to process the ASR output to generate an NLU output. In addition, the fulfillment engine 140A1 and / or 140A2 may generate a fulfillment output based on fulfillment data generated according to a stream of NLU outputs. The fulfillment data may be generated using, for example, automated assistant 115, one or more first-party (IP) applications 171, and / or one or more third-party (3P) applications 172. Finally, TTS engines 150A1 and / or 150A2 may use TTS models to process text data (e.g., text formulated by automated assistant 115, IP application 171, and / or 3P application 172) to generate synthesized speech audio data including computer-generated synthesized speech. Notably, the ML models stored in ML model database 115A may be on-device ML models stored locally at client device 110, or shared ML models accessible to both client device 110 and / or a remote system (e.g., a server).

[0037] In various implementations, the ASR output may include, for example, speech hypotheses (e.g., lexical hypotheses and / or transcription hypotheses) predicted to correspond to the user's spoken utterance captured in the audio data, one or more corresponding prediction values ​​(e.g., probabilities, log-likelihoods, and / or other values) for each of the speech hypotheses, a plurality of phonemes predicted to correspond to the user's spoken utterance captured in the audio data, and / or other ASR outputs. In some versions of those implementations, the ASR engines 120A1 and / or 120A2 may select one or more of the speech hypotheses as recognized text corresponding to the spoken utterance (e.g., based on the corresponding prediction values).

[0038] In various implementations, the NLU output may include, for example, annotated recognition text, which includes one or more annotations of the recognition text for one or more (e.g., all) terms in the terms of the recognition text. For example, the NLU engine 130A1 and / or 130A2 may include a part-of-speech tagger (not depicted) that is configured to annotate terms by the grammatical role of the terms. Additionally or alternatively, the NLU engine 130A1 and / or 130A2 may include an entity tagger (not depicted) that is configured to annotate entity references in one or more fragments of the recognition text, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and imaginary), etc. In some implementations, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some implementations, the knowledge graph may include nodes representing known entities (and in some cases, representing entity attributes), and edges connecting nodes and representing relationships between entities.

[0039] In some implementations, the entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to a class of entities, such as persons) and / or at a lower level of granularity (e.g., to enable identification of all references to a specific entity, such as a specific person). The entity tagger may rely on the content of the natural language input to resolve specific entities, and / or may optionally communicate with a knowledge graph or other entity database to resolve specific entities. Additionally or alternatively, the NLU engine 130A1 and / or 130A2 may include a coreference resolver (not depicted) that is configured to group or "cluster" references to the same entity based on one or more contextual clues. For example, a coreference resolver may be utilized to resolve the term "them" in the input "buy them" to "buy theatre tickets" based on a mention of "theatre tickets" in a client device notification rendered immediately before receiving the natural language input "buy them". In some implementations, one or more components of NLU engine 130A1 and / or 130A2 may rely on annotations from one or more other components of NLU engine 130A1 and / or 130A2. For example, in some implementations, an entity tagger may rely on annotations from a coreference resolver to annotate all mentions of a particular entity. Also, for example, in some implementations, a coreference resolver may rely on annotations from an entity tagger to cluster references to the same entity.

[0040] although Figure 11 is described with respect to a single client device having a single user, but it should be understood that this is for purposes of example and is not meant to be limiting. For example, one or more additional client devices of a user may also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of a user may form an ecosystem of devices that may employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 (e.g., via network 199). As another example, a given client device may be utilized by multiple users in a shared setting (e.g., a user group, a family).

[0041] In some implementations, automated assistant 115, IP application 171, and / or 3P application 172 may utilize secure data exchange system 180 to ensure that any privacy-sensitive data transmitted between them is transmitted in a privacy-sensitive manner. In particular, automated assistant 115, IP application 171, and / or 3P application 172 may utilize secure data exchange engine 160 of secure data exchange system 180 to ensure that any privacy-sensitive data transmitted between them is transmitted in a privacy-sensitive manner. In various implementations, and as Figure 1 As depicted, secure data exchange engine 160 may include client device data engine 161, 3P application data engine 162, word unit engine 163, ML model training engine 164, ML model engine 165, and feedback engine 166. Figure 2 Additional description is provided regarding how automated assistant 115, IP application 171, and / or 3P application 172 utilize secure data exchange engine 160.

[0042] Now turn to Figure 2 , depicting the use of Figure 1 The various components of the present disclosure are used to illustrate example process flows of various aspects of the present disclosure. In particular, Figure 2 The description is about secure data exchange system 180 being implemented by a separate server remote from client device 110, 1P application 171, and 3P application 172. In some implementations, ML model training engine 164 may initially train various encoder-decoder models during a joint training session between automated assistant 115 and a given one of 3P applications 172.

[0043] For example, assume that secure data exchange system 180 is to be used to facilitate the exchange of data between automated assistant 115 and “Example Music Streaming Platform” (a fictitious 3P application in the 3P application). In this example, a first encoder-decoder model 220 specific to automated assistant 115 for “Example Music Streaming Platform” and a second encoder-decoder model 230 specific to “Example Music Streaming Platform” for automated assistant 115 may be trained during a joint training session. Figure 2 The first encoder-decoder model 220 and the second encoder-decoder model 230 are described with respect to training the automated assistant and the “Example Music Streaming Platform”, but it should be understood that this is for the purpose of example and is not meant to be limiting. The first encoder-decoder model 220 and the second encoder-decoder model 230 may be stored in, for example, the ML model database 165A. For example, during a joint training session, other encoder-decoder models may be trained for the automated assistant, the “Example Music Streaming Platform”, other IP applications in the IP applications 171, and / or other 3P applications in the 3P applications 172 to subsequently facilitate secure exchange of data therebetween.

[0044] During the joint training session, the independent server may obtain a corresponding client device data set, a corresponding 3P response data set, and a corresponding word-gram set corresponding to the vocabulary. In various implementations, the corresponding client device data set may be obtained using client device data engine 161 and may include, for example, user request data 201, user group data 202, and / or user profile data 203 obtained from one or more user profiles accessible to automated assistant 115 (e.g., stored in user profile database 165B). User request data 201 may include: corresponding request data, which is based on one or more corresponding user inputs of the corresponding user detected at the corresponding client device among the multiple client devices and causes automated assistant 115 to interact with the "Example Music Streaming Platform"; corresponding suggestion data or corresponding recommendation data, which is based on the corresponding need identified by automated assistant 115 for suggested content or recommended content from the "Example Music Streaming Platform" to be provided for presentation to the corresponding users of the multiple client devices; and / or any other corresponding data generated locally at one or more client devices among the multiple client devices, which any other corresponding data previously caused automated assistant 115 to interact with the "Example Music Streaming Platform".

[0045] Further, the user group data 202 may include a given group to which the corresponding users of the plurality of client devices are assigned by the automated assistant 115 in a plurality of distinct groups. Each group in the distinct groups may be defined at different granularities, and the user may belong to a single group or to a plurality of groups. For example, it is assumed that the corresponding users of the plurality of client devices implementing the automated assistant 115 are grouped in corresponding groups based on the similar musical interests of the corresponding users. In this example, each of the corresponding users who show an interest in rock music may be assigned to a first group, each of the corresponding users who show an interest in country music may be assigned to a second group, and so on for other music genres. In some of these examples, each of the corresponding users who show an interest in both rock music and country music may be assigned to both the first group and the second group. In other examples, each of the corresponding users who show an interest in both rock music and country music may be assigned to a different third group instead of both the first group and the second group. In other words, the user group data 202 may be data representing a cluster of users based on shared common interests.

[0046] In addition, user profile data 203 may include, for example, corresponding user preferences of corresponding users of the plurality of client devices, corresponding query activities of corresponding users of the plurality of client devices, and / or any other data that may be derived from one or more user profiles accessible to automated assistant 115 (e.g., stored in user profile database 165B). Notably, user profile data 203 may have been previously used by the automated assistant to assign the plurality of users to corresponding groups. However, user profile data 203 may include data that is not readily ascertainable from user group data 202.

[0047] In various implementations, the corresponding 3P response data set may be obtained using 3P application data engine 162, and may include, for example, response data 207, 3P group data 208, and / or 3P profile data 209 obtained from one or more user profiles accessible to the "Example Music Streaming Platform" (e.g., stored in 3P profile database 240). Notably, the corresponding 3P response data set obtained by the independent server may include 3P response data for each client device data set in the corresponding client device data set. In other words, the corresponding client device data set may include data that may be provided by the automated assistant to the "Example Music Streaming Platform" to facilitate fulfillment of corresponding requests from multiple client devices, and the corresponding 3P response data set may include data that may be generated by the "Example Music Streaming Platform" to facilitate fulfillment of corresponding requests from multiple client devices.

[0048] For example, assume that a given user of a given client device previously provided the spoken utterance "Assistant, play some music using Example Music Streaming Platform". In this example, a given corresponding client device data set may include: given user request data, the given user request data including at least the intent of "play music", given user group data indicating that the automated assistant 115 has assigned the given user to a third group of 200 distinct automated assistant groups associated with users who like hard rock music, and given user profile data for the given user, the given user profile data indicating that the user's favorite hard rock artist is "Example Rock Artist". Further, in this example, and based on at least some of the given corresponding client device data sets (e.g., based on at least given user request data that includes at least the intent of "play music"), the given corresponding 3P response data set may include: given 3P response data indicating one or more songs or rankings thereof; given 3P group data indicating that the given user has been assigned to the first group of users who like general rock music among the 50 distinct "Example Music Streaming Platform" groups; and given 3P profile data for the given user indicating that the user's favorite rock music genre is hard rock (e.g., rather than punk rock and / or other rock music genres). Thus, for each of the corresponding client device data sets, there may be a given corresponding 3P response data set.

[0049] Notably, in obtaining the corresponding client device data set and the corresponding 3P response data set, the independent server may utilize various techniques to protect the privacy and security of the data. For example, the independent server may utilize differential privacy techniques to ensure that the client device data is not compromised when obtaining data for these joint training sessions. Additionally or alternatively, the independent server may utilize federated learning techniques to ensure that the client device data is not compromised when obtaining data for these joint training sessions. Thus, the secure data exchange system 180 ensures the privacy and security of the client device data not only at inference time but also during the joint training session.

[0050] In various implementations, the corresponding word-gram set corresponding to the vocabulary may be obtained from word-gram database 163A using word-gram engine 163. The corresponding word-gram set corresponding to the vocabulary may be a set of word-grams mutually agreed upon between automated assistant 115 and “Example Music Streaming Platform” that enables secure exchange of data between automated assistant 115 and “Example Music Streaming Platform” in a bidirectional manner via secure data exchange system 180 after training first encoder-decoder model 220 and second encoder-decoder model 230. The word-gram set corresponding to the vocabulary may include word-grams corresponding to words, phrases, symbols, numbers, and / or any other representation that can encode the underlying meaning of the data. When inferred, the corresponding word-gram set may be utilized to generate opaque versions of client device data and 3P response data for secure exchange of the opaque versions via secure data exchange system 180. Further, because the corresponding word-gram set is jointly agreed upon by automated assistant 115 and “Example Music Streaming Platform”, various ML models (e.g., ranking ML models, suggestion ML models, recommendation ML models, sentiment ML models, tendency ML models, NLU models, and / or other ML models) employed by automated assistant 115 and “Example Music Streaming Platform” are able to handle the opaque versions of the client device data and 3P response data as input in addition to signals generated by the corresponding parties (e.g., corresponding request / response data, corresponding group data, corresponding user profile data, etc.). However, each word-gram included in the corresponding word-gram set may be semantically uninterpretable by a human reviewer when examining the word-gram, and therefore, the opaque versions of the client device data and 3P response data may be used by various ML models employed by automated assistant 115 and “Example Music Streaming Platform”, but in a manner that maintains the security of the underlying data.

[0051] In various implementations, the ML model training engine 164 may generate multiple training instances based on the corresponding client device data set, the corresponding 3P response data set, and the corresponding word-gram set. For example, the ML model training engine may generate a given automated assistant application training instance based on a given corresponding client device data set and a corresponding word-gram set, and cause the given automated assistant application training instance to be stored in the automated assistant application training instance database 164A for subsequent use in training the first encoder-decoder model 220. The given automated assistant application training instance may include a training instance input corresponding to the given corresponding client device data set, and a training instance output corresponding to one or more word-grams from the corresponding word-gram set. By training the first encoder-decoder model 220 based on the given automated assistant application training instance (and other automated assistant application training instances), the first encoder-decoder model 220 is trained to generate an opaque version of the given corresponding client device data set included in the training instance input (and an opaque version of the corresponding client device data set included in the corresponding training instance input of the other automated assistant application training instances).

[0052] For example, the ML model training engine 164 may apply a given corresponding client device data set (e.g., the training instance input of the given automated assistant application training instance described above) across the encoder portion 165A1 of the first encoder-decoder model 220 as input to generate an intermediate representation 204 thereof, such as a context vector, etc. Further, the ML model training engine 164 may apply the intermediate representation 204 across the decoder portion 165A2 of the first encoder-decoder model 220 as input to generate a predicted output. In addition, the ML model training engine 164 may compare the predicted output with one or more tokens from the corresponding token set (e.g., the training instance output of the given automated assistant application training instance described above) to generate one or more losses. In addition, the ML model training engine 164 may update the first encoder-decoder model 220 based on the one or more losses (e.g., via backpropagation). The ML model training engine 164 may continue to train the first encoder-decoder model 220 in this manner based on multiple additional automated assistant application training instances stored in the automated assistant application training instance database 164A until one or more training conditions are met. The one or more training conditions may include, for example, that the first encoder-decoder model 220 is trained based on a threshold amount of automated assistant application training instances, that the first encoder-decoder model 220 is trained for a threshold duration, that the first encoder-decoder model 220 satisfies one or more performance criteria (e.g., in terms of recall and / or precision), and / or other training criteria.

[0053] In various implementations, the ML model training engine 164 may generate multiple training instances based on the corresponding 3P response data set, the corresponding 3P response data set, and the corresponding word-gram set. For example, the ML model training engine 164 may generate a given automated assistant application training instance based on a given corresponding client device data set and a corresponding word-gram set, and cause the given automated assistant application training instance to be stored in the automated assistant application training instance database 164A for subsequent use in training the first encoder-decoder model 220. The given automated assistant application training instance may include a training instance input corresponding to the given corresponding client device data set, and a training instance output corresponding to one or more word-grams from the corresponding word-gram set. By training the first encoder-decoder model 220 based on the given automated assistant application training instance (and other automated assistant application training instances), the first encoder-decoder model 220 is trained to generate an opaque version of the given corresponding client device data set included in the training instance input (and an opaque version of the corresponding client device data set included in the corresponding training instance input of the other automated assistant application training instances).

[0054] For example, the ML model training engine 164 may apply a given corresponding client device data set (e.g., the training instance input of the given automated assistant application training instance described above) across the encoder portion 165A1 of the first encoder-decoder model 220 as input to generate an intermediate representation 204 thereof, such as a context vector, etc. Further, the ML model training engine 164 may apply the intermediate representation 204 across the decoder portion 165A2 of the first encoder-decoder model 220 as input to generate a predicted output. In addition, the ML model training engine 164 may compare the predicted output with one or more tokens from the corresponding token set (e.g., the training instance output of the given automated assistant application training instance described above) to generate one or more losses. In addition, the ML model training engine 164 may update the first encoder-decoder model 220 based on the one or more losses (e.g., via backpropagation). The ML model training engine 164 may continue to train the first encoder-decoder model 220 in this manner based on multiple additional automated assistant application training instances stored in the automated assistant application training instance database 164A until one or more training conditions are met. The one or more training conditions may include, for example, that the first encoder-decoder model 220 is trained based on a threshold amount of automated assistant application training instances, that the first encoder-decoder model 220 is trained for a threshold duration, that the first encoder-decoder model 220 satisfies one or more performance criteria (e.g., in terms of recall and / or precision), and / or other training criteria.

[0055] Moreover, for example, the ML model training engine 164 may generate a given 3P application training instance based on a given corresponding 3P response data set and a corresponding word-gram set, and cause the given 3P application training instance to be stored in a given 3P application training instance database 164B for subsequent use in training the second encoder-decoder model 230. It is noteworthy that the given corresponding 3P response data set used to generate the given 3P application training instance in this example may correspond to the 3P response data of the given corresponding client device data set used to generate the given automated assistant application training instance in the above example. The given 3P application training instance may include a training instance input corresponding to the given corresponding 3P response data set, and a training instance output corresponding to one or more additional word-grams from the corresponding word-gram set. Therefore, the given automated assistant application training instance in the above example and the given 3P application training instance in this example may be regarded as a pair of training instances that enable the ML model training engine 164 to ensure that the first encoder-decoder model 220 and the second encoder-decoder model 230 are trained to learn the relationship between the data and the opaque version of the data during joint training. By training the second encoder-decoder model 230 based on a given 3P application training instance (and other 3P application training instances), the second encoder-decoder model 230 is trained to generate an opaque version of a given corresponding 3P response data set included in the training instance input (and an opaque version of the corresponding 3P response data set included in the corresponding training instance input of other 3P application training instances).

[0056] For example, the ML model training engine 164 may apply a given corresponding 3P response data set (e.g., the training instance input of the given 3P application training instance described above) across the encoder portion 165A3 of the second encoder-decoder model 230 as input to generate its intermediate representation 210, such as a context vector, etc. Further, the ML model training engine 164 may apply the intermediate representation 210 across the decoder portion 165A4 of the second encoder-decoder model 230 as input to generate a predicted output. In addition, the ML model training engine 164 may compare the predicted output with one or more additional word-grams from the corresponding word-gram set (e.g., the training instance output of the given 3P application training instance described above) to generate one or more losses. In addition, the ML model training engine 164 may update the second encoder-decoder model 230 based on the one or more losses (e.g., via backpropagation). The ML model training engine 164 may continue to train the second encoder-decoder model 230 in this manner based on multiple additional 3P application training instances stored in the given 3P application training instance database 164B until one or more training conditions are met. One or more training conditions may include, for example, the second encoder-decoder model 230 is trained based on a threshold amount of 3P application training instances, the second encoder-decoder model 230 is trained for a threshold duration, the second encoder-decoder model 230 satisfies one or more performance criteria (e.g., in terms of recall and / or accuracy), and / or other training criteria.

[0057] By training first encoder-decoder model 220 and second encoder-decoder model 230 in this manner, ML model engine 165 of secure data exchange system 180 may utilize trained first encoder-decoder model 220 to facilitate secure transmission of data from automated assistant 115 to “Example Music Streaming Platform.” Further, ML model engine 165 of secure data exchange system 180 may utilize trained second encoder-decoder model 230 to facilitate secure transmission of data from “Example Music Streaming Platform” to automated assistant 115. Thus, secure data exchange system 180 enables secure exchange of data between various disparate parties, thereby reducing the amount of instances where data may be compromised and protecting the privacy of data associated with a user that may be transmitted using secure data exchange system 180.

[0058] For example, again assuming that a given user of a given client device previously provided the spoken utterance "Assistant, play some music using Example Music Streaming Platform". In this example, automated assistant 115 may cause ASR engines 120A1 and / or 120A2 to process the audio data capturing the spoken utterance to generate ASR output (e.g., recognized text corresponding to the spoken utterance). Further, automated assistant 115 may cause NLU engines 130A1 and / or 130A2 to process the ASR output to generate NLU output (e.g., an intent [intent = "play music"] with corresponding slot values ​​[platform = "Example StreamingPlatform"], [artist = "null"], and / or [song = "null"], where "null" indicates that the given user did not provide any slot values ​​for the corresponding artist and song parameters). The automated assistant may utilize the ASR output and / or the NLU output as user request data 201. In addition, the automated assistant 115 may obtain user group data 202 and user profile data 203. In this example, the user group data 202 may include an indication of a given group to which a given user is assigned from among a plurality of disparate groups, such as a given group indicating that the given user is interested in rock music. Further, in this example, the user profile data 203 may indicate one or more favorite rock artists of the given user.

[0059] In this example, automated assistant 115 may cause user request data 201, user group data 202, and / or user profile data 203 to be transmitted as client device data to an independent server via one or more networks (e.g., network 199). Further, in response to receiving the client data, the independent server may process the client device data using the trained first encoder-decoder model 220 to generate opaque client device data 205. In addition, the independent server may transmit the opaque client device data 205 to the "Example Music Streaming Platform" via one or more networks (e.g., network 199). In response to receiving the opaque client device data 205, the "Example Music Streaming Platform" may process the opaque client data 205 using one or more 3P models 206 to generate 3P response data 207. In this example, the one or more 3P models may include, for example, a ranking model that includes a ranking of artists and / or songs that should be played in response to the spoken utterance and based on at least processing the opaque client device data 205. Further, opaque client device data 205 may additionally or alternatively map user group data 202 to 3P group data 208. In other words, user group data 202 encoded by opaque client device data 205 may map a given group data to which a user is assigned to a given 3P group from among a plurality of distinct 3P groups, which given 3P group may be dynamically adapted based on user request data 201 and user profile data 203 encoded in opaque client device data 205. For example, a given user may be assigned to a given group represented by user group data 202, but 3P group data 208 representing the given 3P group to which the user is assigned may be different based on changes in user request data 201 and / or user profile data 203 encoded in opaque client device data 205. Furthermore, opaque client device data 205 may additionally or alternatively map user profile data 203 managed by automated assistant 115 to 3P profile data 209 managed by “Example Music Streaming Platform”.

[0060] Further, in this example, "Example Music Streaming Platform" may cause 3P response data 207, 3P group data 208, and / or 3P profile data 209 to be transmitted as 3P response data to an independent server via one or more networks (e.g., network 199). Further, in response to receiving the 3P response data, the independent server may process the 3P response data using the trained second encoder-decoder model 230 to generate opaque 3P response data 211. In addition, the independent server may transmit the opaque 3P response data 211 to an automated assistant 115 that is executed at least in part at a given client device of a given user via one or more networks (e.g., network 199). In response to receiving the opaque 3P response data 211, automated assistant 115 may determine content 212 to be provided for presentation to a given user via a given client device. In this example, content 212 may include a song to be provided for playback in response to receiving the spoken utterance "Assistant, play some music using Example Music Streaming Platform".

[0061] In various implementations, the feedback engine 166 of the secure data exchange system 180 may monitor a given user's utilization of the content 212 to generate a feedback signal that is used to generate additional training instances that can be used to update the first encoder-decoder model 220 and / or the second encoder-decoder model 230. For example, assuming that the content 212 includes a song to be provided for playback in response to receiving the spoken utterance "Assistant, play some music using Example Music Streaming Platform", and assuming that the given user listens to the entire duration of the song, the feedback engine 166 may use this as a positive feedback signal to generate additional automated assistant application training instances and / or additional 3P application training instances that positively reinforce the first encoder-decoder model 220 and / or the second encoder-decoder model 230. In contrast, assuming that the given user only listens to the first 30 seconds of the song, the feedback engine 166 may use this as a negative feedback signal to generate additional automated assistant application training instances and / or additional 3P application training instances that negatively reinforce the first encoder-decoder model 220 and / or the second encoder-decoder model 230.

[0062] although Figure 2 The example is described with respect to automated assistant 115 and a given 3P application (e.g., "Example Music Streaming Platform"), but it should be understood that this is for the purpose of example and is not meant to be limiting. Figure 2 The described techniques may be employed with respect to automated assistant 115 and any other 3P application. In these examples, various additional encoder-decoder models may be trained in the same or similar manner but with respect to other data generated by the parties and / or with various corresponding word-gram sets. In these examples, the various corresponding word-gram sets may include the same corresponding word-gram set and / or different corresponding word-gram sets. Also, for example, various additional encoder-decoder models may be trained in the same or similar manner but with respect to disparate 3P applications. In these examples, a standalone server may be provided as a service to enable secure transfer of data between disparate 3P applications without using automated assistant 115 in any way.

[0063] Furthermore, although Figure 2 The examples are described with respect to training and / or utilizing first encoder-decoder model 220 and second encoder-decoder model 230, but it should be understood that this is for illustrative purposes and is not meant to be limiting. For example, the same and / or similar techniques may be utilized to train a single encoder-decoder model to facilitate secure back-and-forth transmission of data between automated assistant 115 and “Example Music Streaming Platform,” as corresponding sets of tokens are mutually agreed upon by both automated assistant 115 and “Example Music Streaming Platform” and are used to generate an opaque version of the data. For example, automated assistant application training instances and / or 3P application training instances may be generated in the same or similar manner as described above, but may be used to train a single encoder-decoder model specific to both the automated assistant application and the 3P application, instead of training multiple distinct encoder-decoders specific to the respective applications.

[0064] In addition, despite Figure 2The example of is described with respect to actively utilizing a separate server at inference time, but this is for illustrative purposes and is not meant to be limiting. For example, the trained encoder portion 165A1 and the trained decoder portion 165A4 of the first encoder-decoder model 220 may be provided to the automated assistant 115, and the trained decoder portion 165A2 and the trained encoder portion 165A3 of the first encoder-decoder model 220 may be provided to the "Example MusicStreaming Platform". In this example, an intermediate representation of the data may be transmitted between the parties based on the disparate encoder and decoder portions of the first encoder-decoder model 220 and the second encoder-decoder model 230 being locally implemented by the disparate parties. Moreover, for example, the trained first encoder-decoder model 220 and / or the trained second encoder-decoder model 230 may be utilized to generate a mapping between data and an opaque version of the data, such that the trained first encoder-decoder model 220 and / or the trained second encoder-decoder model 230 need not be actively utilized at inference time.

[0065] Now turn to Figure 3 , depicts a flow chart illustrating an example method 300 for facilitating secure exchange of data from the perspective of a client device. For convenience, the operations of method 300 are described with reference to a system performing the operations. The system of method 300 includes a computing device (e.g., Figure 1 The client device 110, Figure 6 One or more processors, memories, and / or other components of the computing device 610 and / or other client devices of the method 300. In addition, although the operations of the method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0066] At block 352, the system obtains, via an automated assistant application executed at least in part at a client device, client device data of a user to be transmitted to a client device of a given third-party application. The client device data may include, for example, user request data (e.g., based on spoken utterances or other user input, based on an identified need for suggestions or recommendations, etc.), user group data, and / or user profile data.

[0067] At block 354, the system transmits the client device data to the independent server. Transmitting the client device data from the client device to the independent server may cause the independent server to perform various operations, such as those indicated by the operations of blocks 354A through 354E. For example, at block 354A, the system causes the independent server to process the client device data using a previously trained automated assistant application encoder-decoder model specific to the automated assistant application to generate opaque client device data (e.g., as described with respect to FIG. 1 ). Figure 2 and Figure 5 The training is performed as described). At box 354B, the system causes the independent server to transmit the opaque client device data to the given third-party application without transmitting any client device data in the client device data to the given third-party application. At box 354C, the system causes the independent server to receive third-party response data from the given third-party application. In other words, the independent server transforms the client device data into an opaque version of the client device data (e.g., a tokenized version of the client data) that can be used by the given third-party application without endangering the security of the underlying client device data represented by the opaque client device data. The given third-party application uses the opaque client device data and optionally other given third-party application data (e.g., given third-party application group data, given third-party application user profile data) to generate third-party response data in response to the client device data. The given third-party application can transmit the third-party response data back to the independent server.

[0068] Further, at block 354C, the system causes the independent server to process the third-party response data using a previously trained given third-party application encoder-decoder model specific to the given third-party application to generate opaque third-party response data (e.g., as described with respect to Figure 2 and 5 The training is performed as described). At box 354E, the system causes the independent server to transmit the opaque third-party response data to the client device without transmitting any third-party response data in the third-party response data to the client device. At box 356, the system receives the opaque third-party response data from the independent server. At box 358, the system causes the opaque third-party response data to be used to generate content that will be provided for presentation to the user of the client device. In other words, the independent server transforms the third-party response data into an opaque version of the third-party response data (e.g., a tokenized version of the third-party response data) that can be used by the automated assistant without compromising the security of the underlying third-party response data represented by the opaque third-party response data. The system causes the automated assistant to utilize the opaque third-party response data to generate content that will be provided for presentation to the user of the client device. At box 360, the system causes the content to be provided for presentation to the user of the client device. The system may provide the content to be presented to the user of the client device visually and / or audibly. In response to obtaining additional client device data, the system returns to box 352 to perform Figure 3 Additional iterations of method 300 are performed.

[0069] Now turn to Figure 4, depicts a flowchart illustrating an example method 400 for facilitating secure exchange of data from the perspective of an independent server. For convenience, the operations of method 400 are described with reference to a system performing the operations. The system of method 400 includes a computing device (e.g., Figure 1 Secure data exchange system 180, Figure 6 The method 400 may be implemented by one or more processors, memories, and / or other components of a computing device 610, one or more servers, and / or other computing devices. In addition, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0070] At block 452, the system receives client device data from a user's client device and to be transmitted to a given third-party application via an automated assistant application executed at least in part at the client device. At block 454, the system processes the client device data using a previously trained automated assistant application encoder-decoder model specific to the automated assistant application to generate opaque client device data (e.g., as described with respect to Figure 2 and Figure 5 The system transmits the opaque client device data to the given third-party application without transmitting any of the client device data to the given third-party application. Transmitting the opaque client device data from the independent server to the given third-party application may cause the given third-party application to perform various operations, such as those indicated by the operations of blocks 456A and 456B. For example, at block 456A, the system causes the given third-party application to generate third-party response data based on processing at least the opaque client device data. At block 456B, the system causes the given third-party application to transmit the third-party response data to the independent server.

[0071] In other words, the system can transform the client device data into an opaque version of the client device data before transmitting the client device data to the given third-party application to improve the security of the client device data, so that the given third-party application cannot access or observe the underlying client device data. Although the given third-party application cannot access or observe the underlying client device data, various machine learning models utilized by the given third-party application can still utilize the opaque client device data to generate third-party response data based at least on the opaque client device data. The given third-party application can then transmit the third-party response data back to the system implemented by the independent server.

[0072] At block 458, the system processes the third-party response data using a previously trained given third-party application encoder-decoder model specific to the given third-party application to generate opaque third-party response data (e.g., as described with respect to Figure 2 and Figure 5 The system transmits the opaque third-party response data to the client device without transmitting any of the third-party response data in the third-party response data. Transmitting the opaque third-party response data from the independent server to the client device may cause the client device to perform various operations, such as those indicated by the operations of blocks 460A and 460B. For example, at block 460A, the system causes the client device to cause the opaque third-party response data to be used to generate content to be provided for presentation to a user of the client device. At block 460B, the system causes the content to be provided for presentation to a user of the client device. In response to receiving the additional client device data, the system returns to block 452 to perform Figure 4 Additional iterations of method 400 are performed.

[0073] In other words, the system can transform the third-party response data into an opaque version of the third-party response data before transmitting the third-party data to the client device to improve the security of the third-party response data, so that the automated assistant executed at least in part at the client device (and other components of the client device) cannot access or observe the underlying third-party response data. Although the automated assistant (and other components of the client device) cannot access or observe the underlying third-party response data, various machine learning models utilized by the automated assistant (and other components of the client device) can still utilize the opaque third-party response data to generate content based at least on the opaque third-party response data.

[0074] It is worth noting that Figure 3 Method 300 and Figure 4 The client device data and the third-party response data described in method 400 may correspond to privacy-sensitive data. In various implementations, non-privacy-sensitive data may be transmitted directly between the system and a given third-party application without utilizing a separate server. For example, assuming that a given third-party application communicates with the client device data from the above description of Figure 2 In the example, the music preferences of a given user maintained by the system and / or a given third-party application may be considered privacy-sensitive data. Thus, in response to determining that the music preferences of the given user correspond to privacy-sensitive data, the system may utilize an independent server to facilitate the secure exchange of that data. However, the library of all artists or songs maintained by the "Example Music Streaming Platform" may be non-privacy-sensitive, and indications of all artists or songs may be transmitted directly from the "Example Music Streaming Platform" back to the system without utilizing an independent server.

[0075] Now turn to Figure 5 , depicts a flowchart illustrating an example method 500 for training various encoder-decoder models at an independent server, which may then be used to facilitate secure exchange of data. For convenience, the operations of method 500 are described with reference to a system performing the operations. The system of method 500 includes a computing device (e.g., Figure 1 Secure data exchange system 180, Figure 6 The method 500 may be implemented by one or more processors, memories, and / or other components of a computing device 610, one or more servers, and / or other computing devices. In addition, although the operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0076] At block 552, the system obtains a set of tokens corresponding to the vocabulary that will be used to generate an opaque version of the data. The set of tokens corresponding to the vocabulary may include tokens corresponding to words, phrases, symbols, numbers, and / or any other representations that can encode the underlying meaning of the data. The set of tokens may be mutually agreed upon by at least the automated assistant and the given third-party application to facilitate secure transmission of data between the automated assistant and the given third-party application in a bidirectional manner.

[0077] At box 554, the system receives corresponding client device data sets from multiple client devices via corresponding automated assistant applications executed at least in part at corresponding client devices in the multiple client devices. At box 556, the system receives corresponding third-party response data sets from a given third-party application, each third-party response data set in the corresponding third-party response data set being generated by the given third-party application based on one or more client device data sets in the corresponding client device data set. The corresponding third-party response data set may include data previously generated by the given third-party application in response to processing a corresponding client device data set in the corresponding client device data set. It is worth noting that before receiving the corresponding client device data set and the corresponding third-party response data set, the system may employ various privacy techniques to protect the security and / or privacy of the underlying data included in the corresponding client device data set and the corresponding third-party response data set (e.g., differential privacy, federated learning, etc.).

[0078] At block 558A, the system generates a plurality of automated assistant application training instances based on (1) the word-gram set and (2) the corresponding client device data set. At block 560A, the system causes an automated assistant application encoder-decoder model specific to the automated assistant application to be trained based on the plurality of automated assistant application training instances. Figure 2) describes in more detail generating multiple automated assistant application training instances and training an automated assistant application encoder-decoder model.

[0079] At block 558B, the system generates a plurality of given third-party application training instances based on (1) the word-gram set and (3) the corresponding third-party response data set. At block 560B, the system causes a given third-party application encoder-decoder model specific to the given third-party application to be trained based on the plurality of given third-party training instances. Figure 2 ) describes in more detail generating multiple third-party application training instances and training a third-party application encoder-decoder model.

[0080] At block 562, the system causes the independent server to utilize the trained automated assistant application encoder-decoder model and the trained given third-party application encoder-decoder model to facilitate secure exchange of data between the automated assistant application and the given third-party application (e.g., as described with respect to Figure 2 process flow, Figure 3 Method 300, Figure 4 400 and / or in any other manner described herein). The system returns to block 552 to execute Figure 5 Additional iterations of method 500 are performed.

[0081] In some implementations, when executing Figure 5 During additional iterations of method 500, the system can continue to train an automated assistant application encoder-decoder model specific to the automated assistant application and / or a given third-party application encoder-decoder model specific to a given third-party application. The system can continue to train these encoder-decoder models until one or more training conditions (e.g., such as regarding Figure 2 In an additional or alternative implementation, when executing Figure 5 During additional iterations of method 500, the system may initiate training of another automated assistant application encoder-decoder model that is specific to the automated assistant application but for a distinct third-party application and / or an additional given third-party application encoder-decoder model that is specific to the additional given third-party application. Figure 5 When additional iterations of method 500 are performed, the system can initiate training of other encoder-decoder modes for various disparate third-party applications to enable secure transmission between these disparate third parties. In other words, the techniques described herein can be used to provide secure exchange of data as a service, and the independent server can be a true independent third party that does not develop and / or maintain any assistants, agents, applications, or systems that actively utilize the independent server to facilitate secure exchange of data.

[0082] Now turn to Figure 6 , depicts a block diagram of an example computing device 610 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automated assistant component, and / or other components can include one or more components of the example computing device 610.

[0083] The computing device 610 typically includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 (which includes, for example, a memory subsystem 625 and a file storage subsystem 626), a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow for user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0084] The user interface input device 622 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen integrated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 610 or onto a communication network.

[0085] The user interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computing device 610 to a user or another machine or computing device.

[0086] The storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include selected aspects of the methods disclosed herein and data for implementing Figure 1 and Figure 2 The logic of the various components depicted.

[0087] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored by the file storage subsystem 626 in the storage subsystem 624, or in other machines accessible to the processor 614.

[0088] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem 612 may use multiple busses.

[0089] The computing device 610 may be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the Figure 6 The description of computing device 610 depicted in FIG. 6 is intended only as a specific example for illustrating some embodiments. Many other configurations of computing device 610 are possible, and these configurations are similar to those of FIG. Figure 6 The computing devices depicted in FIG. 1 may have more or fewer components than those depicted in FIG.

[0090] In situations where the systems described herein collect or otherwise monitor personal information about a user or may make use of the personal information and / or monitored information, the user may be given the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. Furthermore, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or, in the case of geographic location information, the user's geographic location may be generalized (such as to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, the user may control how information about the user is collected and / or used.

[0091] In some implementations, a method implemented by one or more processors is provided, the method comprising: obtaining, at a client device of the user via an automated assistant application executed at least in part at the client device, client device data of the user to be transmitted to a given third-party application; and transmitting the client device data from the client device to an independent server. Transmitting the client device data to the independent server causes the independent server to: process the client device data using a previously trained automated assistant application encoder-decoder model specific to the automated assistant application to generate opaque client device data; and transmit the opaque client device data to the given third-party application without transmitting any of the client device data to the given third-party application. Further, transmitting the opaque client device data to the given third-party application causes the given third-party application to: generate third-party response data based on processing at least the opaque client device data; and transmit the third-party response data to the independent server. Furthermore, transmitting the response data to the independent server causes the independent server to: process the third-party request data using a previously trained given third-party application encoder-decoder model specific to the given third-party application to generate opaque third-party response data; and transmit the opaque third-party response data to the client device without transmitting any of the third-party response data to the client device. The method further includes: receiving, at the client device, opaque third-party response data from the independent server; causing the opaque third-party response data to be used to generate content to be provided for presentation to a user of the client device; and causing the content to be provided for presentation to the user of the client device.

[0092] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0093] In some implementations, the method may further include: receiving user input including a request from a user via a client device, the request being for an automated assistant application executed at least in part at the client device; and generating request data based on processing the user input including the request, the request data to be transmitted to an independent server to obtain the opaque third-party response data. In some versions of these implementations, the client device data may include the request data.

[0094] In some implementations, the method may further include: identifying, via an automated assistant application executed at least in part at a client device, a need for a suggestion to be provided for presentation to a user; and generating suggestion data based on the need for the suggestion to be provided for presentation to the user, the suggestion data to be transmitted to an independent server to obtain the opaque third-party response data. In some versions of these implementations, the client device data may include the suggestion data.

[0095] In some implementations, the opaque client device data can include one or more word-grams from a set of word-grams corresponding to a vocabulary, the one or more word-grams being generated using a previously trained automated assistant application encoder-decoder model.

[0096] In some versions of these implementations, generating the third party response data based on processing the at least opaque client device data may include processing the at least opaque client device data using a given third party ranking model to generate a given third party ranking as the third party response data.

[0097] In additional or alternative versions of these implementations, generating the third party response data based on processing the at least opaque client device data may include processing the at least opaque client device data using a given third party suggestion model to generate a given third party suggestion as the third party response data.

[0098] In additional or alternative versions of these implementations, the opaque third-party response data may include one or more additional word-grams from the set of word-grams corresponding to the vocabulary, the one or more additional word-grams generated using a previously trained encoder-decoder model for the given third-party application.

[0099] In some implementations, the client device data may be privacy-sensitive data, and transmitting the client device data from the client device to the independent server may be in response to determining that the client device data is privacy-sensitive data. In some versions of these implementations, the method may further include transmitting additional client device data that is not privacy-sensitive directly from the client device to the given third-party application without transmitting the additional client device data to the independent server. In some additional or alternative versions of these implementations, the method may further include receiving additional third-party response data that is not privacy-sensitive directly from the given third-party application at the client device.

[0100] In some implementations, the method may further include: generating a feedback signal based on user input received in response to causing content to be provided for presentation to a user of the client device, the feedback signal to be used to update a previously trained automated assistant application encoder-decoder model and / or a previously trained given third-party application encoder-decoder model; and transmitting the feedback signal from the client device to an independent server. In some versions of these implementations, transmitting the feedback signal to the independent server may further cause the independent server to: update the previously trained automated assistant application encoder-decoder model and / or the previously trained given third-party application encoder-decoder model based on the feedback signal.

[0101] In some implementations, a previously trained automated assistant application encoder-decoder model can be specific to the automated assistant application for a given third-party application to facilitate the secure exchange of data between the automated assistant application and the given third-party application, and a previously trained given third-party application encoder-decoder model can be specific to the automated assistant application for a given third-party application to facilitate the secure exchange of data between the automated assistant application and the given third-party application.

[0102] In some implementations, a method implemented by one or more processors is provided, the method comprising: receiving, at an independent server via an automated assistant application executed at least in part at a client device, client device data from a user's client device and to be transmitted to a given third-party application; processing the client device data using a previously trained automated assistant application encoder-decoder model specific to the automated assistant application to generate opaque client device data; and transmitting the opaque client device data from the independent server to the given third-party application without transmitting any of the client device data to the given third-party application. Transmitting the opaque client device data to the given third-party application causes the given third-party application to: generate third-party response data based on processing at least the opaque client device data; and transmit the third-party response data to the independent server. The method further comprises: processing the third-party response data using a previously trained given third-party application encoder-decoder model specific to the given third-party application to generate opaque third-party response data; and transmitting the opaque third-party response data from the independent server to the client device without transmitting the third-party response data to the client device. Transmitting the opaque third-party response data to the client device causes the client device to: cause the opaque third-party response data to be used to generate content to be provided for presentation to a user of the client device; and cause the content to be provided for presentation to the user of the client device.

[0103] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0104] In some implementations, the method may further include: receiving, at an independent server via an automated assistant application executed at least in part at a client device, additional client device data from a user's client device and to be transmitted to an additional given third-party application; processing the additional client device data using an additional previously trained automated assistant application encoder-decoder model specific to the automated assistant application to generate additional opaque client device data; and transmitting the additional opaque client device data from the independent server to the additional given third-party application without transmitting any of the additional client device data to the additional given third-party application. Transmitting the additional opaque client device data to the additional given third-party application causes the additional given third-party application to: generate additional third-party response data based on processing at least the additional opaque client device data; and transmit the additional third-party response data to the independent server. The method may further include: processing the additional third-party response data using an additional previously trained additional given third-party application encoder-decoder model specific to the additional given third-party application to generate additional opaque third-party response data; and transmitting the additional opaque third-party response data from the independent server to the client device without transmitting the additional third-party response data to the client device. Transmitting the additional opaque third-party response data to the client device causes the client device to: cause the additional opaque third-party response data to be used to generate additional content to be provided for presentation to a user of the client device; and cause the additional content to be provided for presentation to the user of the client device.

[0105] In some versions of these implementations, a previously trained automated assistant application encoder-decoder model may be automated assistant application-specific for a given third-party application to facilitate secure exchange of data between the automated assistant application and the given third-party application, a previously trained additional automated assistant application encoder-decoder model may be automated assistant application-specific for an additional given third-party application to facilitate secure exchange of data between the automated assistant application and the additional given third-party application, and a previously trained additional automated assistant application encoder-decoder model may be different from a previously trained automated assistant application encoder-decoder model.

[0106] In some further versions of these implementations, a previously trained given third-party application encoder-decoder model may be specific to the given third-party application for the automated assistant application to facilitate secure exchange of data between the automated assistant application and the given third-party application, a previously trained additional given third-party application encoder-decoder model may be specific to the additional given third-party application for the automated assistant application to facilitate secure exchange of data between the automated assistant application and the additional given third-party application, and a previously trained given third-party application encoder-decoder model may be different from a previously trained additional given third-party application encoder-decoder model.

[0107] In some implementations, a method implemented by one or more processors is provided, the method comprising: obtaining, at an independent server, a set of word elements corresponding to a vocabulary that will be used to generate an opaque version of data; receiving, at the independent server, corresponding client device data sets from a plurality of client devices via corresponding automated assistant applications that are at least partially executed at corresponding client devices in a plurality of client devices; receiving, at the independent server, corresponding response data sets from a given third-party application, each response data set in the corresponding response data sets being generated by the given third-party application based on one or more client device data sets in the corresponding client device data set; generating a plurality of automated assistant application training instances based on (i) the set of word elements and (ii) the corresponding client device data sets; causing an automated assistant application encoder-decoder specific to the automated assistant application to decode the response data sets; The invention relates to a method for generating a third-party application encoder-decoder model based on multiple automated assistant application training instances; generating multiple third-party application training instances based on (i) a word-gram set and (iii) a corresponding response data set; causing a given third-party application encoder-decoder model specific to a given third-party application to be trained based on the multiple given third-party application training instances; and after causing the automated assistant application encoder-decoder model to be trained based on the multiple automated assistant application training instances, and after causing the given third-party application encoder-decoder model to be trained based on the multiple given third-party application training instances: causing an independent server to utilize the trained automated assistant application encoder-decoder model and the trained given third-party application encoder-decoder model to facilitate secure exchange of data between a corresponding automated assistant application and a given third-party application executed at least in part at multiple client devices.

[0108] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0109] In some implementations, generating a given automated assistant application training instance among multiple automated assistant application training instances may include: identifying given client device data from a corresponding client device data set to be used as a training instance input for the given automated assistant application training instance; and identifying one or more word-grams from a word-gram set to be used as a training instance output for the given automated assistant application training instance.

[0110] In some versions of these implementations, generating a given third-party application training instance among multiple third-party application training instances may include: identifying given response data from a corresponding response data set to be used as a training instance input for the given third-party application training instance; and identifying one or more additional word-grams from a word-gram set to be used as training instance output for the given third-party application training instance.

[0111] In some implementations, the method may further include: causing differential privacy technology to be utilized by multiple client devices to cause corresponding client device data sets to be transmitted to an independent server in a privacy-sensitive manner; and causing differential privacy technology to be utilized by a given third-party application to cause corresponding response data sets to be transmitted to the independent server in a privacy-sensitive manner.

[0112] In some implementations, the method may further include: causing federated learning technology to be utilized by multiple client devices to cause corresponding client device data sets to be transmitted to an independent server in a privacy-sensitive manner; and causing federated learning technology to be utilized by a given third-party application to cause corresponding response data sets to be transmitted to an independent server in a privacy-sensitive manner.

[0113] In some implementations, a method implemented by one or more processors is provided, the method comprising: obtaining, at a client device of the user via an automated assistant application executed at least in part at the client device, client device data of the user to be transmitted to a given third-party application; and transmitting the client device data from the client device to an independent server. Transmitting the client device data to the independent server causes the independent server to: process the client device data using a previously trained encoder-decoder model specific to both the automated assistant application and the given third-party application to generate opaque client device data; and transmit the opaque client device data to the given third-party application without transmitting any of the client device data to the given third-party application. Further, transmitting the opaque client device data to the given third-party application causes the given third-party application to: generate third-party response data based on processing at least the opaque client device data; and transmit the third-party response data to the independent server. In addition, transmitting the response data to the independent server causes the independent server to: process the third-party request data using a previously trained encoder-decoder model specific to both the automated assistant application and the given third-party application to generate opaque third-party response data; and transmit the opaque third-party response data to the client device without transmitting any of the third-party response data to the client device. The method further includes: receiving, at the client device, opaque third-party response data from the independent server; causing the opaque third-party response data to be used to generate content to be provided for presentation to a user of the client device; and causing the content to be provided for presentation to the user of the client device.

[0114] In addition, some implementations include one or more processors (e.g., central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units (TPUs)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause performance of any of the above methods. Some implementations also include one or more non-transitory computer-readable storage media storing computer instructions that can be executed by one or more processors to perform any of the above methods. Some implementations also include a computer program product that includes instructions that can be executed by one or more processors to perform any of the above methods.

Claims

1. A method implemented by one or more processors, the method comprising: obtaining, at a client device of a user via an automated assistant application executed at least in part at the client device, client device data of the user to be transmitted to a given third-party application; transmitting the client device data from the client device to a separate server, wherein transmitting the client device data to the independent server causes the independent server to: processing the client device data using a previously trained automated assistant application encoder-decoder model specific to the automated assistant application to generate opaque client device data; and transmitting the opaque client device data to the given third-party application without transmitting any of the client device data to the given third-party application; wherein transmitting the opaque client device data to the given third-party application causes the given third-party application to: generating third party response data based on processing at least the opaque client device data; and transmitting the third-party response data to the independent server; and wherein transmitting the response data to the independent server causes the independent server to: processing the third-party request data using a previously trained given third-party application encoder-decoder model specific to the given third-party application to generate opaque third-party response data; and transmitting the opaque third-party response data to the client device without transmitting any third-party response data among the third-party response data to the client device; receiving, at the client device, the opaque third-party response data from the independent server; causing the opaque third-party response data to be used to generate content to be provided for presentation to the user of the client device; as well as The content is caused to be provided for presentation to the user of the client device.

2. The method of claim 1, further comprising: receiving, via the client device, from the user user input comprising a request for an automated assistant application executing at least in part at the client device; as well as Request data is generated based on processing the user input including the request, and the request data is transmitted to the independent server to obtain the opaque third-party response data.

3. The method of claim 2, wherein the client device data comprises the request data.

4. The method of claim 1, further comprising: identifying, via an automated assistant application executed at least in part at the client device, a need for suggestions to be provided for presentation to the user; as well as Suggestion data is generated based on the need for the suggestion to be provided for presentation to the user, and the suggestion data is transmitted to the independent server to obtain the opaque third-party response data. The method of claim 4 , wherein the client device data includes the suggestion data.

6. A method as claimed in any preceding claim, wherein the opaque client device data comprises one or more word-grams from a set of word-grams corresponding to a vocabulary, the one or more word-grams being generated using the previously trained automated assistant application encoder-decoder model.

7. The method of claim 6, wherein generating the third party response data based on processing at least the opaque client device data comprises: At least the opaque client device data is processed using a given third-party ranking model to generate a given third-party ranking as the third-party response data.

8. The method of claim 6, wherein generating the third party response data based on processing at least the opaque client device data comprises: At least the opaque client device data is processed using a given third-party suggestion model to generate a given third-party suggestion as the third-party response data.

9. The method of claim 6, wherein the opaque third-party response data includes one or more additional word-grams from the set of word-grams corresponding to the vocabulary, the one or more additional word-grams being generated using the previously trained given third-party application encoder-decoder model.

10. The method of any preceding claim, wherein the client device data is privacy-sensitive data, and wherein transmitting the client device data from the client device to the independent server is responsive to a determination that the client device data is privacy-sensitive data.

11. The method of claim 10, further comprising: Additional client device data that is not privacy sensitive is transmitted directly from the client device to the given third-party application without transmitting the additional client device data to the independent server.

12. The method of claim 11, further comprising: Non-privacy sensitive additional third-party response data is received directly from the given third-party application at the client device.

13. The method of any preceding claim, further comprising: generating a feedback signal based on user input received in response to causing the content to be provided for presentation to the user of the client device, the feedback signal to be used to update the previously trained automated assistant application encoder-decoder model and / or the previously trained given third-party application encoder-decoder model; as well as The feedback signal is transmitted from the client device to the independent server.

14. The method of claim 13, wherein transmitting the feedback signal to the independent server further causes the independent server to: The previously trained automated assistant application encoder-decoder model and / or the previously trained given third-party application encoder-decoder model are updated based on the feedback signal.

15. A method as claimed in any preceding claim, wherein the previously trained automated assistant application encoder-decoder model is specific to the given third-party application to facilitate the secure exchange of data between the automated assistant application and the given third-party application, and wherein the previously trained given third-party application encoder-decoder model is specific to the given third-party application to facilitate the secure exchange of data between the automated assistant application and the given third-party application.

16. A method implemented by one or more processors, the method comprising: receiving, at the standalone server via an automated assistant application executed at least in part at the client device, client device data from a client device of a user and to be transmitted to a given third-party application; processing the client device data using a previously trained automated assistant application encoder-decoder model specific to the automated assistant application to generate opaque client device data; transferring the opaque client device data from the independent server to the given third-party application without transferring any of the client device data to the given third-party application, wherein transmitting the opaque client device data to the given third-party application causes the given third-party application to: generating third party response data based on processing at least the opaque client device data; and transmitting the third-party response data to the independent server; processing the third-party response data using a previously trained given third-party application encoder-decoder model specific to the given third-party application to generate opaque third-party response data; as well as transmitting the opaque third-party response data from the independent server to the client device without transmitting the third-party response data to the client device, wherein transmitting the opaque third-party response data to the client device causes the client device to: causing the opaque third-party response data to be used to generate content to be provided for presentation to the user of the client device; and The content is caused to be provided for presentation to the user of the client device.

17. The method of claim 16, further comprising: receiving, at the standalone server via the automated assistant application executed at least in part at the client device, additional client device data from the client device of the user and to be transmitted to an additional given third-party application; processing the additional client device data using an additional previously trained automated assistant application encoder-decoder model specific to the automated assistant application to generate additional opaque client device data; transferring the additional opaque client device data from the independent server to the additional given third-party application without transferring any of the additional client device data to the additional given third-party application, wherein transmitting the additional opaque client device data to the additional given third-party application causes the additional given third-party application to: generating additional third party response data based on processing at least the additional opaque client device data; and transmitting the additional third-party response data to the independent server; processing the additional third-party response data using an additional previously trained additional given third-party application encoder-decoder model specific to the additional given third-party application to generate additional opaque third-party response data; as well as transmitting the additional opaque third-party response data from the independent server to the client device without transmitting the additional third-party response data to the client device, wherein transmitting the additional opaque third-party response data to the client device causes the client device to: causing the additional opaque third-party response data to be used to generate additional content to be provided for presentation to the user of the client device; and The additional content is caused to be provided for presentation to the user of the client device.

18. The method of claim 17, wherein the previously trained automated assistant application encoder-decoder model is specific to the automated assistant application for the given third-party application to facilitate secure exchange of data between the automated assistant application and the given third-party application, wherein the previously trained additional automated assistant application encoder-decoder model is specific to the automated assistant application for the additional given third-party application to facilitate the secure exchange of data between the automated assistant application and the additional given third-party application, and wherein the previously trained additional automated assistant application encoder-decoder model is different from the previously trained automated assistant application encoder-decoder model.

19. The method of claim 18, wherein the previously trained given third-party application encoder-decoder model is specific to the given third-party application for the automated assistant application to facilitate the secure exchange of data between the automated assistant application and the given third-party application, wherein the previously trained additional given third-party application encoder-decoder model is specific to the additional given third-party application for the automated assistant application to facilitate the secure exchange of data between the automated assistant application and the additional given third-party application, and wherein the previously trained given third-party application encoder-decoder model is different from the previously trained additional given third-party application encoder-decoder model.

20. A method implemented by one or more processors, the method comprising: obtaining, at a separate server, a set of word-grams corresponding to the vocabulary to be used to generate an opaque version of the data; receiving, at the standalone server, corresponding sets of client device data from a plurality of client devices via corresponding automated assistant applications executed at least in part at respective ones of the plurality of client devices; receiving, at the independent server, corresponding response data sets from a given third-party application, each of the corresponding response data sets being generated by the given third-party application based on one or more of the corresponding client device data sets; generating a plurality of automated assistant application training instances based on (i) the word-gram set and (ii) the corresponding client device data set; causing an automated assistant application encoder-decoder model specific to the automated assistant application to be trained based on the plurality of automated assistant application training instances; generating a plurality of third-party application training instances based on (i) the word-gram set and (iii) the corresponding response data set; causing a given third-party application encoder-decoder model specific to the given third-party application to be trained based on the plurality of given third-party application training instances; as well as After causing the automated assistant application encoder-decoder model to be trained based on the plurality of automated assistant application training instances, and after causing the given third-party application encoder-decoder model to be trained based on the plurality of given third-party application training instances: The independent server is caused to utilize the trained automated assistant application encoder-decoder model and the trained given third-party application encoder-decoder model to facilitate secure exchange of data between the corresponding automated assistant application and the given third-party application executed at least in part at the plurality of client devices.

21. The method of claim 20, wherein generating a given automated assistant application training instance in the plurality of automated assistant application training instances comprises: identifying, from the corresponding set of client device data, given client device data to be used as training instance input for the given automated assistant application training instance; as well as One or more word-grams are identified from the set of word-grams to be used as training instance output for the given automated assistant application training instance.

22. The method of claim 21, wherein generating a given third-party application training instance among the plurality of third-party application training instances comprises: identifying, from the set of corresponding response data, given response data to be used as training instance input for the given third-party application training instance; as well as One or more additional word-grams are identified from the set of word-grams to be used as training instance outputs for the given third-party application training instance.

23. The method of any one of claims 20 to 22, further comprising: causing differential privacy techniques to be utilized by the plurality of client devices to cause the corresponding sets of client device data to be transmitted to the independent server in a privacy-sensitive manner; as well as The differential privacy technique is caused to be utilized by the given third-party application to cause the corresponding set of response data to be transmitted to the independent server in the privacy-sensitive manner.

24. The method of any one of claims 20 to 22, further comprising: causing a federated learning technique to be utilized by the plurality of client devices to cause the corresponding sets of client device data to be transmitted to the independent server in a privacy-sensitive manner; as well as The federated learning technique is caused to be utilized by the given third-party application to cause the corresponding set of response data to be transmitted to the independent server in the privacy-sensitive manner.

25. A method implemented by one or more processors, the method comprising: obtaining, at a client device of a user via an automated assistant application executed at least in part at the client device, client device data of the user to be transmitted to a given third-party application; transmitting the client device data from the client device to a separate server, wherein transmitting the client device data to the independent server causes the independent server to: processing the client device data using a previously trained encoder-decoder model specific to both the automated assistant application and the given third-party application to generate opaque client device data; and transmitting the opaque client device data to the given third-party application without transmitting any of the client device data to the given third-party application; wherein transmitting the opaque client device data to the given third-party application causes the given third-party application to: generating third party response data based on processing at least the opaque client device data; and transmitting the third-party response data to the independent server; and wherein transmitting the response data to the independent server causes the independent server to: processing the third-party request data using the previously trained encoder-decoder model specific to both the automated assistant application and the given third-party application to generate opaque third-party response data; and transmitting the opaque third-party response data to the client device without transmitting any third-party response data among the third-party response data to the client device; receiving, at the client device, the opaque third-party response data from the independent server; causing the opaque third-party response data to be used to generate content to be provided for presentation to the user of the client device; as well as The content is caused to be provided for presentation to the user of the client device.

26. A computing device comprising: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the one or more processors to perform the operations of any one of claims 1 to 25.

27. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of any one of claims 1 to 25.

Images