Reducing metadata transmitted with automated assistant requests

Abstract

Implementations relate to maintaining metadata of a client device stored on a remote computing device to reduce overall network utilization and / or latency in response to automated assistant requests received at the client device. The metadata of the client device is identified and provided to the remote computing device along with a request requiring additional processing by the remote computing device. Current metadata is compared to previously provided metadata, which is provided to the remote computing device along with a previous request, and one or more operations are generated based on a difference between the current metadata and the previous metadata. The remote computing device performs one or more operations on the remote version of the metadata to generate a new remote version of the metadata for the client device. The remote computing device further processes the request with the new remote version of the metadata.

Description

BACKGROUND OF THE INVENTION

[0001] Humans can engage in human-computer conversations with interactive software applications referred to herein as "automation assistants" (also known as "digital agents", "chatbots", "interactive personal assistants", "intelligent personal assistants", "assistant applications", "conversational agents", etc.). For example, humans (who may be referred to as "users" when they interact with an automation assistant) can provide commands and / or requests to an automation assistant using oral natural language input (i.e., utterances) and / or by providing text (e.g., typed) natural language input. In some cases, the oral natural language input can be converted to text and then processed. The automation assistant responds to the request by providing a responsive user interface output, which can include audible and / or visual user interface output.

[0002] As described above, many automation assistants are configured to interact via oral utterances (such as a call indicator followed by an oral query). To protect user privacy and / or conserve resources, users typically must explicitly invoke the automation assistant, and then the automation assistant will fully process the oral utterance. The explicit invocation of the automation assistant typically occurs in response to receiving a certain user interface input at the client device. The client device includes an assistant interface that provides an interface to the user of the client device for interfacing with the automation assistant (e.g., receiving oral and / or typed input from the user and providing audible and / or graphical responses), and interfaces with one or more additional components that implement the automation assistant (e.g., a remote server device that processes the user input and generates an appropriate response).

[0003] Some user interface inputs that can be used to invoke an automation assistant via the client device include hardware and / or virtual buttons at the client device for invoking the automation assistant (e.g., tapping a hardware button, selecting a graphical interface element displayed by the client device). Many automation assistants can additionally or alternatively be invoked in response to one or more oral invocation phrases, which are also referred to as "hot words / phrases" or "trigger words / phrases". For example, oral invocation phrases such as "Hey Assistant", "OK Assistant", and / or "Assistant" can be spoken to invoke the automation assistant.

[0004] In some cases, an automated assistant can be executed at least in part on multiple devices. The computing power of client computing devices is typically limited and may therefore not have the resources to process requests from users. The request can be transmitted to a remote computing device with more computing resources than the client device for processing in order to determine the action to be performed based on the request. For some requests, additional information available on the client device can be provided along with the request to enable the remote computing device to process the request. Thus, in addition to the request, client device metadata can also be provided along with the request to facilitate processing of the request by the remote computing device. Summary of the Invention

[0005] Some implementations disclosed herein relate to maintaining a remote version of metadata by determining a difference between previous metadata and current metadata of a client device on which an automated assistant is being executed at least in part. Some implementations include receiving a request from a user, which can be processed by an automated assistant that is being executed in part on the client device and in part on a remote computing device. Some versions of those implementations include identifying, at the client device, locally stored previous metadata that reflects a remote version of the metadata maintained on the remote device, the metadata having been previously provided by the client device to the remote computing device in response to a previous request. Some versions of those versions further include determining, at the client device, a difference between the previous metadata (i.e., the metadata at the most recent request) and the current metadata of the client device, and providing, by the client device to the remote computing device, a set of operations to transform the remote version of the previous metadata into the remote version of the current metadata. The client device can store the previous metadata such that, upon receipt of a subsequent request, a new difference between the current metadata and the previous metadata of the client device can be determined, and a set of operations that can transform the previously transmitted metadata into the current metadata can be provided along with the subsequent request. Notably, the set of operations transmitted by the client device to the remote computing device in response to a request is smaller in size (i.e., fewer bytes) than the current metadata. Nevertheless, the remote computing device can effectively recreate the current metadata at the remote computing device using the set of operations and the remote version of the client device's metadata. In these and other ways, network resources are saved while still enabling the remote computing device to utilize the current metadata to determine actions in response to requests, such as actions that cannot be resolved without the metadata and / or actions that are more robust and / or accurate in view of the metadata.

[0006] For example, a user may use one or more invocation phrases followed by a request (such as, "Call Bob") to invoke an automated assistant. The automated assistant may be executed partially on the client device and partially on a remote computing device, such that at least some requests can be handled by the remote computing device. In the instance of the request "Call Bob", the remote automated assistant may handle the processing of the request, but may require the user's contacts (e.g., contact name and phone number) to process the request. When the user makes a request to place a call for the first time, the client automated assistant may provide all the contact information (i.e., initial metadata) locally stored on the client device along with the request. For example, the client automated assistant may provide an encrypted version of the contact information, which can be decrypted at the remote computing device. In response, the remote automated assistant may fulfill the request and further process the request by determining the phone number of "Bob" based on the provided metadata (i.e., the user's contacts). Additionally, the remote automated assistant may store the provided metadata for use when the user submits a new request to place a call. Similarly, the client automated assistant may store the provided metadata on the client device.

[0007] When the user submits a new request to place a call (such as, "Call Sue"), the client device automated assistant may first identify the metadata required by the remote automated assistant to process the request. However, before sending the metadata, the client automated assistant may compare the contacts with the stored metadata previously provided to the remote automated assistant. If the metadata has not changed (i.e., the stored metadata previously provided to the remote computing device is the same as the current metadata), then the request may be provided without any additional metadata. In response, the remote automated assistant may utilize the previously sent version of the metadata to process the request, such as determining the phone number of "Sue". Thus, in the case of unchanged metadata, no contact information needs to be sent each time the user submits a request to place a call, thereby saving computing resources.

[0008] In some implementations, the user may add, delete, and / or update metadata between requests that require metadata. For example, the user may submit a request to place a call and then add additional contacts before submitting a second request to place a call. In some implementations, the client automated assistant may identify the metadata (e.g., the user's contact list) and compare the metadata with the last metadata provided to the remote automated assistant. A set of operations along with the request may be provided to the remote computing device, which when executed by the remote automated assistant, may transform the previously sent metadata into the current metadata.

[0009] For example, a user can first issue a request of "Call Bob", and then provide metadata including the user's contacts to a remote automated assistant. Subsequently, the user can add a new contact "Sue" and the corresponding phone number. After the user has added the new contact, a request of "Call Sue" can be made. The client automated assistant can determine that the current metadata (i.e., the user's contacts) does not match the last metadata provided to the remote automated assistant. In response, the client automated assistant can determine the difference; in this example, the contact entry of "Sue" is added. Then, the client automated assistant can provide an operation (e.g., "Add 'Sue: 555-555-5555'") without providing any additional metadata such as the contact information of "Bob" and / or other contacts. Therefore, once the initial metadata is provided, subsequent requests that require the metadata can be fulfilled without providing all the metadata along with the request. By reducing the need to send all the metadata with each request, computing and / or network resources can be saved by reducing the amount of data transmitted with the request.

[0010] The above description is provided as an overview of only some of the implementations disclosed herein. Those implementations and other implementations are described in more detail herein.

[0011] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter that appear at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram of an example environment in which the implementations disclosed herein can be implemented.

[0013] Figure 2 shows a series of events that occur on a client device and the state of the metadata of the client device when the events occur.

[0014] Figure 3 is a flowchart showing the previous metadata stored on a client device and the operations generated based on the difference between the current metadata and the previous metadata.

[0015] Figure 4 is a flowchart showing the previous metadata stored on a remote device and the result of the metadata after performing one or more operations on the metadata.

[0016] Figure 5 is a flowchart showing an example method according to various implementations disclosed herein.

[0017] Figure 6is a flowchart showing an example method in accordance with various implementations disclosed herein.

[0018] Figure 7 Shows an example architecture of a computing device. DETAILED DESCRIPTION

[0019] First, turn to Figure 1 , which shows an example environment in which various implementations may be performed. Figure 1 Includes a client device 100 that executes an instance of a client automated assistant 120. Additionally, the example environment includes a remote device 110, which may be, for example, a server that communicates with the client device 100. The remote device 110 is executing one or more cloud-based automated assistant components, such as a remote automated assistant 130, which may be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) that include the remote device 110. The "cloud" computing systems may include one or more computing systems that are communicatively coupled to the assistant device 100 via one or more local area networks and / or wide area networks (e.g., the Internet). The client automated assistant 120, via interaction with the remote automated assistant 130, may form what appears to the user to be a logical instance of an automated assistant, using which the user may engage in a human-machine conversation.

[0020] The assistant device 100 may be, for example: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart TV, and / or a wearable device that includes a computing device (e.g., a watch with a computing device, glasses with a computing device, a virtual or augmented reality computing device).

[0021] The client automation assistant 120, in combination with the remote automation assistant 130, conducts a human-machine conversation session with the user via the user interface input and output devices of the client device 100. To protect user privacy and / or save resources, in many cases, the user typically has to explicitly invoke the client automation assistant 120 before the automation assistant will comprehensively process spoken utterances. The explicit invocation of the client automation assistant 120 can occur in response to a certain user interface input received at the client device 100. For example, the user interface input that can optionally be used to invoke the client automation assistant 120 via the client device 100 can include the actuation of hardware and / or virtual buttons of the client device 100. Additionally, the client automation assistant 120 can include one or more local engines, such as an invocation engine operable to detect the presence of one or more spoken general invocation wake words. The invocation engine can invoke the client automation assistant 120 in response to detecting one of the spoken invocation wake words. For example, the invocation engine can invoke the automation assistant 100 in response to detecting spoken invocation wake words such as "Hey Assistant", "OK Assistant", and / or "Assistant". The invocation engine can continuously process (e.g., when not in an "inactive" mode) a stream of audio data frames based on the output from one or more microphones of the client device 100 to monitor for the occurrence of a spoken invocation phrase. When monitoring for the occurrence of a spoken invocation phrase, the invocation engine discards any audio data frames that do not include a spoken invocation phrase (e.g., after temporarily storing them in a buffer). However, when the invocation engine detects the occurrence of a spoken invocation phrase in the processed audio data frames, the invocation engine can invoke the automation assistant 120. As used herein, "invoking" the automation assistant 120 can include causing one or more previously inactive functions of the client automation assistant 120 to be activated. For example, invoking the client automation assistant 120 can include causing one or more local engines and / or cloud-based automation assistant components to further process the audio data frame based on which the invocation phrase was detected, and / or one or more subsequent audio data frames (whereas prior to the invocation, no further processing of the audio data frames occurred). For example, local and / or cloud-based components can use an ASR model to process the captured audio data in response to the invocation of the client automation assistant 120.

[0022] The client automation assistant 120 and / or the remote automation assistant may include one or more request processor 195 components, and the one or more request processor components may process user spoken utterances received by the client device 100. For example, one or more of the client automation assistant 120 and / or the remote automation assistant 130 may include an automatic speech recognition (ASR) engine, a natural language understanding (NLU) engine, a text-to-speech (TTS) engine, and / or a fulfillment engine. In some implementations, one or more of these engines may be omitted and implemented only by a part of the automation assistant (e.g., implemented only by the cloud-based automation assistant component 130) and / or additional engines may be provided (e.g., the invocation engine described above).

[0023] The ASR engine may process audio data capturing the spoken utterance to generate an identification of the spoken utterance. For example, the ASR engine may utilize one or more ASR machine learning models to process the audio data to generate a prediction of the identified text corresponding to the utterance. In some of those implementations, the ASR engine may generate a corresponding confidence metric for each of one or more identified terms, and the confidence metric indicates the confidence that the predicted term corresponds to the spoken utterance.

[0024] The TTS engine may convert text to synthesized speech and may rely on one or more speech synthesis neural network models to do so. For example, the TTS engine may be utilized to convert a text response to audio data that includes a synthesized version of the text, and the synthesized version is rendered audibly via the hardware speaker of the client device 100.

[0025] The NLU engine determines the semantic meaning of the audio and / or the text converted from the audio by the ASR engine and determines assistant actions corresponding to those semantic meanings. In some implementations, the NLU engine determines the assistant actions as intents and / or parameters determined based on the identification by the ASR engine. In some cases, the NLU engine may parse the intent and / or parameter based on a single utterance of the user, and in other cases, may generate prompts based on the unparsed intent and / or parameter, those prompts are rendered to the user, and the NLU engine utilizes the user response to those prompts to parse the intent and / or parameter. In those cases, the NLU engine may optionally work in cooperation with a dialogue manager engine that determines the unparsed intent and / or parameter and / or generates corresponding prompts. The NLU engine may utilize one or more NLU machine learning models to determine the intent and / or parameter.

[0026] The fulfillment engine can cause the execution of assistant actions determined by the NLU engine. For example, if the NLU engine determines the assistant action of "turning on the kitchen lights", the fulfillment engine can cause the transmission of corresponding data (either directly to the lights or to a remote server associated with the manufacturer of the lights) to cause the "kitchen lights" to be "turned on". As another example, if the NLU engine determines the assistant action of "providing a summary of the user's meetings today", the fulfillment engine can access the user's contacts, identify the phone number of the user indicated in the spoken words, and cause one or more applications executed on the client device 100 to initiate a phone call with the intended party.

[0027] In some implementations, due to the computing resource limitations of the client computing device, some or all of the processing of the request can be remotely handled by the remote automation assistant 130 executing on the remote computing device 110. For example, due to limited computing resources (such as processor power, battery life, and / or limited memory), the client automation assistant 120 may not be configured to handle requests received by the client device 100. For example, the client automation assistant 120 may include a call engine, as described above, but may not be configured to perform ASR and / or NLU. Thus, when a request is received, the audio data (or in some cases, the text representing the audio data) is transmitted to the remote automation assistant 130 for further processing. The remote device 110 may have more computing resources to allow the remote automation assistant 130 to perform ASR, NLU, and / or other processing. Once processed, the remote automation assistant 130 can determine how to fulfill the request and facilitate causing the execution of one or more actions. For example, the remote automation assistant 130 can determine that the request is to make a phone call and can generate a request to make a phone call. This request can be provided to the client device, and the client automation assistant 120 can further facilitate providing the request to an application to cause the phone call to be made.

[0028] In some implementations, the remote automation assistant 130 may need at least some metadata locally stored on the client device to process requests. The metadata database 140 may include, for example, user contact information, user settings of the client device 100 and / or one or more applications executed on the client device, indications of applications installed on the client device 100, capabilities of the client device 100 (e.g., indications of UI components), and / or other metadata that is locally accessible on the client device 100 but may not be available to the remote device 110. When the user utters a verbal statement, the client automation assistant 120 may provide some or all of the available metadata to the remote automation assistant 130 such that the remote automation assistant 130 can process the request. For example, for a request of "Call Sue", the remote automation assistant 130 may need to access the user's contacts such that the phone number of "Sue" can be identified and included in the request transmitted to the phone call application.

[0029] In some implementations, the metadata may include indications of applications currently installed on the client device. For example, the metadata may include indications that the client device has a calendar application, a music playback application, and an application that can make phone calls. In some cases, the user may install additional applications and / or uninstall applications between requests to the automation assistant application. Thus, at any given time, the state of the applications currently installed on the client device can be determined and that state is dynamic based on user actions between state determinations.

[0030] In some implementations, the metadata may include capabilities of the client device. Capabilities may include, for example, hardware indications associated with the client device (e.g., GUI, headphones, audio capabilities, connected external peripherals) and / or other metadata that may include indications of device capabilities (e.g., processor capabilities, memory capacity, battery life). Thus, the metadata identified for the client device can indicate what types of requests the client device can handle and / or whether the client device is configured to handle a particular type of request.

[0031] In some implementations, the metadata may include information specific to a particular application installed on the client device. For example, the client device may have a locally installed calendar application and the calendar application may have metadata associated therewith that indicates entries in the calendar application. Also, for example, as previously described, the messaging application and / or the application that makes phone calls may have a contact list that the user can add to, delete from, and / or update between requests that utilize the application.

[0032] In some implementations, the local automation assistant 120 may determine that a request should be handled at least in part by the remote automation assistant 130, and that the remote automation assistant 130 will need additional information in the form of metadata to process the request. For example, the remote automation assistant 130 may include NLU components not available to the client device 100. To determine the intent of the request, at least a portion of the metadata locally stored on the client device 100 may be needed so that the NLU components of the remote automation assistant can determine the intent and / or otherwise process the request. Thus, in some cases, the client device 100 may provide the metadata via an utterance (or text representative of the utterance), and the remote automation assistant 130 may utilize the metadata to process the utterance.

[0033] For example, a user may say the utterance “OK Assistant, Call Bob”. The phrase “OK Assistant” may be handled by an invocation engine that is a component of the client automation assistant 120, but due to resource limitations, the client automation assistant may not be able to perform ASR and / or NLU (or may have limited execution capabilities). Thus, the utterance (or text representation) may be provided to the remote automation assistant 130, which may have more computing resources and thus more ability to process requests. The client automation assistant 120 may utilize the unprocessed and / or partially processed utterance to provide metadata that can be utilized by the remote automation assistant 130. For example, the metadata database 140 may include contact information locally stored on the client device 100. The contact information may be provided to the remote automation assistant 130 so that the remote automation assistant 130 can generate requests to target one or more components and / or applications executing on the client device 100, such as the application 190.

[0034] In some implementations, the client automation assistant 120 may not be configured to process received audio data and may provide a metadata corpus with all requests that allows the remote automation assistant 130 to handle the processing of any request. For example, the client automation assistant 120 may not be able to determine the intent of the request and thus may not be able to determine what metadata to provide to the remote computing device 110. Thus, for each request, the client automation assistant 120 may provide metadata that includes, for example, application data from multiple applications, device capabilities, indications of applications installed on the client device 100, and / or additional metadata such that regardless of the request, the remote automation assistant 130 can access the metadata needed to process the request.

[0035] In some implementations, remote automated assistant 130 and / or components of remote device 110 can store metadata in remote metadata database 180. For subsequent requests similar to the one submitted by the user (e.g., a second request to "call" a contact), remote metadata database 180 has a copy of the metadata to process the request, so client-side automated assistant 120 does not need to transmit the metadata with the spoken utterance. Thus, for subsequent calls, when the metadata has not changed between requests, there is no need to transmit the metadata with the request, thereby saving computing resources. Remote automated assistant 130 can utilize the copy in remote metadata database 180 to process the request.

[0036] However, in some cases, users may update, add, and / or delete metadata between requests. For example, refer to Figure 2 , initial metadata 205 includes two entries: "Sue" and "Bob". The user speaks the request "Call Bob", and initial metadata 205 can be provided to remote device 120 along with the partially processed request. Subsequently, the user updates contact list 215, resulting in new metadata 220. After updating the contact information, the user issues another request to place a phone call ("Call Jim" 225). Since remote device 110 has a copy of initial metadata 205 (in this case, excluding the entry "Jim"), operation 220 is provided to remote automated assistant 120 along with the audio data before remote automated assistant 130 processes the request.

[0037] Reference again Figure 1 , client automated assistant 120 includes a metadata difference engine 160 that can determine differences between two sets of metadata. For example, when metadata is transmitted to remote device 110, a copy of the transmitted metadata can be stored in previous metadata database 150. New metadata can be identified and / or generated, and metadata difference engine 160 can compare the new metadata to the metadata stored in previous metadata database 150. A difference between the previously transmitted metadata and the current (i.e., "new" metadata) can be generated to determine what may have changed since the previous metadata was transmitted (e.g., metadata was added, deleted from, and / or the user otherwise changed the metadata).

[0038] For example, refer to Figure 3, examples of generating metadata differences between two sets of metadata are provided. As shown, the initial metadata 205 includes two entries for the user's contacts. After the user adds a contact, the new metadata 220 includes three entries (i.e., "Jim" is added). The two sets of metadata are provided to the metadata difference engine 160, which determines the difference 305 between the two sets of metadata. As previously described, when the user submits a request such as making a phone call, the differential metadata 230 is generated and can be provided to the remote automation assistant 130.

[0039] In some implementations, the operation generator 165 can determine one or more operations based on the identified differences between the previous metadata and the current metadata, and the one or more operations can generate the current metadata when executed on the previous metadata. For example, returning to Figure 3 the example of, the entry "Jim" is included in the metadata 220 but not in the metadata 205. In response, the operation generator 165 can determine that the operation "Add Jim: 222-222-2222" can generate the metadata 220 when executed on the metadata 205.

[0040] Once the operations are provided to the remote automation assistant 130, the operation execution module 170 can identify the previous metadata previously provided by the client automation assistant 120 and execute one or more operations to generate new metadata that matches the current metadata of the client device 100. For example, referring to Figure 4 , the metadata 205 is a copy of the metadata initially provided to the remote automation assistant 130. The metadata 205 includes two entries: "Sue" and "Bob". The remote automation assistant 130 can identify the initial metadata 205 in the metadata database 180. For example, when the client automation assistant 120 first transmits a request to the remote automation assistant 130, the request is transmitted together with the initial metadata 205, which is stored in the metadata database 180 and used by the remote automation assistant 130 to process the accompanying request. Then, the new request 225 is received using the operation 230, which was previously generated by the operation generator 165.

[0041] Before processing the request 225, the operation execution module 170 may determine the current metadata of the client device 100 based on the previous metadata 205 and the operation 230. For example, the operation 230 includes the operation "Add Jim: 222-222-2222", which can be performed by the operation execution module 170 on the previous metadata 205, resulting in new (i.e., "current") metadata 220. As shown, the current metadata 220 now includes all the information included in the initial metadata 205 and adds the new entry "Jim". Then, the request processor 195 of the remote automation assistant 130 may utilize the current metadata 220 to process the request and send the fulfillment data 405 to the client automation assistant 120 for fulfillment. In addition, the current metadata 220 may be stored in the metadata database 180 for processing future requests.

[0042] As shown, the remote automation assistant 130 utilizes the new metadata (i.e., the "Jim" entry) to generate a response. However, in some implementations, the remote automation assistant 130 may utilize any metadata included in the metadata database 180, whether the metadata is new or previously provided. For example, if the user adds "Jim" to the contact information between requests, resulting in the operation "Add Jim...", then this operation will be transmitted with the next request, even if the next request does not include a request to "Call Jim" (e.g., a request to "Call Sue"). Thus, the operation is provided with each request, regardless of whether the operation generates new metadata to be used or the request requires previously provided metadata. In other words, the operation is independent of the request, and even in cases where the operation generates new metadata, processing the request may not require the new metadata.

[0043] In some implementations, as previously described, the operation execution module 170 may perform add, delete, update, and / or other operations on the metadata stored in the metadata database 180. For example, the user may uninstall an application that is executing on the client device 100, resulting in the operation "Delete Application 1". After performing this operation, the new metadata will include the applications executing on the client device 100 but not include "Application 1". Also, for example, the user may connect headphones to the client device 100, resulting in audio being output from the headphones instead of from the speakers of the client device 100. An operation "Change audio output from device to headphones" may be generated to indicate the change in the state of the client device 100. Once the operation has been performed, the metadata will reflect the change.

[0044] In some implementations, the remote automation assistant 130 can transmit a confirmation along with the processed request to the client automation assistant, indicating that the metadata has been successfully updated. For example, the operation execution module 170 can first determine that the metadata has been successfully transformed and provide an indication that the transformation is complete before the client device 100 replaces the previous metadata with the current metadata. In the case where the transformation is unsuccessful, the previous metadata can be maintained, new differences can be determined, and / or the operation can be retransmitted so that the metadata stored in the metadata database 180 is always up-to-date and accurate, without the possibility of losing additions, deletions, and / or updates to the metadata.

[0045] In some implementations, the remote automation assistant 130 can handle requests from multiple devices and / or multiple users using the same device. The remote automation assistant 130 can maintain separate metadata for each user and / or each device to ensure that the current state of the client device providing a given request is known. For example, in some implementations, an identifier indicating the user and / or device providing the request can be provided along with the request, and this identifier can be associated with specific metadata stored in the metadata database 180. Thus, for "User 1", the identifier "User 1" can be provided along with the request "Call Jim", and the metadata for "User 1" can be identified in the metadata database. Similarly, when the request originates from "User 2", different metadata can be used to handle the request "Call Bob", and the identifier "User 2" can be provided along with the request to indicate that the metadata associated with "User 2" is to be utilized.

[0046] Reference Figure 5 , a flowchart of a method is provided showing generating one or more operations and transmitting the one or more operations along with a request to update a remote version of metadata stored on a remote device. In some implementations, one or more steps of the method can be omitted or combined, and / or one or more additional steps can be included in the method. The method can be performed by Figure 1 one or more of the components shown in the environment of

[0047] At step 505, a request to perform an action is received. The request can be received by a device that shares one or more characteristics with the client device 100. In some implementations, the request can be received as audio data spoken by a user of the client device 100. The client device 100 is executing an automated assistant 120 that can at least partially perform at least some processing to handle the request. In some implementations, at least a portion of the automated assistant communicates with a remote automated assistant 130 that can perform at least some processing to handle the request. For example, the client device 100 can receive audio data including the request, and the client automated assistant 120 can transmit the audio data to the remote automated assistant 130 for further processing.

[0048] At step 510, previous metadata of the client device is identified. The metadata can include, for example, an indication of an application being executed on the client device, the capabilities of the client device, application-specific information (e.g., calendar entries, user contacts), and / or other information related to the state of the client device 100. In some implementations, the client automated assistant 120 can perform an initial processing of the request (e.g., some initial ASR, NLU) to determine the type of the request and can identify specific metadata that can be used to perform additional processing. For example, a user can speak the request "Call Bob", and the client automated assistant 120 can determine that the request is for the phone call application and can further identify the metadata including the user's contacts. In some implementations, the client automated assistant 120 can not perform any initial processing on the received request, but can identify the metadata of the device status information including all aspects of the client device 100 (e.g., metadata including all applications currently installed on the client device, device capabilities, application data, and / or other metadata related to the state of the client device 100).

[0049] When providing the request to the remote automated assistant 130 for further processing, the metadata of the device can be provided together with the request. For example, to process the request "Call Bob", the remote automated assistant 130 may need the user's contacts that can be stored on the client device 100. Thus, when the request is first received, all the metadata of the client device 100 can be provided to the remote computing device 110 together with the request. Subsequently, when a new request is received, the metadata initially provided to the remote automated assistant 130 (i.e., "previous metadata") is identified at step 510.

[0050] At step 515, the current metadata of the client device is identified. The current metadata is the metadata of client device 100 when a new request is received. In some cases, the user may change one or more aspects of client device 100 between requests such that the current metadata is different from the previous metadata provided with the previous request. In these cases, the current metadata and the previous metadata will not be the same. Before the request can be transmitted to the remote automation assistant, the remote version of the metadata stored in metadata database 180 must be updated to reflect the current state of client device 100.

[0051] At step 520, one or more operations are generated based on the difference between the previous metadata and the current metadata. As previously mentioned, in some cases, the user may change one or more aspects of client device 100 such that the metadata associated with the state of client device 100 is different from the previous metadata provided to remote automation assistant 130. Metadata difference engine 160 can determine the difference between the current metadata stored in metadata database 140 and the previous metadata stored in previous metadata database 150. Any additions, deletions, and / or updates to the metadata can be identified and operations can be generated that reflect the difference between the previous metadata and the current metadata. For example, if based on the difference in metadata, an application named "App 1" has been installed since the previous request was received, an operation "Add App 1" can be generated.

[0052] At step 525, the request and one or more operations are transmitted to the remote computing device for further processing. Operation execution module 170 can execute one or more of the provided operations to transform the previous metadata into the current metadata. For example, metadata database 180 may include the previous metadata that was initially provided to remote automation assistant 130 with the request, and / or that has been transformed one or more times based on operations provided with previous requests. In some implementations, operation execution module 170 can identify the metadata associated with client device 100 and / or the user of client device 100 that provided the request, and execute one or more operations to generate a mirrored version of the current metadata of client device 100. Once the current metadata has been generated, remote automation assistant 130 can use the newly generated version of the metadata to process the request.

[0053] Reference Figure 6 , a flowchart is provided showing a method that causes actions to be performed for version determination based on updated metadata stored on a remote device. In some implementations, one or more steps of the method may be omitted or combined, and / or one or more additional steps may be included in the method. The method may be performed by one or more of the components shown in the Figure 1 environment.

[0054] At step 605, the remote computing device receives a request and one or more operations. The request may be transmitted by a computing device that shares one or more characteristics with the client device 100. The request and one or more operations may be received by a computing device that shares one or more characteristics with the remote device 110. As previously described with respect to Figure 5 what is described, the one or more operations may include additions, deletions, and / or updates that may be performed on a previous version of the metadata stored on the remote computing device 110. For example, initially provided metadata representing the current state of the client computing device 100 may be provided along with the initial request. Techniques for sharing one or more characteristics as described with respect to Figure 5 are used to generate the one or more operations.

[0055] At step 610, previous metadata of the client device is identified. The previous metadata is a remote version of the state of the client device 100 when the most recent request was provided to the remote automation assistant 130. For example, when the remote device 110 receives a first request, metadata and / or one or more operations that may be performed to generate metadata indicating the current state of the client device 100 may be provided along with the request. The metadata may be stored in a database that shares one or more characteristics with the metadata database 180. When the remote automation assistant 130 receives subsequent requests and one or more operations, the previous metadata may be identified in the metadata database 180.

[0056] In some implementations, the metadata database 180 may store metadata for multiple computing devices and / or multiple users of the computing devices. In some implementations, an identifier of the transmitting client device and / or the user providing the request may be provided along with the request and one or more operations. For example, an indication of "User 1" and one or more operations to be performed to generate a remote version of the current state of the client device for "User 1" may be provided along with a request from "User 1". The remote automation assistant 130 may identify the corresponding metadata based on the indication of "User 1", which may be unique to the user and include, for example, the user's contacts, the user's calendar entries, and / or one or more applications accessible to the user.

[0057] At step 615, the previous metadata is updated by performing one or more operations. The operations can include, for example, adding additional information to the previous metadata, deleting information from the previous metadata, and / or otherwise updating the metadata. Once one or more operations are performed on the metadata, the resulting new metadata reflects the current state of the metadata of the client device. Thus, instead of transmitting all of the metadata of the client device along with the request, only one or more operations indicating the changes to the metadata are transmitted. The new metadata version can be stored in the metadata database 180 for further use in processing the request and for subsequent requests.

[0058] At step 620, an action is determined based on the request and the updated metadata. The action can be determined by performing one or more processes on the request, such as ASR, NLU, and / or other methods that can determine the action corresponding to the request. In some implementations, to determine the action, the remote automation assistant 130 may require additional metadata of the client computing device 100. To identify the necessary information, the remote automation assistant 130 can identify the metadata corresponding to the client device in the metadata database 180. As previously mentioned, the metadata stored in the metadata database 180 may have been updated based on one or more operations received along with the request. Thus, the metadata stored in the metadata database 180 is a remote version of the same metadata that existed on the client device 100 when the request was transmitted.

[0059] At step 625, the action is performed by one or more computing devices. For example, the remote automation assistant 130 can generate fulfillment data that can be provided to the client automation assistant 120 such that the client automation assistant 120 can determine the destination for providing the fulfillment data and further facilitate the fulfillment of the request by causing the action to be performed. By way of example, a user can submit a request of "Call Jim" which is provided to the remote automation assistant 130. The remote automation assistant 130 can identify the phone number of "Jim" based on the remote version of the metadata stored on the remote computing device 110 and further generate an action, such as "Call 555-555-5555", that can be returned to the client automation assistant 120. The client automation assistant 120 can be configured to further facilitate the fulfillment of the action, such as by directing the fulfillment data to the application 190 being executed on the client device 100.

[0060] Figure 7FIG. 710 is a block diagram of an example computing device 710 that may optionally be used to perform one or more aspects of the techniques described herein. Computing device 710 generally includes at least one processor 714 that communicates with a plurality of peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724 (including, for example, a memory subsystem 725 and a file storage subsystem 726), a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow a user to interact with computing device 710. Network interface subsystem 716 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0061] User interface input device 722 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into computing device 710 or onto a communication network.

[0062] User interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways of outputting information from computing device 710 to a user or to another machine or computing device.

[0063] Storage subsystem 724 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include selected aspects of methods for performing Figure 5 and Figure 6 and / or logic for implementing the various components depicted in Figure 1 .

[0064] These software modules are typically executed by the processor 714 alone or in combination with other processors. The memory 725 used in the storage subsystem 724 may include multiple memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution and a read-only memory (ROM) 732 for storing fixed instructions therein. The file storage subsystem 726 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical disk drive, or a removable media cartridge. Modules implementing the functionality of certain implementations may be stored in the storage subsystem 724 by the file storage subsystem 726 or in other machines accessible by the processor 714.

[0065] The bus subsystem 712 provides the mechanism for enabling the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0066] The computing device 710 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the Figure 7 description of the depicted computing device 710 is only intended as a specific example for the purpose of illustrating some implementations. Many other configurations of the computing device 710 are possible, which have more or fewer components compared to the Figure 7 depicted computing device.

[0067] In some implementations, a method implemented by one or more processors of a client device is provided, and the method includes receiving a request from a user to perform an action by an automated assistant. The automated assistant includes a local automated assistant component executing on the client device and a remote automated assistant component executing on a remote device. In response to receiving the request, the method further includes identifying previous metadata stored on the client device, where the previous metadata represents the previous state of the client device at a previous time of the most recent previous request to the automated assistant; identifying current metadata, where the current metadata represents the current state of the client device; generating one or more operations based on the difference between the previous metadata and the current metadata, the one or more operations that, when executed by the remote device, will transform the previous metadata into the current metadata; and transmitting the request and the one or more operations to the remote automated assistant component. Transmitting the request and the one or more instructions causes the remote device to update the remote version of the metadata by performing the one or more operations on a version of the previous metadata stored on the remote device; determining an action based on the request and the updated remote version; and causing the action to be executed.

[0068] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0069] In some implementations, the method further includes the client device receiving from the remote device an acknowledgement of a successfully updated remote version of the metadata. In response to receiving the acknowledgement, the method further includes updating the previous metadata to reflect the current metadata.

[0070] In some implementations, the previous metadata includes one or more indications of applications installed on the client device at a previous time, and wherein the current metadata includes (a) an indication of additional applications currently installed on the client device and not among the applications indicated by the previous metadata and / or (b) an indication of the absence of a given application among the applications indicated by the previous metadata that is currently installed on the client device. In some of those implementations, the execution of the action includes using the additional applications.

[0071] In some implementations, the previous metadata includes one or more indications of the capabilities of the client device at a previous time, and wherein the current metadata includes (a) an indication of additional capabilities of the client device that are different from the capabilities indicated by the previous metadata and / or (b) an indication of the absence of a given capability among the capabilities indicated by the previous metadata.

[0072] In some implementations, the previous metadata includes one or more contacts of the user, and wherein the current metadata includes (a) contacts not among the contacts indicated by the previous metadata and / or (b) an indication of the absence of a given contact among the contacts indicated by the previous metadata.

[0073] In some implementations, the current metadata includes additional information not included in the previous metadata, and wherein one or more operations include an instruction to add the additional information.

[0074] In some implementations, the previous metadata includes additional information not included in the current metadata, and wherein one or more operations include an instruction to delete the additional information.

[0075] In other implementations, a method implemented by one or more processors is provided, and the method includes receiving from a client device: a request to perform an action on an automated assistant, and one or more operations, where the automated assistant includes a local automated assistant component executing on the client device and a remote automated assistant component executing on a remote device. In response to receiving the request and the one or more operations, the method further includes identifying a remote version of metadata stored on the remote device, where the remote version of the metadata represents the previous state of the client device when the immediately preceding request was received from the client device; updating the remote version of the metadata by performing the one or more operations on the remote version of the metadata; determining an action based on the request and the updated remote version; and causing the action to be performed.

[0076] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0077] In some implementations, the method further includes receiving an identifier of a user of the client device and / or a device client along with the request and the one or more operations, where identifying the remote version of the metadata is based on the received identifier, and where the remote version of the metadata is unique to the client device and / or the user.

[0078] In some implementations, the method further includes transmitting to the client device an acknowledgement of successfully updating the remote version of the metadata, where transmitting the acknowledgement causes the client device to update a previous version of the metadata stored on the client device to reflect the current metadata of the client device.

[0079] In some implementations, the request includes audio data, and determining the action includes performing at least one of automatic speech recognition, natural language understanding, and / or speech-to-text analysis based on the request.

[0080] In some implementations, the updated remote version includes additional metadata not included in the remote version, and the action is based on at least a portion of the additional metadata.

[0081] In other implementations, a system is provided and the system includes a client computing device and a remote computing device, each computing device including at least one corresponding processor and a corresponding memory storing code executable by the corresponding processor. The client computing device is configured to receive a user request to perform an action on an automated assistant executing on the client computing device. In response to receiving the user request, the client computing device is further configured to identify metadata stored on the client device at a previous time of the immediately preceding user request, where the previous metadata represents a previous state of the client device at the previous time; identify current metadata, where the current metadata represents a current state of the client device; generate one or more operations based on a difference between the previous metadata and the current metadata, the one or more operations when executed converting the previous metadata to the current metadata; and transmit the request and the one or more operations to the remote computing device. The remote computing device is configured to receive the request and the one or more operations from the client device. In response to receiving the request and the one or more operations, the remote computing device is further configured to identify a remote version of the metadata stored on the remote device, where the remote version of the metadata represents the previous state of the client computing device when the immediately preceding request was received; update the remote version of the metadata by performing the one or more operations on the remote version of the metadata; determine an action based on the request and the updated remote version; and cause the action to be executed.

[0082] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0083] In some implementations, when causing the action to be executed, the remote computing device is further configured to transmit fulfillment data to the client computing device; and the client computing device is further configured to: receive the fulfillment data and perform the action using the fulfillment data.

[0084] In some implementations, when causing the action to be executed, the remote computing device is configured to transmit fulfillment data to an additional remote computing device.

[0085] In some implementations, the remote computing device is further configured to transmit an acknowledgement of successfully updating the remote version of the metadata. The client computing device is configured to update the previous metadata to reflect the current metadata.

[0086] In some implementations, the previous metadata includes one or more indications of applications installed on the client device at the previous time, and the current metadata includes (a) an indication of additional applications currently installed on the client device and not among the applications indicated by the previous metadata and / or (b) an indication of a given application that is missing from among the applications indicated by the previous metadata and is currently installed on the client device.

[0087] In some implementations, the previous metadata includes one or more indications of the capabilities of the client device at a previous time, and the current metadata includes (a) an indication of additional capabilities of the client device that are different from the capabilities indicated by the previous metadata and / or (b) an indication of the absence of a given capability among the capabilities indicated by the previous metadata.

[0088] In some implementations, the previous metadata includes one or more contacts of a user.

[0089] Although several implementations have been described and illustrated herein, various other means and / or structures may be used for performing the functions and / or obtaining the results and / or one or more of the advantages described herein, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation many equivalents to the implementations described herein. Accordingly, it is to be understood that the foregoing implementations are presented by way of example only, and that implementations may be practiced otherwise than as specifically described and claimed within the scope of the appended claims and their equivalents. Implementations of the present disclosure relate to each and every individual feature, system, article, material, kit, and / or method described herein. Additionally, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure if such features, systems, articles, materials, kits, and / or methods do not mutually contradict.

[0090] Various implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described herein. Other implementations may include an automated assistant client device (e.g., a client device including at least an automated assistant interface for interfacing with a cloud-based automated assistant component), the automated assistant client device including a processor operable to execute the stored instructions to perform a method such as one or more of the methods described herein. Still other implementations may include a system of one or more servers, the one or more servers including one or more processors operable to execute the stored instructions to perform a method such as one or more of the methods described herein.

[0091] In some implementations discussed herein that may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, and the user's activities and demographic information, relationships between users, etc.), one or more opportunities are provided to the user to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use a user's personal information only after receiving clear authorization from the relevant user for the collection, storage, and / or use of the user's personal information.

[0092] For example, the user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user whose personal information is to be collected to allow control over the information collection related to that user, providing permission or authorization regarding whether information is collected and which parts of the information are collected. For example, one or more such control options can be provided to the user via a communication network. Additionally, specific data can be processed in one or more ways before it is stored or used such that personally identifiable information is removed. As an example, the identity of the user can be processed such that no personally identifiable information can be determined. As another example, the user's geographical location can be generalized to a larger area such that the user's specific location cannot be determined.

Claims

1. A method implemented by one or more processors of a client device, comprising: receiving, from a user, a request to perform an action by an automated assistant, wherein the automated assistant includes a local automated assistant component executing on the client device and a remote automated assistant component executing on a remote device; and in response to receiving the request: identifying previous metadata stored on the client device, wherein the previous metadata represents a previous state of the client device at a previous time of a previous request to the automated assistant; identifying current metadata, wherein the current metadata represents a current state of the client device; generating, based on a difference between the previous metadata and the current metadata, one or more operations that, when executed by the remote device, convert the previous metadata to the current metadata; transmitting the request and the one or more operations to the remote automated assistant component, wherein transmitting the request and the one or more instructions causes the remote device to: update a remote version of the metadata by performing the one or more operations on a version of the previous metadata stored on the remote device; determine the action based on the request and the updated remote version; and cause the action to be executed.

2. The method according to claim 1, further comprising: receiving, by the client device, from the remote device, an acknowledgement of successfully updating the remote version of the metadata; in response to receiving the acknowledgement: updating the previous metadata to reflect the current metadata.

3. The method according to any one of the preceding claims, wherein the previous metadata includes one or more indications of applications installed on the client device at the previous time, and wherein the current metadata includes (a) an indication of additional applications currently installed on the client device and not among the applications indicated by the previous metadata and / or (b) an indication of the absence of a given application among the applications indicated by the previous metadata that is currently installed on the client device.

4. The method according to claim 3, wherein execution of the action includes using the additional applications.

5. The method according to any one of the preceding claims, wherein the previous metadata includes one or more indications of the capabilities of the client device at the previous time, and wherein the current metadata includes (a) an indication of additional capabilities of the client device that are different from the capabilities indicated by the previous metadata and / or (b) an indication of the absence of a given capability among the capabilities indicated by the previous metadata.

6. The method according to any one of the preceding claims, wherein the previous metadata includes one or more contacts of the user, and wherein the current metadata includes (a) contacts not among the contacts indicated by the previous metadata and / or (b) an indication of the absence of a given contact among the contacts indicated by the previous metadata.

7. The method according to any one of the preceding claims, wherein the current metadata includes additional information not included in the previous metadata, and wherein the one or more operations include instructions for adding the additional information.

8. The method according to any one of the preceding claims, wherein the previous metadata includes additional information not included in the current metadata, and wherein the one or more operations include instructions for deleting the additional information.

9. A method implemented by one or more processors of a remote device comprising: receiving from a client device: a request to perform an action on an automated assistant, and one or more operations, wherein the automated assistant includes a local automated assistant component executing on the client device and a remote automated assistant component executing on the remote device; and in response to receiving the request and the one or more operations: identifying a remote version of the metadata stored on the remote device, wherein the remote version of the metadata represents a previous state of the client device when the immediately preceding request was received from the client device; updating the remote version of the metadata by performing the one or more operations on the remote version of the metadata; determining the action based on the request and the updated remote version; and causing the action to be performed.

10. The method according to claim 9, further comprising: receiving an identifier of the client device and / or the user of the client of the device, together with the request and the one or more operations, wherein identifying the remote version of the metadata is based on the received identifier, and wherein the remote version of the metadata is unique to the client device and / or the user.

11. The method according to any one of claims 9 and 10, further comprising: transmitting to the client device a confirmation of successfully updating the remote version of the metadata, wherein transmitting the confirmation causes the client device to update a previous version of the metadata stored on the client device to reflect the current metadata of the client device.

12. The method according to any one of claims 9 to 11, wherein the request includes audio data, and wherein determining the action comprises: performing at least one of automatic speech recognition, natural language understanding, and / or speech-to-text analysis based on the request.

13. The method according to any one of claims 9 to 12, wherein the updated remote version includes additional metadata not included in the remote version, and wherein the action is based on at least a portion of the additional metadata.

14. A system comprising: a client computing device and a remote computing device, each computing device including at least one corresponding processor and a corresponding memory, the memory storing corresponding processor-executable code, wherein the client computing device is configured to: receive a user request to perform an action on an automated assistant executing on the client computing device; in response to receiving the user request: Identify previous metadata stored on the client device at a previous time of a preceding user request, where the previous metadata represents a previous state of the client device at the previous time; Identify current metadata, where the current metadata represents a current state of the client device; Generate one or more operations based on a difference between the previous metadata and the current metadata, the one or more operations, when executed, convert the previous metadata to the current metadata; And Transmit the request and the one or more operations to the remote computing device; And Where the remote computing device is configured to: Receive the request and the one or more operations from the client device; And In response to receiving the request and the one or more operations: Identify a remote version of the metadata stored on the remote device, where the remote version of the metadata represents the previous state of the client computing device when the preceding request was received; Update the remote version of the metadata by performing the one or more operations on the remote version of the metadata; Determine the action based on the request and the updated remote version; And Cause the action to be executed.

15. The system of claim 14, where when causing the action to be executed, the remote computing device is configured to: Transmit fulfillment data to the client computing device; and Where the client computing device is further configured to: Receive the fulfillment data, and Execute the action using the fulfillment data.

16. The system of any one of claims 14 and 15, where when causing the action to be executed, the remote computing device is configured to: Transmit fulfillment data to an additional remote computing device.

17. The system of any one of claims 14 to 16, where the remote computing device is further configured to: Transmit an acknowledgement that the remote version of the metadata has been successfully updated; and Where the client computing device is configured to: Update the previous metadata to reflect the current metadata.

18. The system of any one of claims 14 to 17, where the previous metadata includes one or more indications of applications installed on the client device at the previous time, and where the current metadata includes (a) an indication of additional applications currently installed on the client device and not among the applications indicated by the previous metadata and / or (b) an indication of a lack of a given application among the applications indicated by the previous metadata that is currently installed on the client device.

19. The system of any one of claims 14 to 18, where the previous metadata includes one or more indications of the capabilities of the client device at the previous time, and where the current metadata includes (a) an indication of additional capabilities of the client device that are different from the capabilities indicated by the previous metadata and / or (b) an indication of a lack of a given capability among the capabilities indicated by the previous metadata.

20. The system according to any one of claims 14 to 19, wherein the previous metadata includes one or more contacts of the user.

21. A computer program comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform the method according to any one of claims 1 to 13.

22. One or more computing devices configured to perform the method according to any one of claims 1 to 13.

Images