Sending messages from smart speakers and smart displays via smartphones

Abstract

The invention relates to sending messages from smart speakers and smart displays via a smartphone. Techniques are described herein for sending messages using a messaging application on a client device, such as a smartphone, using a smart device, such as an independent assistant-centric interactive speaker and / or an independent assistant-centric interactive display with a speaker. The method comprises the following steps: a first device receives a request for sending a message to a second user from a first user; determining that a message passing application corresponding to the request is unavailable on the first device; and in response to determining that the messaging application corresponding to the request is unavailable on the first device: selecting a second device on which the messaging application corresponding to the request is available; and sending a command to the second device, the command causing the second device to send a message from the first user to the second user using a messaging application on the second device.

Description

Description of the case

[0001] This application is a divisional application of Chinese invention patent application 202080100650.7, with a filing date of December 15, 2020. Technical Field

[0002] This application relates to sending messages from smart speakers and smart displays via smartphones. Background Art

[0003] A human may interact with an interactive software application referred to herein as an "automated assistant". For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide input to the automated assistant, which may cause the automated assistant to generate and provide a responsive output, control one or more smart devices, and / or perform one or more other functions. The input provided by the user may be, for example, touch input (e.g., via a touch screen), gestures (e.g., via camera detection), and / or spoken natural language input (i.e., speech detected via a microphone), which may in some cases be converted to text (or other semantic representations) and then further processed.

[0004] Many users may use multiple client devices to engage with the automated assistant. For example, some users may have a coordinated "ecosystem" of client devices, such as one or more smartphones, one or more tablets, one or more wearable computing devices, one or more smart TVs, one or more stand-alone assistant-centric interactive speakers, and one or more stand-alone assistant-centric interactive displays with speakers, among other client devices. The user may use any of these client devices to engage in a human-machine conversation with the automated assistant (assuming the automated assistant client is installed).

[0005] There are technologies that enable a user (e.g., a single user, multiple users in a household, coworkers, cohabitants, etc.) to use a messaging application corresponding to a request to send a message to another user using an automated assistant client in response to a request from the user received by an automated assistant client, the request being present on a particular client device on which the automated assistant client receiving the request is running. However, such technologies have disadvantages such as low reliability, because if the messaging application corresponding to the request received by the automated assistant client is not present on the particular client device on which the automated assistant client receiving the request is running, the automated assistant client may not be able to send a message to the other user in response to the request. In addition, by requiring the messaging application to be installed on an additional client device, even if the messaging application is already installed on another client device associated with the user, such technologies can also increase the burden on computing and / or network resources. Summary of the invention

[0006] Embodiments disclosed herein relate to using a smart device, such as a stand-alone assistant-centric interactive speaker and / or a stand-alone assistant-centric interactive display with a speaker, to send a message using a messaging application on a client device, such as a smartphone. For example, a first device (e.g., a smart device) may receive a request from a first user to send a message to a second user. It may then be determined that the messaging application corresponding to the request is not available on the first device. In response to determining that the messaging application corresponding to the request is not available on the first device, a second device (e.g., a client device, such as a smartphone) may then be selected on which the messaging application corresponding to the request is available, and a command may then be sent to the second device to cause the second device to send the message from the first user to the second user using the messaging application on the second device.

[0007] In some implementations, determining that the messaging application corresponding to the request is not available on the first device may include determining that the messaging application is not present on the first device and determining that the messaging application is not accessible from the first device using an application programming interface.

[0008] In some embodiments, the first device may determine the account of the first user based on the context information, and selecting the second device may be based on the account of the first user. In some embodiments, the request may be included in an utterance, and determining the account of the first user may include determining that the utterance was spoken by the first user based on a speaker identification feature of the utterance. In other embodiments, the request may be included in an utterance spoken by a person, and determining the account of the first user may include receiving an image of the person from a camera of the first device, and determining that the utterance was spoken by the first user based on the image.

[0009] In some implementations, the first device may be a non-smartphone device, and the second device may be a smartphone device of the first user. In some implementations, the request may specify a messaging application, and the messaging application may send the message from the second device to a third device associated with the second user using end-to-end encryption.

[0010] In some implementations, the receiving, determining, selecting, and sending may be performed by a first automated assistant application running on a first device, and the command may be interpreted by a second automated assistant application running on a second device to cause the second automated assistant application to call a messaging application running on the second device to cause the messaging application to send a message from the first user to the second user. In some implementations, the second automated assistant application may call the messaging application without displaying the second automated assistant application output, and the messaging application may send the message without displaying the messaging application output.

[0011] In some embodiments, the command can be interpreted by the second automated assistant application running on the second device so that the second automated assistant application responds by treating the request as if it were directly received by the second automated assistant application running on the second device. In some embodiments, a reply can be received from the second device indicating that the message has been sent to the second user, and in response to receiving the reply, the first device can output an audio or visual indication that the message has been sent.

[0012] In some implementations, the determining, the selecting, and the sending may be performed by the first device. In other implementations, the determining, the selecting, and the sending may be performed by a computer server external to the first device and the second device.

[0013] In some additional or alternative embodiments, a first automated assistant application running on a first device associated with a first user (e.g., a client device such as a smartphone) can receive a message received by a messaging application running on the first device. The first automated assistant application can then determine a second device associated with the first user (e.g., a smart device), where the second device is running a second automated assistant application and the messaging application is not available on the second device. The first automated assistant application can then send the message to the second automated assistant application running on the second device.

[0014] In some implementations, a first automated assistant application running on a first device can receive a command from a second automated assistant application running on a second device to send a reply message using a messaging application. The first automated assistant application can then call the messaging application to cause the messaging application to send a reply message.

[0015] In some implementations, the first automated assistant application running on the first device can send a confirmation to the second automated assistant application running on the second device indicating that the reply message was sent. In some implementations, the first device is a smartphone device and the second device is a non-smartphone device. In some implementations, sending the message can include sending a command to cause the second automated assistant application running on the second device to present the message at the second device.

[0016] In some additional or alternative embodiments, a request to send a message to a second user may be received from a first user. A determination may be made as to whether a messaging application corresponding to the request is available on a system (e.g., a smart device). In response to determining that the messaging application corresponding to the request is available on the system, the message may be sent from the first user to the second user using the messaging application on the system. In response to determining that the messaging application corresponding to the request is not available on the system, an external device (e.g., a client device such as a smart phone) on which the messaging application corresponding to the request is available may be selected, and a command may be sent to the external device causing the external device to send the message from the first user to the second user using the messaging application on the external device.

[0017] In some embodiments, the receiving, the determining, the selecting, and the sending can be performed by a first automated assistant application running on a first device, and the command can be interpreted by a second automated assistant application running on a second device so that the second automated assistant application calls a messaging application running on the second device so that the messaging application sends a message from the first user to the second user.

[0018] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An example environment is schematically depicted in which selected aspects of the present disclosure may be implemented, according to various implementations.

[0020] Figure 2 Flowcharts illustrating example methods for practicing selected aspects of the present disclosure are depicted.

[0021] Figure 3 Another flow chart illustrating an example method for practicing selected aspects of the present disclosure is depicted.

[0022] Figure 4Another flow chart illustrating an example method for practicing selected aspects of the present disclosure is depicted.

[0023] Figure 5 An example architecture of a computing device is illustrated. DETAILED DESCRIPTION

[0024] Figure 1 An example environment 100 is schematically depicted in accordance with various implementations in which selected aspects of the present disclosure may be implemented. Figure 1 Any computing device depicted elsewhere in the drawings may include logic such as one or more microprocessors (e.g., central processing units or "CPUs", graphics processing units or "GPUs") that execute computer-readable instructions stored in memory, or other types of logic such as application specific integrated circuits ("ASICs") and field programmable gate arrays ("FPGAs"), etc. This may be implemented using one or more server computing devices that form what is sometimes referred to as a "cloud infrastructure". Figure 1 Some systems such as computer server 170 are depicted in , although this is not required.

[0025] In embodiments, environment 100 may include one or more smart devices 110, such as a standalone assistant-centric interactive speaker, a standalone assistant-centric interactive display with a speaker, a smart appliance such as a smart TV, and / or a wearable device including a user of a computing device (e.g., a watch with a user of a computing device, glasses with a user of a computing device, a virtual or augmented reality computing device, etc.). Additional and / or alternative smart devices 110 may be provided. In embodiments, smart device 110 may be a non-smartphone device.

[0026] In embodiments, the smart device 110 may include an instance of the smart device automated assistant 120. The smart device automated assistant 120 may process user input received from an input device of the I / O component 130, such as voice input detected via a microphone of the I / O component 130, touch input received via a touch screen display of the I / O component 130, images detected via a camera of the I / O component 130, etc. In addition, the smart device automated assistant 120 may optionally present various outputs via an output device of the I / O component 130, such as a speaker and / or a touch screen display of the I / O component 130.

[0027] In an embodiment, the smart device 110 may optionally include one or more messaging applications 135. Each of the messaging applications 135 may be an application that provides the following functionality: sending messages from a user of the smart device 110 to one or more other users (e.g., a user of another smart device 110, a user of a client device 140-1, ..., 140-n, or any other user of any other device), and receiving messages sent to the user of the smart device 110 from one or more other users.

[0028] In embodiments, environment 100 may also include client devices 140-1, ..., 140-n, each of which may be a mobile phone computing device and may be associated with a different user. For example, client device 140-1 may be a mobile phone computing device associated with a first user (e.g., a smartphone device of the first user), and client device 140-n may be a mobile phone computing device associated with an nth user (e.g., a smartphone device of the nth user). Additional and / or alternative client devices 140-1, ..., 140-n associated with various users may be provided, including desktop computing devices, laptop computing devices, and / or tablet computing devices, etc.

[0029] In embodiments, each of client devices 140-1, ..., 140-n may include an instance of client device automated assistant 150. Client device automated assistant 150 may process user input received from input devices of I / O component 155, such as voice input detected via a microphone of I / O component 155, touch input received via a touch screen display of I / O component 155, images detected via a camera of I / O component 155, etc. In addition, client device automated assistant 150 may optionally present various outputs via output devices of I / O component 155, such as a speaker and / or touch screen display of I / O component 155.

[0030] In an embodiment, each of the client devices 140-1, ..., 140-n may include one or more messaging applications 160. Each of the messaging applications 160 may be an application that provides the following functionality: sending messages from a user of a particular client device 140-1, ..., 140-n to one or more other users (e.g., a user of another client device 140-1, ..., 140-n or any other user of any other device), and receiving messages sent to the user of the particular client device 140-1, ..., 140-n from the one or more other users. The messaging application 160 may utilize end-to-end encryption to send and receive messages. For example, the messaging application 160 may encrypt messages so that the encrypted messages can only be decrypted by the user who is the intended recipient of the message at a device associated with the user who is the intended recipient of the message (e.g., using another instance of the messaging application 160). In other embodiments, the messaging application 160 may send messages that are not encrypted (e.g., as plaintext) or that use other forms of encryption (e.g., not end-to-end encryption).

[0031] In embodiments, environment 100 may include computer server 170, which may be a node in a cloud infrastructure (cloud computing environment). Computer server 170 may include cloud-based automated assistant component 180. In various embodiments, one or more instances of smart device automated assistant 120 of smart device 110 and / or one or more instances of client device automated assistant 150 of client devices 140-1, ..., 140-n may interface with cloud-based automated assistant component 180 in processing input and / or generating output based on input and / or generating commands based on input.

[0032] For example, the smart device automated assistant 120 of the smart device 110 may transmit audio data corresponding to the voice input received after invoking the smart device automated assistant 120 at the smart device 110 to the cloud-based automated assistant component 180. The invocation of the smart device automated assistant 120 at the smart device 110 may be based on detecting an invocation phrase (e.g., "OK, assistant"), an interaction of a hardware button or a graphical button that invokes the smart device automated assistant 120, a gesture detected via a camera of the I / O component 130, and / or other invocation signals. The cloud-based automated assistant component 180 may convert the audio data to text using a speech-to-text (STT) processor 181, and perform NLU on the text using a natural language understanding (NLU) engine 182 to determine an appropriate response. For example, the appropriate response may be a text response that may optionally be converted to generated speech using a text-to-speech (TTS) processor 183 and transmitted to the smart device automated assistant 120 to present the generated speech via a speaker of the I / O component 130.

[0033] In another example, client device automated assistant 150 of client devices 140-1, ..., 140-n may transmit audio data corresponding to speech input received following an invocation of client device automated assistant 150 at client devices 140-1, ..., 140-n to cloud-based automated assistant component 180. The invocation of client device automated assistant 150 at client devices 140-1, ..., 140-n may be based on detecting an invocation phrase (e.g., "OK, Assistant"), an interaction of a hardware button or graphical button to invoke client device automated assistant 150, in response to a gesture detected via a camera of I / O component 155, and / or other invocation signals. Cloud-based automated assistant component 180 may convert the audio data to text using STT processor 181 and perform NLU on the text using NLU engine 182 to determine an appropriate response. For example, an appropriate response may be a text response, which may optionally be converted to generated speech using TTS processor 183 and transmitted to client device automated assistant 150 for presentation of the generated speech via a speaker of I / O component 155 .

[0034] In an embodiment, the smart device 110, the client devices 140-1, ..., 140-n and the computer server 170 can communicate with each other via a general Figure 1The computer network 190 is communicatively coupled to one or more wide area networks (e.g., the Internet) indicated by the computer network 190. Note that the smart device 110, the client devices 140-1, ..., 140-n, and the computer server 170 can utilize one or more local area networks to access the computer network 190 and / or communicate locally with each other. Such local area networks may include Wi-Fi networks and / or mesh networks.

[0035] Figure 2 1 is a flowchart illustrating an example method 200 for sending a message using a messaging application 160 on a client device 140-1, ..., 140-n, such as a mobile phone computing device, using a smart device 110, such as a stand-alone assistant-centric interactive speaker, a stand-alone assistant-centric interactive display with a speaker, a smart appliance such as a smart TV, and / or a wearable device of a user, according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of a smart device 110, a client device 140-1, ..., 140-n, and / or a computer server 170. Furthermore, while the operations of method 200 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0036] At block 210, the system may receive a request from the first user via the first device to send a message to the second user. In an embodiment, at block 210, the smart device automated assistant 120 of the smart device 110 (i.e., the first device) may receive a request from the first user to send a message to the second user in the form of a spoken utterance received via the I / O component 130 of the smart device 110 (e.g., via a microphone).

[0037] Still referring to block 210, in embodiments, the smart device automated assistant 120 of the smart device 110 may be invoked based on the smart device automated assistant 120 detecting an invocation phrase (e.g., “OK Assistant”), an interaction of a hardware button or graphical button that invokes the smart device automated assistant 120, in response to a gesture detected via a camera of the I / O component 130, and / or other invocation signals. In embodiments, after invoking the smart device automated assistant 120, the smart device automated assistant 120 may process audio data (e.g., spoken utterances following the invocation phrase) received by a microphone of the I / O component 130 of the smart device 110 to identify in the audio data a request from the first user to send a message to the second user.

[0038] Still referring to box 210, in an embodiment, the smart device automated assistant 120 of the smart device 110 can convert the audio data corresponding to the voice input received after invoking the smart device automated assistant 120 into text and perform NLU on the text to identify the request from the first user to send a message to the second user, which is contained in the audio data. In other embodiments, the smart device automated assistant 120 of the smart device 110 can transmit the audio data corresponding to the voice input received after the invocation of the smart device automated assistant 120 at the smart device 110 to the cloud-based automated assistant component 180 of the computer server 170. The cloud-based automated assistant component 180 can then convert the audio data using the STT processor 181 and perform NLU on the text using the NLU engine 182 to identify the request from the first user to send a message to the second user contained in the audio data.

[0039] Still referring to block 210, in an embodiment, the request received by the smart device automated assistant 120 may include a message, a message recipient (e.g., a second user), and optionally a messaging application that the smart device automated assistant 120 may identify from the request. For example, the smart device automated assistant 120 of the smart device 110 may convert a voice input (e.g., an utterance) received after the invocation of the smart device automated assistant 120 into text, "Tell User1 using MessagingApp1, I am leaving for the grocery store in 10 minutes." In this example, the smart device automated assistant 120 may perform NLU to identify "I am leaving for the grocery store in 10 minutes" as a message, User1 as a message recipient, and MessagingApp1 as a messaging application.

[0040] Still referring to block 210 , in an embodiment, if the request received by the smart device automation assistant 120 of the smart device 110 does not specify a messaging application, the smart device automation assistant 120 may use preconfigured rules to determine the messaging application (e.g., using the messaging application most recently specified by the first user with respect to any message recipient, or using the messaging application most recently specified by the first user with respect to a second user, etc.).

[0041] Still reference Figure 2In block 220, the system may determine whether the situation is that the messaging application corresponding to the request is not available on the first device. In an embodiment, in block 220, the smart device automated assistant 120 of the smart device 110 (i.e., the first device) may determine whether the messaging application corresponding to the request received in block 210 (determined in block 210) is not available on the smart device 110.

[0042] Still referring to block 220, in an embodiment, if the messaging application corresponding to the request neither exists (is available locally) on the smart device 110 (e.g., the messaging application is not one of the messaging applications 135 installed on the smart device 110) nor is accessible from the smart device 110 (e.g., the messaging application is not remotely accessible by the smart device 110 using an application programming interface (API) or a web service), then the smart device automation assistant 120 of the smart device 110 may determine that the situation is not a situation where the messaging application corresponding to the request received at block 210 is not available on the smart device 110. In an embodiment, if the messaging application corresponding to the request exists (is available locally) on the smart device 110 (e.g., the messaging application is one of the messaging applications 135 installed on the smart device 110) and / or is accessible from the smart device 110 (e.g., the messaging application is accessible by the smart device 110 using an API or a web service), then the smart device automation assistant 120 of the smart device 110 may determine that the situation is not a situation where the messaging application corresponding to the request received at block 210 is not available on the smart device 110.

[0043] Still referring to block 220, in an embodiment, if the smart device automation assistant 120 of the smart device 110 determines that the situation is that the messaging application corresponding to the request received at block 210 is not available on the smart device 110, the flow proceeds to block 240. On the other hand, in an embodiment, if the smart device automation assistant 120 of the smart device 110 determines that the situation is not that the messaging application corresponding to the request received at block 210 is not available on the smart device 110, the flow proceeds to block 230.

[0044] Still reference Figure 2 In block 230, the system may send a message from the first user to the second user using a messaging application corresponding to the request available on the first device. In an embodiment, in block 230, the smart device automated assistant 120 of the smart device 110 (i.e., the first device) may cause a message to be sent from the first user to the second user according to the request received in block 210 using a messaging application corresponding to the request available on the smart device 110.

[0045] Still referring to block 230, in an embodiment, if the messaging application corresponding to the request is one of the messaging applications 135 installed on the smart device 110, the smart device automated assistant 120 may send an action intent request to the messaging application 135 corresponding to the request. The action intent request may include the message from block 210 and the message recipient (e.g., the second user), and in response to receiving the action intent request, the messaging application 135 may send the message to the message recipient according to the received action intent request. The action intent request may optionally include a message sender (e.g., the first user), which may specify a user account from which the message is to be sent. The action intent request may also optionally include an identification of the messaging application 135 corresponding to the request to be used to send the message.

[0046] In an embodiment, an action intent request may refer to a request from an application (e.g., smart device automation assistant 130) to perform a specific action (e.g., send a message) via the application or a separate application (e.g., one of the messaging applications 135). For example, the action intent request may be generated by a third-party application and identify one or more of an action to be performed, a custom identifier, content data associated with the action to be performed, and / or any other data that may be associated with the operation performed by the application. The action intent request may be initially received by an operating system, a device application, an automation assistant, and / or any other engine or module available to the device that initiated the action intent request. The action intent request may then be cataloged or otherwise queued at the initiating device and made accessible to one or more instances of the smart device automation assistant 120 and the client device automation assistant 150 of each of the client devices 140-1, ..., 140-n.

[0047] Still referring to block 230, in an embodiment, if a messaging application corresponding to the request is remotely accessible from the smart device 110 (e.g., the messaging application is accessible by the smart device 110 using an API or a Web service), the smart device automation assistant 120 may use the API, Web service, or another remotely accessible interface to send a message from the first user to the second user according to the request received at block 210. In an embodiment, the smart device automation assistant 120 may provide the message from block 210 and the message recipient (e.g., the second user) to the API, Web service, or other remotely accessible user interface. The smart device automation assistant 120 may optionally provide the API, Web service, or other remotely accessible user interface with the message sender (e.g., the first user), who may specify a user account from which the message is to be sent. In addition, the smart device automation assistant 120 may optionally provide an identification of a messaging application 135 corresponding to the request to be used to send the message to the API, Web service, or other remotely accessible user interface.

[0048] Still reference Figure 2 At block 240, the system at the first device may determine the account of the first user based on context information. In an embodiment, at block 240, the smart device automated assistant 120 of the smart device 110 (i.e., the first device) may determine the account of the first user (i.e., the user from whom the request was received at block 210) based on context information such as speaker recognition features and / or biometric data.

[0049] Still referring to block 240, in some embodiments, the request received in block 210 may be included in an utterance of the first user, and the smart device automated assistant 120 may determine the account of the first user by determining that the utterance was spoken by the first user based on speaker identification features (e.g., Mel-frequency cepstral coefficient (MFCC) features) of one or more utterances of known users, which together may form a feature vector. For example, the smart device 110 may store speaker identification features for each of the known users who said “OK, Assistant”. In response to the speaker identification features in the currently received utterance being sufficiently matched with the stored speaker identification features of the known user User 1 who said “OK, Assistant”, the smart device 110 may classify the utterance as being spoken by the known user User 1 (User 1).

[0050] Still referring to block 240, in some implementations, the smart device automated assistant 120 can determine the account of the first user based on biometric data detected via one or more I / O components 130 of the smart device 110. For example, the smart device automated assistant 120 can determine the account of the first user based on matching the biometric data such as facial data, fingerprint data, etc. received at block 210 with the biometric authentication data stored in association with the account. Other biometric identifiers may additionally or alternatively be used.

[0051] Still referring to box 240, in some embodiments, the smart device automated assistant 120 can determine the account of the first user by processing audio data received by a microphone of the I / O component 130 and / or image data (e.g., a facial image) received by a camera of the I / O component 130 using a machine learning model to generate an embedding and comparing the generated embedding to embeddings associated with known accounts.

[0052] Still referring to box 240, in some embodiments, the request received in box 210 can be included in the speech spoken by the person, and the smart device automated assistant 120 can determine the account of the first user by receiving an image of the person from the camera of the I / O component 130 of the smart device 110 and determining that the speech was spoken by the first user based on the image.

[0053] Still referring to block 240, in other embodiments, the smart device automated assistant 120 of the smart device 110 can transmit voice data and / or other biometric data detected via one or more I / O components 130 of the smart device 110 to the cloud-based automated assistant component 180 of the computer server 170. The cloud-based automated assistant component 180 can then use the voice data and / or other biometric data to determine the account of the first user. For example, the cloud-based automated assistant component 180 can determine the account of the first user based on the voice data received at block 210 and transmitted to the cloud-based automated assistant component 180 at block 240, matching the voice verification data stored in association with the account. Other biometric identifiers, such as face, fingerprint, etc., can be used in addition or alternatively.

[0054] Still reference Figure 2, at block 250, the system may select a second device on which the messaging application corresponding to the request is available. In an embodiment, at block 250, the smart device automated assistant 120 of the smart device 110 (i.e., the first device) may select one of the client devices 140-1, ..., 140-n on which the messaging application corresponding to the request is available as the second device. In an embodiment, selecting the second device may be based on the account of the first user determined at block 240. In other embodiments, the smart device automated assistant 120 of the smart device 110 may utilize a cloud-based automated assistant component 180 of a computer server 170 to select the second device.

[0055] Still referring to block 250, in an embodiment, the smart device automated assistant 120 may query the cloud-based automated assistant component 180 to identify a client device among the client devices 140-1, ..., 140-n that is associated with the account of the first user determined at block 240. In other embodiments, the smart device automated assistant 120 may identify a client device among the client devices 140-1, ..., 140-n that is associated with the account of the first user determined at block 240 based on information stored on the smart device 110 and / or by querying one or more of the client devices 140-1, ..., 140-n.

[0056] Still referring to box 250, in an embodiment, the smart device automation assistant 120 selects the client device 140-1, ..., 140-n identified as being associated with the account of the first user as the second device. Optionally, the smart device automation assistant 120 can query the client devices 140-1, ..., 140-n identified as being associated with the account of the first user to confirm that the messaging application corresponding to the request is available. If multiple client devices 140-1, ..., 140-n are identified as being associated with the account of the first user, the smart device automation assistant 120 can select the confirmed client device 140-1, ..., 140-n that the messaging application corresponding to the request is available and has been provided in response to the query.

[0057] Still referring to box 250, if the smart device automation assistant 120 determines that the messaging application corresponding to the request is not available on any of the client devices 140-1,...,140-n identified as associated with the first user's account, the smart device automation assistant 120 can, for example, use a speaker or display of the I / O component 130 of the smart device 110 to return an error message to the first user, indicating that the message cannot be sent using the messaging application corresponding to the request.

[0058] In an example, at block 240, the smart device automated assistant 120 of the smart device 110 may determine that the account of the first user (i.e., the user who issued the request received at block 210) is Account X. Then, at block 250, the smart device automated assistant 120 may query the cloud-based automated assistant component 180 to request information identifying one or more client devices associated with Account X. The smart device automated assistant 120 may then receive information from the cloud-based automated assistant component 180 that identifies the client device 140-1 as being associated with Account X. In response, the smart device automated assistant 120 may select the client device 140-1 as a second device on which the messaging application is available.

[0059] Still reference Figure 2 , at block 260, the system may send a command to the second device, the command causing the second device to send a message from the first user to the second user using a messaging application on the second device. In an embodiment, at block 260, the smart device automated assistant 120 of the smart device 110 (i.e., the first device) may send a command to the client devices 140-1, ..., 140-n (i.e., the second device) selected at block 250, the command causing the client devices 140-1, ..., 140-n selected at block 250 to send a message from the first user to the second user using a messaging application 160 corresponding to the request on the client devices 140-1, ..., 140-n according to the request received at block 210. In other embodiments, the smart device automated assistant 120 of the smart device 110 may utilize the cloud-based automated assistant component 180 of the computer server 170 to send a command to the second device, the command causing the second device to send a message from the first user to the second user using a messaging application on the second device.

[0060] Still referring to block 260, in an embodiment, the smart device automation assistant 120 on the smart device 110 may send an action-intent request as a command to the client devices 140-1, ..., 140-n selected at block 250. In an embodiment, the action-intent request may optionally include a request to wake the client devices 140-1, ..., 140-n from a sleep state, or the smart device automation assistant 120 on the smart device 110 may optionally send a request to wake the client devices 140-1, ..., 140-n from a sleep state to the client devices 140-1, ..., 140-n selected at block 250 before sending the action-intent request. In an embodiment, the action-intent request may include the message from block 210 and the message recipient (e.g., the second user) and the identification of the messaging application 160 (specified in the request received at block 210 or determined at block 210 based on the request). The action-intent request may optionally include a message sender (e.g., the first user), which may specify a user account from which the message is to be sent.

[0061] Still referring to block 260, the client device automation assistant 150 of the client device 140-1, ..., 140-n selected at block 250 may receive an action-intent request sent by the smart device automation assistant 120 of the smart device 110. In response to receiving the action-intent request, the client device automation assistant 150 of the client device 140-1, ..., 140-n selected at block 250 may interpret the action-intent request to cause the client device automation assistant 150 to respond by treating the request received by the smart device 110 at block 210 as directly responded to by the client device automation assistant 150 of the client device 140-1, ..., 140-n selected at block 250. In an embodiment, in response to receiving the action-intent request, the client device automation assistant 150 of the client device 140-1, ..., 140-n selected at block 250 may cause a message to be sent from the first user to the second user according to the request received at block 210 using the messaging application 160 corresponding to the request.

[0062] Still referring to block 260, in embodiments, the client device automation assistant 150 of the client device 140-1, ..., 140-n selected at block 250 may send a message to the messaging application 160 corresponding to the request without displaying the client device automation assistant output on the I / O component 155 of the client device 140-1, ..., 140-n selected at block 250. Additionally, in embodiments, the messaging application 160 corresponding to the request may send a message without displaying the messaging application output on the I / O component 155 of the client device 140-1, ..., 140-n selected at block 250, which may save power on the client device 140-1, ..., 140-n.

[0063] Still referring to block 260, in an embodiment, the client device automation assistant 150 of the client device 140-1, ..., 140-n selected at block 250 may send the other action intent request to the messaging application 160 corresponding to the other action intent request. In other embodiments, the client device automation assistant 150 of the client device 140-1, ..., 140-n selected at block 250 may forward the action intent request received from the smart device automation assistant 120 of the smart device 110 to the messaging application 160 corresponding to the request. The action intent request sent or forwarded to the messaging application 160 corresponding to the request may include the message from block 210 and the message recipient (e.g., the second user). The action intent request may optionally include the message sender (e.g., the first user), which may specify the user account from which the message is sent. The action intent request may also optionally include an identification of the messaging application 160 corresponding to the request to be used to send the message. In an embodiment, in response to receiving the action intent request, the messaging application 160 may send the message to the message recipient according to the received action intent request.

[0064] Still referring to block 260, in an embodiment, after successfully sending the message to the message recipient according to the action intent request, the messaging application 160 may send a reply to the client device automation assistant 150 of the client device 140-1, ..., 140-n selected in block 250. The reply may represent a confirmation that the message was successfully sent to the message recipient. In an embodiment, in response to receiving the reply, the client device automation assistant 150 of the client device 140-1, ..., 140-n selected in block 250 may send another reply to the smart device automation assistant 120 of the smart device 110, the other reply confirming that the message was successfully sent to the message recipient. In other embodiments, in response to receiving the reply, the client device automation assistant 150 of the client device 140-1, ..., 140-n selected in block 250 may forward the reply received from the messaging application 160 to the smart device automation assistant 120 of the smart device 110.

[0065] Still reference Figure 2 , at block 270, the system may receive a reply from the second device indicating that the message is sent to the second user. In an embodiment, at block 270, the smart device automation assistant 120 of the smart device 110 (i.e., the first device) may receive a reply from the client device automation assistant 150 of the client device 140-1, ..., 140-n (i.e., the second device) selected at block 250, the reply indicating that the message is sent to the second user. In an embodiment, the smart device automation assistant 120 of the smart device 110 receives the reply sent at block 260 by the client device automation assistant 150 of the client device 140-1, ..., 140-n selected at block 250. In other embodiments, the smart device automation assistant 120 of the smart device 110 receives the reply from the messaging application 160, which is forwarded at block 260 by the client device automation assistant 150 of the client device 140-1, ..., 140-n selected at block 250.

[0066] Still reference Figure 2 In response to receiving the reply at block 270 , the system may output an audio or visual indication at the first device that the message was sent at block 280 . In an embodiment, in response to receiving the reply at block 270 , the smart device automated assistant 120 of the smart device 110 (i.e., the first device) may output an audio or visual indication that the message was sent via an I / O component 130 of the smart device 110 , such as a speaker and / or a touch screen display of the I / O component 130 .

[0067] Still referring to block 280, in an example, in response to receiving the reply, the smart device automated assistant 120 of the smart device 110 may use a text-to-speech processor to convert the text message “Your message has been sent to UserX” into generated speech, which may then be presented via a speaker of the I / O component 130. In another example, in response to receiving the reply, the smart device automated assistant 120 of the smart device 110 may cause the text message “Your message has been sent to UserX” to be presented via a touch screen display of the I / O component 130.

[0068] In other embodiments, the smart device 110 may include additional applications in addition to the messaging application 135, and the client devices 140-1, ..., 140-n may include additional applications in addition to the messaging application 160. The request received by the smart device automation assistant 120 at block 210 may be a request to perform an action (e.g., turn on a smart light, or log a 10k run) using a specific application (e.g., a smart light application or a run logging application). At block 220, the smart device automation assistant 120 may determine that the specific application corresponding to the request is not available on the smart device 110 (e.g., the specific application is not included in the messaging application 135 or the additional applications on the smart device 110 and cannot be remotely accessed by the smart device 110 using an API or a web service). At block 250, the smart device automation assistant 120 may select a client device 140-1, ..., 140-n on which the specific application is available, and at block 260, the smart device automation assistant 120 may send a command to the client device 140-1, ..., 140-n that causes the client device 140-1, ..., 140-n to perform an action using the specific application. At block 270, the smart device automation assistant 120 may receive a response indicating that the action is performed by the specific application on the client device 140-1, ..., 140-n, and at block 280, in response to receiving the response, the smart device automation assistant 120 may output an audio or visual indication that the action is performed using the specific application.

[0069] Figure 3140-n, and / or a computer server 170. In addition, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0070] At block 310, the system may receive, via a first automated assistant application running on a first device associated with a first user, a message received by a messaging application running on the first device. In embodiments, at block 310, client device automated assistant 150 (i.e., first automated assistant application) of client devices 140-1, ..., 140-n (i.e., first devices associated with the first user) may receive a message received by messaging application 160 running on client devices 140-1, ..., 140-n. In embodiments, messaging application 160 may utilize end-to-end encryption to send and receive messages.

[0071] Still reference Figure 3 , at block 320, the system may determine, via the first automated assistant application, a second device associated with the first user. In an embodiment, the second device may be running the second automated assistant application and the messaging application may not be available on the second device. In an embodiment, at block 320, the client device automated assistant 150 (i.e., the first automated assistant application) of the client devices 140-1, ..., 140-n may determine that the smart device 110 is a second device associated with the first user. In an embodiment, the smart device 110 may be running the smart device automated assistant 120 (i.e., the second automated assistant application), and the messaging application 160 may not be available on the smart device 110 (e.g., the messaging application 160 does not exist in the messaging application 135 on the smart device 110, and the messaging application 160 cannot be accessed from the smart device 110).

[0072] Still reference Figure 3, at block 330, the system may send a message to a second automated assistant application running on a second device via the first automated assistant application. In an embodiment, at block 330, client device automated assistant 150 (i.e., first automated assistant application) of client devices 140-1, ..., 140-n may send the message received at block 310 to smart device automated assistant 120 (i.e., second automated assistant application) running on smart device 110 (i.e., second device). In an embodiment, client device automated assistant 150 may generate an action intent request including the message and send the action intent request to smart device automated assistant 120. In an embodiment, the action intent request includes a command to cause smart device automated assistant 120 running on smart device 110 (e.g., using I / O component 130) to present the message at smart device 110.

[0073] Still reference Figure 3 At block 340, the system may receive, via the first automated assistant application running on the first device, a command to send a reply message using a messaging application from a second automated assistant application running on the second device. In an embodiment, at block 340, client device automated assistant 150 (i.e., first automated assistant application) running on client devices 140-1, ..., 140-n (i.e., first device) may receive a command to send a reply message using a messaging application 160 from smart device automated assistant 120 (i.e., second automated assistant application) running on smart device 110 (i.e., second device). In an embodiment, client device automated assistant 150 may receive an action intent request as a command, the action intent request including a reply message and optionally including an identification of a message recipient and messaging application 160.

[0074] Still reference Figure 3 At block 350, the system may call the messaging application via the first automated assistant application to cause the messaging application to send a reply message. In an embodiment, at block 350, the client device automated assistant 150 (i.e., the first automated assistant application) running on the client device 140-1, ..., 140-n (i.e., the first device) may call the messaging application 160 to cause the messaging application 160 to send a reply message according to the command received at block 340.

[0075] Still reference Figure 3, at block 360, the system may send a confirmation indicating that the reply message was sent to the second automated assistant application running on the second device via the first automated assistant application running on the first device. In an embodiment, at block 360, the client device automated assistant 150 (i.e., the first automated assistant application) running on the client devices 140-1, ..., 140-n (i.e., the first device) may send a confirmation indicating that the reply message was sent at block 350 to the smart device automated assistant 120 (i.e., the second automated assistant application) running on the smart device 110 (i.e., the second device). In an embodiment, in response to receiving the confirmation, the smart device automated assistant 120 may output an audio or visual indication that the reply was sent via an I / O component 130 of the smart device 110, such as a speaker and / or a touch screen display of the I / O component 130.

[0076] Figure 4 1 is a flowchart illustrating an example method 400 of sending a message using a messaging application 160 on a client device 140-1, ..., 140-n, such as a mobile phone computing device, using a smart device 110, such as a stand-alone assistant-centric interactive speaker, a stand-alone assistant-centric interactive display with a speaker, a smart appliance such as a smart TV, and / or a wearable device of a user, in accordance with embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as smart device 110, client devices 140-1, ..., 140-n, and / or one or more components of a computer server 170. Furthermore, while the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0077] At block 410 , the system may receive a request from a first user to send a message to a second user. In an embodiment, at block 410 , the smart device automated assistant 120 of the smart device 110 may receive a request from a first user to send a message to a second user.

[0078] Still reference Figure 4, at block 420, the system may determine whether the messaging application corresponding to the request is available on the system. In an embodiment, at block 420, the smart device automation assistant 120 of the smart device 110 may determine whether the messaging application corresponding to the request received at block 410 is included in the messaging application 135 on the smart device 110 or is remotely accessible from the smart device 110. If the smart device automation assistant 120 determines that the messaging application corresponding to the request received at block 410 is included in the messaging application 135 on the smart device 110 or is remotely accessible from the smart device 110, the flow proceeds to block 430. On the other hand, if the smart device automation assistant 120 determines that the messaging application corresponding to the request received at block 410 is not included in the messaging application 135 on the smart device 110 or is remotely accessible from the smart device 110, the flow proceeds to block 440.

[0079] Still reference Figure 4 In response to determining that the messaging application corresponding to the request is available on the system, the system may send a message from the first user to the second user using the messaging application on the system at block 430. In an embodiment, in response to determining at block 420 that the messaging application corresponding to the request received at block 410 is included in the messaging application 135 on the smart device 110 or is remotely accessible from the smart device 110, the smart device automated assistant 120 may cause the messaging application 135 corresponding to the request to send the message received at block 410 from the first user to the second user.

[0080] Still reference Figure 4 In response to determining that the messaging application corresponding to the request is not available on the system, at block 440, the system may select an external device on which the messaging application corresponding to the request is available. In an embodiment, at block 440, in response to determining at block 420 that the situation is not that the messaging application corresponding to the request received at block 410 is included in the messaging application 135 on the smart device 110 or is remotely accessible from the smart device 110, the smart device automated assistant 120 may select one of the client devices 140-1, ..., 140-n on which the messaging application corresponding to the request is available (e.g., the messaging application corresponding to the request is included in the messaging application 160 on the client devices 140-1, ..., 140-n).

[0081] Still reference Figure 4, at block 450, the system may send a command to the external device, the command causing the external device to send a message from the first user to the second user using a messaging application on the external device. In an embodiment, at block 450, the smart device automated assistant 120 of the smart device 110 uses the messaging application 160 on the client device 140-1, ..., 140-n corresponding to the request to send a command to the client device 140-1, ..., 140-n selected at block 440, the command causing the client device 140-1, ..., 140-n to send the message received from the first user at block 410 to the second user.

[0082] Figure 5 5 is a block diagram of an example computing device 510 that can optionally be used to perform one or more aspects of the techniques described herein. The computing device 510 generally includes at least one processor 514 that communicates with a plurality of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 including, for example, a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow for user interaction with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0083] User interface input devices 522 may include: a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into computing device 510 or over a communication network.

[0084] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computing device 510 to a user or another machine or computing device.

[0085] The storage subsystem 524 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include a computer program for executing Figure 2 , 3and selected aspects of the methods of 4 and implementations thereof Figure 1 The logic of the various components described in .

[0086] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory subsystem 525 included in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storage of instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 can provide permanent storage for program and data files, and may include a hard drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible to the processor 514.

[0087] The bus subsystem 512 provides a mechanism for enabling the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0088] Computing device 510 may be of various types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of computing device 510 depicted in FIG. 5 is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of computing device 510 may have more Figure 5 The computing devices depicted may have greater or fewer components.

[0089] Embodiments may address the problems of low reliability and overuse of computing and / or network resources by providing methods and systems for sending messages using messaging applications. In particular, some embodiments may improve the functionality of a computer by providing methods and systems for sending messages using a messaging application on a client device such as a smartphone using a smart device such as a standalone assistant-centric interactive speaker and / or a standalone assistant-centric interactive display with a speaker. Thus, through the use of rules for improving computer-related technology, embodiments allow a computer to perform functions that were not previously possible for a computer to perform. Furthermore, embodiments use technologies that are by definition rooted in computer technology (e.g., end-to-end encryption, STT processors, NLU, TTS processors, etc.).

[0090] In situations where the systems described herein collect personal information about a user or may utilize personal information, the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, certain data may be processed in one or more ways before it is stored or used to remove personally identifiable information. For example, the user's identity may be processed so that no personally identifiable information about the user can be determined, or the user's geographic location may be summarized (such as to a city, zip code, or state level) in the case of obtaining geographic location information so that the user's specific geographic location cannot be determined. Thus, the user may have control over how information about the user is collected and / or used.

[0091] Although several embodiments have been described and illustrated herein, various other devices and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications to which this teaching is used. Those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein using no more than routine experiments. Therefore, it should be understood that the foregoing embodiments are presented only as examples, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: receiving, by the first device, a request from the first user to send a message to the second user; determining that a messaging application corresponding to the request is not available on the first device; as well as In response to determining that the messaging application corresponding to the request is not available on the first device: determining, by the first device, an account of the first user based on the context information; selecting, based on the account of the first user, a second device on which the messaging application corresponding to the request is available; as well as A command is sent to the second device causing the second device to send the message from the first user to the second user using the messaging application on the second device.

2. The method according to claim 1, wherein: The request is included in the utterance, and Determining the account of the first user includes determining that the utterance was spoken by the first user based on a speaker identification feature of the utterance.

3. The method according to claim 1, wherein: The request is included in an utterance spoken by a person, and Determining the account of the first user includes: receiving an image of the person from a camera of the first device, and It is determined based on the image that the utterance was spoken by the first user.

4. The method according to claim 1, wherein: The first device is a non-smartphone device, and The second device is a smartphone device of the first user.

5. The method according to claim 1, wherein: the receiving, the determining that the messaging application corresponding to the request is unavailable, the selecting, and the sending are performed by a first automated assistant application running on the first device; as well as The command is interpreted by a second automated assistant application running on the second device to cause the second automated assistant application to invoke the messaging application running on the second device to cause the messaging application to send the message from the first user to the second user.

6. A method implemented by one or more processors, the method comprising: receiving, by the first device, a request from the first user to send a message to the second user; determining that a messaging application corresponding to the request is not available on the first device; as well as In response to determining that the messaging application corresponding to the request is not available on the first device: selecting a second device on which the messaging application corresponding to the request is available; as well as sending a command to the second device causing the second device to send the message from the first user to the second user using the messaging application on the second device, in: The request specifies the messaging application, and The messaging application sends the message from the second device to a third device associated with the second user using end-to-end encryption.

7. The method according to claim 6, wherein: The first device is a non-smartphone device, and The second device is a smartphone device of the first user.

8. The method according to claim 6, wherein: The receiving, the determining, the selecting, and the sending are performed by a first automated assistant application running on the first device; as well as The command is interpreted by a second automated assistant application running on the second device to cause the second automated assistant application to invoke the messaging application running on the second device to cause the messaging application to send the message from the first user to the second user.

9. The method according to claim 6, wherein: The second automated assistant application invokes the messaging application without displaying second automated assistant application output, and The messaging application sends the message without displaying messaging application output.

10. A method implemented by one or more processors, the method comprising: receiving, by the first device, a request from the first user to send a message to the second user; determining that a messaging application corresponding to the request is not available on the first device; as well as In response to determining that the messaging application corresponding to the request is not available on the first device: selecting a second device on which the messaging application corresponding to the request is available; as well as sending a command to the second device causing the second device to send the message from the first user to the second user using the messaging application on the second device, wherein the command is interpreted by a second automated assistant application running on the second device such that the second automated assistant application responds by treating the request as if it were directly received by the second automated assistant application running on the second device.

11. A method implemented by one or more processors, the method comprising: receiving, by the first device, a request from the first user to send a message to the second user; determining that a messaging application corresponding to the request is not available on the first device; as well as In response to determining that the messaging application corresponding to the request is not available on the first device: selecting a second device on which the messaging application corresponding to the request is available; as well as sending a command to the second device causing the second device to send the message from the first user to the second user using the messaging application on the second device, Wherein, the determining, the selecting and the sending are performed by a computer server external to the first device and the second device.

12. A method implemented by one or more processors, the method comprising: receiving, by a first automated assistant application executing on a first device associated with a first user, a message received by a messaging application executing on the first device; determining, by the first automated assistant application, a second device associated with the first user, wherein the second device is running a second automated assistant application and the messaging application is not available on the second device; Sending the message by the first automated assistant application running on the first device to the second automated assistant application running on the second device; receiving, by the first automated assistant application running on the first device, from the second automated assistant application running on the second device, a command to send a reply message using the messaging application; and The messaging application is called by the first automated assistant application to cause the messaging application to send the reply message.

13. The method according to claim 12, further comprising: An acknowledgement is sent by the first automated assistant application running on the first device to the second automated assistant application running on the second device indicating that the reply message was sent.

14. The method according to claim 12, wherein: The first device is a smartphone device, and the second device is a non-smartphone device.

15. A method implemented by one or more processors, the method comprising: receiving, by a first automated assistant application executing on a first device associated with a first user, a message received by a messaging application executing on the first device; determining, by the first automated assistant application, a second device associated with the first user, wherein the second device is running a second automated assistant application and the messaging application is not available on the second device; and Sending, by the first automated assistant application, the message to the second automated assistant application running on the second device; Wherein sending the message includes sending a command to cause the second automated assistant application running on the second device to present the message at the second device.

16. A method implemented by one or more processors, the method comprising: Receiving, by a first device, a request to send a first message, the request being for sending the first message corresponding to a first messaging application; determining, by the first device, that the first device cannot send the first message based on determining that the first messaging application is not available on the first device; In response to determining that the first device cannot send the first message: selecting, by the first device, a second device based on the request to send the first message; as well as The first device sends a command to the second device to enable the second device to send the first message; receiving, by the first device, a request to send a second message, the request being for sending the second message corresponding to a second messaging application; determining, by the first device, that the first device is capable of sending the second message based on determining that the second messaging application is available on the first device; as well as In response to determining that the first device is capable of sending the second message, the second message is sent by the first device.

17. The method according to claim 16, wherein: Selecting the second device is further based on determining that the first messaging application is available on the second device.

18. The method according to claim 16, wherein: Determining that the first messaging application is not available on the first device includes determining that the first messaging application is not present on the first device and determining that the first messaging application is not accessible from the first device using an application programming interface.

19. The method of claim 16, wherein: The receiving the request to send the first message, the determining that the first device cannot send the first message, the selecting the second device, and the sending the command to cause the second device to send the first message are performed by a first automated assistant application running on the first device; as well as The command is interpreted by a second automated assistant application running on the second device to cause the second automated assistant application to invoke a messaging application running on the second device to cause the messaging application to send the first message.

20. The method of claim 19, wherein: The second automated assistant application invokes the messaging application without displaying second automated assistant application output, and The messaging application sends the first message without displaying messaging application output.

21. The method of claim 16, further comprising: receiving a reply from the second device indicating that the first message was sent; as well as In response to receiving the reply, the first device outputs an audio or visual indication that the message was sent.

22. A method implemented by one or more processors, the method comprising: receiving an indication from the first user of a request to send a message to the second user; determining whether the system has access to a messaging application corresponding to the request; In response to determining that the system has access to a messaging application corresponding to the request: sending the message from the first user to the second user using the messaging application corresponding to the request; as well as In response to determining that the system does not have access to a messaging application corresponding to the request: selecting an external device on which the messaging application corresponding to the request is available; as well as A command is sent to the external device, which, when received by the external device, causes the external device to send the message from the first user to the second user using the messaging application corresponding to the request.

23. The method according to claim 22, wherein: The system is executed locally on a first client device of the first user, and wherein the external device is a second client device of the first user.

24. The method according to claim 23, wherein: The messaging application is not available at the first client device of the first user, and wherein the messaging application is available at the second client device of the first user.

25. The method according to claim 22, wherein: The system is remotely executed at a remote server remote from a first client device of the first user, and wherein the external device is the first client device of the first user.

26. The method according to claim 25, wherein: The messaging application is not available at the remote server remote from the first client device of the first user, and wherein the messaging application is available at the first client device of the first user.

27. The method according to claim 25, wherein: Selecting the external device on which the messaging application corresponding to the request is available comprises: The first client device of the user is selected as the external device from among at least the first client device of the first user and a second client device of the first user.

28. The method according to claim 27, wherein: Selecting the first client device of the user as the external device from among at least the first client device of the first user and the second client device of the first user is based on determining that the messaging application is available at the first client device of the first user and not available at the second client device of the first user.

29. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 28.

30. A computer-readable storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 28.

31. A client device comprising one or more processors for executing the method according to any one of claims 1 to 28.

Images