Methods and systems for generating selectable suggestions using temporary machine learning models

Abstract

The implementations set forth herein relate to selectively relying on additional suggestion model(s) when generating selectable suggestions, while also maintaining access to a default suggestion model. Selectable suggestions can be generated using one or more additional multi-domain machine learning (ML) models that are optional for a client application, regardless of whether the default suggestion model is still useful for generating suitable suggestions. In some implementations, as a client application adopts various additional multi-domain ML models, a particular model can be identified as improving suggestions of the client application based at least on user feedback and / or other data. The particular model can then be selected to replace and / or supplement the default suggestion model in order to provide more accurate suggestions that, when selected, initialize actions that can save time and computing resources.

Description

Background Technology

[0001] Humans can engage in human-computer dialogue using interactive software applications referred to herein as “automated assistants” (also known as “digital agents,” “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “conversational agents,” etc.). For example, humans (who may be referred to as “users” when interacting with automated assistants) can provide commands and / or requests using spoken natural language input (i.e., utterances) and / or by providing text-based (e.g., typed) natural language input, which in some cases can be converted to text and then processed.

[0002] In some contexts, users may be presented with selectable suggestions to enable the computing device to perform specific actions, such as invoking an automation assistant to initiate an action. These suggestions can be based on application content rendered on the computing device when the user, for example, sends a message, browses the internet, and / or otherwise interacts with the application. While some applications can include native modules or application programming interfaces (APIs) for generating such suggestions, the processes used to generate these suggestions may be inadequate. For example, when processing using previously employed suggestion models, corresponding text between users may not result in any suitable suggestions appearing on the computing device's interface (e.g., on the application's interface and / or the automation assistant's interface).

[0003] Such situations can cause users to miss opportunities to initialize automation assistants or other applications to complete time-, energy-, and other resource-saving tasks. Furthermore, while the default recommendation model may be advantageous in certain scenarios, updates to the default recommendation model (or replacement of the default recommendation model with an alternative model) may not be pushed to user devices frequently. This lack of rapid model updates and / or model replacements hinders the identification of multiple time- and / or energy-saving action recommendations. These recommendations could be determined using multiple updated recommendation models instead of multiple default recommendation models. Multiple updated recommendation models can be more robust than multiple default recommendation models (e.g., providing greater coverage). However, while such updated recommendation models may be more robust, they may lack accuracy and could disrupt other services if deployed too quickly (e.g., before sufficient training and / or validation). This lack of accuracy and service disruption when rendering and / or selecting multiple inaccurate recommendations can also lead to a waste of various device and / or network resources. Summary of the Invention

[0004] The implementation described herein relates to selectively relying on additional suggestion models when generating selectable suggestions related to application content generated by a client application. The client application may otherwise employ suggestion processes (e.g., rule-based methods, application suggestion models, and / or other processes) to render selectable suggestions to the user. For example, selectable suggestions may be selectable graphical user interface elements that, when selected, cause another application to initiate an action. For instance, the client application could be a messaging application that renders selectable GUI elements that, when selected, cause an automation assistant to initiate an action (e.g., scheduling an event). With prior user permission, this action may be based on conversation content between users within the messaging application. To identify specific actions to suggest, the messaging application or a separate application (e.g., an automation assistant) can use an application suggestion model to process certain application content; this application suggestion model may be a default model associated with the automation assistant. In some cases, when an automation assistant relies on an untested suggestion model, it can lead to latency and mismanagement of computational resources. To ensure efficient conversion between suggestion models, the default suggestion model can remain available when generating suggestions and, under certain conditions, can simultaneously and / or additionally rely on other additional suggestion models.

[0005] For example, a user can use a messaging application to communicate with another user about a specific event. The specific event can be represented by application content rendered at the messaging application's interface. When processing application content, a first suggestion model can be relied upon by default to facilitate the rendering of selectable suggestion elements for the user. In some implementations, processing application content using the first suggestion model can result in, for example, suggesting one or more automated assistant commands to be invoked by the user. However, when processing does or does not result in any suggestion being identified and / or any suggestion having a threshold probability and / or threshold score being identified, a second suggestion model can be used to help identify the suggestions to be rendered.

[0006] In some implementations, a second suggestion model can be used to further process application content to generate one or more selectable suggestions for the user. For example, an automation assistant can employ a second suggestion model and / or one or more additional suggestion models to help identify one or more assistant actions that may be useful to the user and / or otherwise save computing resources. When the automation assistant identifies an assistant action or other content that the messaging application wants to suggest, the automation assistant can deliver the assistant action or other content to the messaging application. In response, the messaging application can choose to render, for example, a selectable suggestion element that identifies the assistant action identified using the second suggestion model. In this way, the messaging application can continue to rely on suggestions generated using a default process (e.g., using a first suggestion model) to render suggestions, but can also selectively switch to relying on other additional suggestion models available via the automation assistant or other applications.

[0007] In some implementations, the task of the first application may be to select a suitable supplementary suggestion model from multiple different supplementary suggestion models. The first application can be provided on a client computing device along with a second application, which can provide an interface through which the user can receive suggestions. The first application can be updated via a network connection to allow the client application to access the supplementary suggestion model that has recently become available. The second application can optionally be updated via a network connection and can attempt to generate suggestions for the user regardless of whether the first application is updated. For example, the second application can use an application programming interface (API) to interact with the first application to receive suggestion data from it. Initially, the second application can execute functions such as `assistant_suggestion(application_content(), application_suggestion_model)` to receive suggestions based on application content and application suggestion model. The slot "application_content()" can reference the application content generated by the second application, and the slot "application_suggestion_model" can identify the model that the first application will use when processing the application content. Even after the first application is updated to provide access to one or more different additional suggestion models, the first application is still able to support "application_suggestion_model".

[0008] For example, when the updated first application receives a request to facilitate the execution of a function from the second application, the updated first application can process "application_content()" to facilitate the provision of suggestion data to the second application. In some implementations, when processing "application_content()" with "application_suggestion_model" fails to result in identifying a suggestion, the second application can choose not to render the suggestion based on the application content. However, the first application can optionally use "application_content()" along with another model to continue identifying suggestions. For example, based on the application content, the first application can identify one or more additional suggestion models to be used when generating suggestions from the application content. In some implementations, the additional suggestion models can be regarded as "experimental models" by the entity providing the first application, so that the entity can avoid assigning the additional suggestion models as the default models without prior testing. Therefore, by allowing the first application to use additional suggestion models to generate suggestion data, the additional suggestion models can be tested without significantly affecting the operability of the second application, which otherwise depends on different application suggestion models. In other words, the entity providing the model for the second application can test and / or train other suggestion models without completely revoking access to other suitable application suggestion models during testing.

[0009] Once the first application has identified a suitable supplemental suggestion model to test, it can use the supplemental suggestion model and application content to generate suggestion data—even though the second application does not explicitly request the first application to adopt the supplemental suggestion model. The second application can then determine, based on the suggestion data, whether to render one or more selectable suggestion elements. In some implementations, user feedback (e.g., input and / or other user engagement) can be identified when the second application uses a specific supplemental suggestion model to render selectable suggestion elements. With prior user permission, user feedback can be represented as feedback data accessible to the first application and / or the entity providing the first application. That specific supplemental suggestion model can then be further trained based on feedback data from various devices and / or user interactions. This can improve the accuracy of user suggestions rendered by a specific application using the supplemental suggestion model.

[0010] The above description serves as an overview of some embodiments of this disclosure. These and other embodiments are further described in more detail below.

[0011] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., multiple central processing units (CPUs), multiple graphics processing units (GPUs), and / or multiple tensor processing units (TPUs)) to perform methods, such as one or more of the methods described above and / or elsewhere herein. Other embodiments may include a system of one or more computers including one or more processors operable to execute the stored instructions to perform methods, such as one or more of the methods described above and / or elsewhere herein.

[0012] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. Attached Figure Description

[0013] Figure 1A , Figure 1B and Figure 1C The illustration shows a view of a user interacting with a computing device that can use one or more suggestion models to perform an experimental process for generating suggestions, while also supporting one or more existing default suggestion models.

[0014] Figure 2 The diagram illustrates a system that selectively relies on (multiple) additional suggestion models when generating selectable suggestions, while simultaneously maintaining access to the default suggestion model.

[0015] Figure 3A and Figure 3B The diagram illustrates a method for retaining the default suggested model while allowing other models to be used to determine the accuracy of other models.

[0016] Figure 4 This is a block diagram of an example computer system. Detailed Implementation

[0017] Figure 1A , Figure 1B and Figure 1C The illustrations depict views 100, 140, and 160 of a user 110 interacting with a computing device 112 capable of performing an experimental process for generating recommendations using one or more recommendation models—while also supporting one or more existing default recommendation models. For example, Figure 1AThe illustration shows a view 100 of a user 110 interacting with a computing device 112 to access a messaging application 102. While interacting with the computing device 112, the messaging application 102 can render application content, such as natural language content in group messages between the user 110 and one or more other people, as shown in state 108. The user 110 can use their hand 114 to input natural language content into fields 106 of the messaging application 102. One or more portions of the content rendered at interface 116 (e.g., text displayed at interface 116) can be considered application content, which can form the basis for one or more suggestions that can be rendered at interface 116 of the computing device 112. Suggestions can be rendered as selectable suggestion elements, which, when selected, can cause one or more applications to run to facilitate actions related to the application content for the user. For example, the selectable suggestion element can include content that can: be provided to the automation assistant as a verbal command, be incorporated into the application's input fields, be transmitted to another device and / or application, and / or otherwise be used to assist the user 110 and / or device or module in performing one or more operations.

[0018] In some implementations, messaging application 102 can request suggestion data 124 by sending suggestion request 122 to an additional application 120 accessible via computing device 112. In response to receiving suggestion request 122, additional application 120 can access a first suggestion model 126-A of one or more different suggestion models 126. In some implementations, additional application 120 and / or messaging application 102 can identify the first suggestion model 126-A based on a first suggestion model 126-A previously used in processing prior application content and / or generating one or more suggestions for messaging application 102. Additionally or alternatively, additional application 120 and / or messaging application 102 can identify the first suggestion model 126-A based on the first suggestion model 126-A being assigned as the default model used in processing application content and generating one or more suggestions for messaging application 102.

[0019] In some implementations, the add-on application 120 may include a suggestion engine 118 that employs one or more suggestion models when generating suggestion data 124 for the messaging application 102. Additionally or alternatively, the suggestion data 124 may be used by the messaging application 102 and / or the add-on application 120 to render one or more selectable suggestion elements at an interface on the computing device 112. In some implementations, the add-on application 120 may use a first suggestion model 126-A to generate the suggestion data 124, but determines that the suggestion data 124 does not identify suggestions suitable for rendering to the user 110. For example, a probability of not meeting a threshold may be assigned to one or more suggestions, the threshold being established based on the interactions of one or more users with various selectable suggestion elements. Additionally or alternatively, the add-on application 120 may attempt to generate suggestion data using the first suggestion model 126-A, but the process of generating the suggestion data may not result in identifiable suggestions.

[0020] When add-on application 120 fails to identify one or more suggestions, or fails to identify one or more suggestions suitable for rendering at interface 116, add-on application 120 and / or messaging application 102 can employ different suggestion models to identify other suggestions (e.g., corresponding to one or more other actions). In some implementations, add-on application 120 can use (multiple) different suggestion models to generate suggestion data without explicit requests from messaging application 102 for: (i) using different suggestion models and / or (ii) attempting to generate other suggestion data. Additionally or alternatively, when suggestion data 124 is determined to be suitable for rendering to user 110, add-on application 120 can generate suggestion data using application content.

[0021] like Figure 1B Provided in view 140, the additional application 120 can access application content 146, which can characterize Figure 1A and / or Figure 1B The content of interface 116 provided in the document. Additional application 120 can use the second proposed model 126-B to process application content 146. The second proposed model 126-B can be different from the first proposed model 126-A and can optionally be trained using at least some training data not used to train the first proposed model 126-A. Additionally or alternatively, the second proposed model 126-B can be a machine learning model of a different type than the machine learning model of the first proposed model 126-A. For example, the second proposed model 126-B can be a multi-domain neural network model, while the first proposed model 126-A may not be a multi-domain neural network model.

[0022] The add-on application 120 can generate suggestion data 148 based on processing application content 146 using suggestion engine 118 and second suggestion model 126-B. Suggestion data 148 can be provided by the add-on application 120 to messaging application 102, and messaging application 102 can render one or more selectable suggestion elements 144 based on suggestion data 148. Additionally or alternatively, the add-on application 120 can generate suggestion data 148 to render one or more selectable suggestion elements 144 based on suggestion data 148. The user 110 can select the selectable suggestion element 144 (as shown in state 150) by: using their hand 114; providing verbal utterances (e.g., “Assistant, tell me the delivery options nearby”); and / or, providing another input within a threshold time period instructing the user 110 to initiate an action associated with the selectable suggestion element 144.

[0023] In some implementations, a first suggestion model 126-A can remain available to aid in suggestion generation, while a second suggestion model 126-B can also be used to aid in suggestion generation. Other suggestion models 126 can also be available while the first suggestion model 126-A remains available, at least for use in processing data to facilitate the generation of selectable suggestions and / or other content to assist one or more users. For example, the second suggestion model 126-B can be used by computing device 112 as a test to determine the accuracy and / or suitability of suggestions generated using the second suggestion model 126-B. While this test is in progress, the additional application 120 can utilize the first suggestion model 126-A, which can be a suggestion model that has been reliably used by the additional application 120 to provide appropriate suggestion data prior to the test. In this way, testing various models will not interrupt the reliable process of suggestion generation—at least until the tested model has been historically used to generate reliable suggestion data. For example, suggestion data can be considered reliable when one or more probabilities and / or scores of a particular suggestion characterized by the suggestion data meet a threshold. In some implementations, when it is determined that the second suggestion model 126-B is useful for generating suggestions at least as suitable as those generated using the first suggestion model 126-A, the second suggestion model 126-B can be assigned as an alternative to the first suggestion model 126-A. For example, the degree of relevance between suggestions generated using the second suggestion model 126-B and another suggestion generated using the first suggestion model 126-A can be determined. When the relevance meets a threshold, the second suggestion model 126-B can be assigned as the default suggestion model and / or replace the first suggestion model 126-A. Additionally or alternatively, when the relevance fails to meet the threshold, the second suggestion model 126-B can be further trained and / or further evaluated as the second suggestion model 126-B is continuously tested.

[0024] In some implementations, when user 110 selects the selectable suggestion element 144, feedback data 162 can be generated by messaging application 102 and / or supplementary application 120, and processed by supplementary application 120. Feedback data 162 can be used to further refine the accuracy of the suggestion model and / or further train the suggestion model. For example, in response to user 110 selecting the selectable suggestion element 144 within a threshold time period, supplementary application 120 (e.g., an automation assistant) can be invoked and initiate an internet search for "nearby delivery options". When supplementary application 120 identifies a search result to suggest to user 110, supplementary application 120 can populate field 106 of messaging application 102 with links leading to the search result.

[0025] Feedback data 162 provided from messaging application 102 to additional application 120 can characterize the user 110's selection of suggestions(s). Feedback data 162 can then be processed by additional application 120 and / or a separate application or device to further train a second suggestion model 126-B. In some embodiments, the second suggestion model 126-B is trained using training data 164 generated by additional application 120 based on feedback data 162. Additionally or alternatively, the second suggestion model 126-B can be further trained so that suggestions generated by the second suggestion model 126-B will correspond to a more frequent probability of satisfying a threshold. In some embodiments, the second suggestion model 126-B can be accessed via a remote computing device that, with the user's prior permission, trains it using feedback data from various client devices and / or various users. When it is determined after testing that the second suggestion model 126-B is sufficiently trained, the second suggestion model 126-B can be assigned to replace the first suggestion model 126-A. In other words, when the second proposal model 126-B is fully trained, it can be assigned as the "default" model to be used when generating proposals—instead of the first proposal model 126-A as the "default" model.

[0026] Figure 2The illustration shows system 200, which selectively relies on (multiple) additional suggestion models when generating selectable suggestions, while maintaining access to and / or adopting a default suggestion model. Automation assistant 204 can operate as part of an assistant application provided on one or more computing devices, such as computing device 202 and / or server devices. Users can interact with automation assistant 204 via (multiple) assistant interfaces 220, which can be microphones, cameras, touchscreen displays, user interfaces, and / or any other device that provides an interface between the user and the application. For example, a user can initialize automation assistant 204 by providing verbal, textual, and / or graphical input to assistant interface 220 to cause automation assistant 204 to initialize one or more actions (e.g., providing data, controlling peripheral devices, accessing agents, generating input and / or output, etc.). Alternatively, automation assistant 204 can be initialized based on the processing of contextual data 236 using one or more trained machine learning models. Context data 236 can characterize one or more features of the environment in which the automation assistant 204 can access, and / or predict one or more features of the user who intends to interact with the automation assistant 204.

[0027] Computing device 202 may include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures to allow a user to control application 234 of computing device 202 or another device via the touch interface. In some embodiments, computing device 202 may not have a display device, thereby providing audible user interface output instead of graphical user interface output. Furthermore, computing device 202 may provide a user interface for receiving spoken natural language input from a user, such as a microphone. In some embodiments, computing device 202 may include a touch interface and may not have a camera, but may optionally include one or more other sensors.

[0028] Computing device 202 and / or other third-party client devices can communicate with the server device via a network such as the Internet. Additionally, computing device 202 and any other computing devices can communicate with each other via a local area network (LAN) such as a Wi-Fi network. Computing device 202 can offload computing tasks to the server device to save computing resources at computing device 202. For example, the server device can host automation assistant 204, and / or computing device 202 can send input received at one or more assistant interfaces 220 to the server device. However, in some embodiments, automation assistant 204 can be hosted at computing device 202 and can execute various processes associated with the operation of automation assistant at computing device 202.

[0029] In various implementations, all or fewer aspects of the automation assistant 204 can be implemented on the computing device 202. In some of these implementations, aspects of the automation assistant 204 are implemented via the computing device 202 and can interface with a server device, which can implement other aspects of the automation assistant 204. The server device can optionally serve multiple users and their associated assistant applications via multithreading. In implementations where all or fewer aspects of the automation assistant 204 are implemented via the computing device 202, the automation assistant 204 can be an application separate from the operating system of the computing device 202 (e.g., installed "on top" of the operating system) – or can alternatively be implemented directly by the operating system of the computing device 202 (e.g., considered an application of the operating system, but integrated with the operating system).

[0030] In some implementations, the automation assistant 204 may include an input processing engine 206, which may employ multiple different modules to process inputs and / or outputs from the computing device 202 and / or the server device. For example, the input processing engine 206 may include a voice processing engine 208, which is capable of processing audio data received at the assistant interface 220 to identify text contained within the audio data. The audio data may be transmitted from, for example, the computing device 202 to the server device so that computing resources can be saved at the computing device 202 when a network connection is available. Additionally or alternatively, the audio data may be processed specifically at the computing device 202.

[0031] The process for converting audio data into text can include: a speech recognition algorithm that uses a neural network, and / or a statistical model for identifying groups of audio data corresponding to words or phrases. The text converted from the audio data can be parsed by data parsing engine 210 and made available as text data to automation assistant 204. This text data can be used to generate and / or identify command phrases, intents, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by data parsing engine 210 can be provided to parameter engine 212 to determine whether the user has provided input corresponding to a specific intent, action, and / or routine that can be performed by automation assistant 204 and / or an application or agent that can be accessed via automation assistant 204. For example, assistant data 238 can be stored at a server device and / or computing device 202 and can include data defining one or more actions that can be performed by automation assistant 204 and parameters required to perform the actions. Parameter engine 212 can generate one or more parameters for intents, actions, and / or slot values ​​and provide one or more parameters to output generation engine 214. The output generation engine 214 can communicate with the assistant interface 220 using one or more parameters to provide output to the user, and / or communicate with one or more applications 234 to provide output to one or more applications 234.

[0032] In some implementations, the automation assistant 204 can be an application that can be installed "on top" of the operating system of the computing device 202 and / or can form part (or all) of the operating system of the computing device 202. The automation assistant application includes, and / or can access on-device speech recognition, on-device natural language understanding, and on-device performance. For example, on-device speech recognition can be performed using an on-device speech recognition module that processes audio data (detected by microphone(s)) using an end-to-end speech recognition machine learning model locally stored at the computing device 202. On-device speech recognition generates recognized text for spoken utterances (if any) present in the audio data. Furthermore, for example, on-device natural language understanding (NLU) can be performed using an on-device NLU module that processes the recognized text generated using on-device speech recognition, along with optional context data, to generate NLU data.

[0033] NLU data can include multiple intents corresponding to spoken utterances and multiple optional parameters (e.g., slot values) for those intents. On-device execution can be performed using an on-device execution module that utilizes NLU data (from on-device NLU) and optional additional local data to determine multiple actions to be taken to parse the multiple intents of the spoken utterances (and multiple optional parameters for those intents). This can include determining local and / or remote responses to the spoken utterances (e.g., answers), multiple interactions with locally installed applications for execution based on the spoken utterances, multiple commands transmitted to multiple Internet of Things (IoT) devices (directly or via corresponding remote systems) based on the spoken utterances, and / or other parsing actions to be performed based on the spoken utterances. On-device execution can then initiate local and / or remote execution / implementation of the determined multiple actions to parse the spoken utterances.

[0034] In various implementations, remote speech processing, remote NLU, and / or remote execution can be utilized at least selectively. For example, recognized text can be selectively transmitted to at least a plurality of remote automation assistant components for remote NLU and / or remote execution. For example, recognized text can be selectively transmitted for remote execution in parallel with on-device execution, or transmitted in response to failure of on-device NLU and / or on-device execution. However, on-device speech processing, on-device NLU, on-device execution, and / or on-device execution can be preferred, at least because of the reduced latency they provide when parsing spoken utterance (due to the absence of the (multiple) client-server round trips required for parsing spoken utterance). Furthermore, on-device functionality can be the only available functionality in the absence of network connectivity or when network connectivity is limited.

[0035] In some implementations, computing device 202 may include one or more applications 234, which may be provided by a third-party entity different from the entity providing computing device 202 and / or automation assistant 204. The application state engine of automation assistant 204 and / or computing device 202 may access application data 230 to determine one or more actions that can be performed by one or more applications 234, and the state of each of the one or more applications 234 and / or the state of the corresponding device associated with computing device 202. The device state engine of automation assistant 204 and / or computing device 202 may access device data 232 to determine one or more actions that can be performed by computing device 202 and / or one or more devices associated with computing device 202. Furthermore, application data 230 and / or any other data (e.g., device data 232) may be accessed by automation assistant 204 to generate context data 236, which may characterize the context in which a particular application 234 and / or device is performing, and / or the context in which a particular user is accessing computing device 202, accessing application 234 and / or any other device or module. When one or more applications 234 are executed at computing device 202, device data 232 can characterize the current operating state of each application 234 executed at computing device 202. Furthermore, application data 230 can characterize one or more features of the executing application 234, such as the content of one or more graphical user interfaces rendered in the direction of the one or more applications 234.

[0036] In some implementations, computing device 202 may include a suggestion request engine 222 capable of processing requests received from one or more applications 234 of computing device 202 and / or individual computing devices. The request enables automation assistant 204 to identify one or more actions, content, and / or other data that can be suggested to the user. For example, a suggestion request may enable suggestion request engine 222 to access content rendered by the application providing the suggestion request. Additionally or alternatively, with prior user permission, suggestion requests may enable suggestion request engine 222 to access content currently, previously, or ultimately rendered at the interface of computing device 202. The content accessed by suggestion request engine 222 can then be processed according to the suggestion process.

[0037] In some implementations, a content processing engine 216 can be used to select a suggestion process, which can identify a specific suggestion process based on a suggestion request. For example, the content processing engine 216 can initially identify a suggestion process previously used to provide suggestions to the application 234 that provided the suggestion request. Additionally or alternatively, the content processing engine 216 can initially identify a suggestion process assigned to the default suggestion process for the application 234 that provided the suggestion request. Processing of content that can include application data 230, device data 232, and / or context data 236 can result in the generation of suggestion data. The suggestion data can characterize one or more suggestions that can be rendered at the interface of the computing device 202 to help the user initiate a specific action and / or access some other content. However, in some cases, processing of content using the default suggestion process may not result in any suggestions being identified.

[0038] Once the content processing engine 216 has completed processing the content using the initial suggestion process (e.g., using the first suggestion model of suggestion model 228), the content processing engine 216 can use another suggestion model (e.g., a second suggestion model of suggestion model 228) to process the content or other content. This other suggestion model can be, but is not limited to, a test or experimental model used by the entity to determine the accuracy of other suggestion models. For example, the first suggestion model can be the default suggestion model previously used by the automation assistant 204 when generating suggestions, and the second suggestion model can be a model trained using training data different from the training data used to train the first suggestion model. Additionally or alternatively, the first suggestion model can be a suggestion model of a different type than the second suggestion model.

[0039] When the content processing engine 216 processes content using other suggestion models, it can generate additional suggestion data. This additional suggestion data can characterize suggestions that are the same as or different from those identified based on the initial suggestion process used to process the content. The application providing the suggestion request can receive each instance of the suggestion data from the automation assistant 204 and selectively render one or more selectable suggestion elements using the suggestion data. Additionally or alternatively, the automation assistant 204 can selectively render one or more selectable suggestion elements using instances of the suggestion data.

[0040] In some implementations, the automation assistant 204 may include a feedback processing engine 218 capable of processing input characterizing the interaction between a user and one or more selectable suggestion elements. The processing of the input can generate feedback data, which the model assignment engine 226 can use to assign accuracy values ​​to appropriate suggestion models. In some implementations, when it is determined that the accuracy of a second suggestion model is greater than that of a first suggestion model, the automation assistant 204 can assign the second suggestion model as the "default" model. In some implementations, each model can be available on a separate server device and can be trained based on feedback data from various different users and / or their respective devices. Additionally or alternatively, a training engine 224 can be provided at the automation assistant 204 to train each model locally based on the feedback data generated by the feedback processing engine 218.

[0041] In some implementations, the second proposed model can be held at the server computing device to preserve computing resources at the client computing device—at least until the second proposed model has been deemed sufficiently accurate to be provided to the client computing device. Maintaining the "default" model at the client computing device preserves computing resources, such as network bandwidth, which would otherwise be consumed by transmitting data to various server devices via various network connections. Furthermore, maintaining the default model at the client computing device—where no other experimental models exist—mitigates latency between proposal generation, as experiments with other models are not specifically performed at the client computing device. Therefore, by restricting the hosting of certain models to the server computing device, computing resources, such as memory and processing bandwidth, can be preserved at the client computing device.

[0042] In some implementations, the model assignment engine 226 can generate and / or modify model data characterizing the suitability of a particular model when generating selectable suggestions associated with the application. The model assignment engine 226 can modify model data associated with a particular suggestion model based on feedback data generated from interactions between the user and selectable suggestion elements. When an interaction is determined to be positive and / or when the corresponding feedback data is determined to meet a threshold, the model data can be modified to reflect this determination. Additionally or alternatively, when feedback data for one or more interactions is determined to meet a threshold, the model data can be modified to assign the corresponding suggestion model as the default suggestion model. Additionally or alternatively, when an interaction is determined to be negative and / or when the corresponding data is determined to not meet a threshold, the model data can be modified to reflect this determination. Additionally or alternatively, when feedback data for one or more interactions is determined to not meet a threshold, the model data can be modified to further characterize the corresponding suggestion model as less accurate when used to generate suggestions. Then, it is possible to use changes in model data to rank certain models above others, and ultimately assign the most accurate model as the default model for use when generating recommendations for specific applications and / or multiple applications.

[0043] Figure 3A and Figure 3B The illustrations depict methods 300 and 320 for retaining the default suggestion model while simultaneously allowing the use of other models to determine their accuracy. Method 300 can be performed by one or more computing devices, applications, and / or any other means or modules that can be associated with an automation assistant. Method 300 can include an operation 302 determining whether a suggestion request has been received. Suggestion requests can be received from the application with which the user is interacting, and can also be received by additional applications such as an automation assistant or other applications. Suggestion requests can include data that enables the automation assistant to process the suggestion request using a suggestion model to facilitate the automation assistant providing suggestion data back to the application.

[0044] When it is determined that a suggestion request has been received, method 300 can proceed to operation 304; otherwise, the application can continue to determine whether a suggestion request has been received. Operation 304 may include determining whether the current suggestion process is available for generating suggestions for the application. The current suggestion process may include processing application content using a set of rules, one or more machine learning models, and / or any other means or modules for generating suggestion data for the application. When the current suggestion process is not available for generating suggestions, method 300 can proceed to operation 308. Otherwise, when the current suggestion process is available, method 300 can proceed to operation 306.

[0045] Operation 306 can include generating recommendation data using the current recommendation process. For example, the current recommendation process can include processing application content (e.g., a screenshot of a computing device's display) using one or more rules, models, and / or other process data to generate recommendation data. Method 300 can then proceed to operation 308, which can include determining whether another recommendation model is available to be used when processing application content to generate additional recommendation data. The other recommendation model can be, but is not limited to, a backup or "fallback" recommendation model that can be used regardless of whether the recommendation process results in the generation of any recommendations. In some implementations, the other recommendation model can be available at the application, an add-on application (e.g., an automation assistant), and / or a separate computing device to facilitate further training of the other recommendation model before deploying it across various devices.

[0046] When an alternative suggestion model is determined to be available, method 300 can proceed to operation 310, which can include generating additional suggestion data using the alternative suggestion model. When generating the additional suggestion data, the additional application can use the alternative suggestion model and application content, which can be the same as or different from the application content used in the current suggestion process of operation 306. For example, the application content used for the alternative suggestion model could be a screenshot, while the other application content used in the current suggestion process might not include screenshots. Conversely, the other application content used in the current suggestion process can include different screenshots and / or different application data that the application and / or the additional application can access.

[0047] When it is determined that another suggestion is unavailable, method 300 can optionally proceed from operation 308 to optional operation 312, which can include providing one or more default suggestions. The default suggestion can be, but is not limited to, a selectable suggestion element that, when selected, causes an additional application to initialize and execute a default action. For example, the default action could be audibly and / or visually asking the user what help they want (e.g., “How can I help you?”). Additionally or alternatively, the default action could be accessing a separate device and / or application, and / or accessing the internet. Thereafter, method 300 can proceed to operation 302 and continue monitoring suggestion requests.

[0048] When operation 310 is executed, method 300 can proceed from operation 310 to operation 314, which can include causing the application to render one or more selectable suggestion elements based on generated suggestion data and / or additional suggestion data. For example, the selectable elements(s) can be rendered by the application or additional applications, and when selected, the application and / or additional applications can initialize the execution of one or more actions. Method 300 can continue from operation 314 via continuing element "A" (in... Figure 3A(Circle 300 places) to continue to element "A" (in Figure 3B (Circle at position 320) to optional operation 316.

[0049] Optional operation 316 may include determining whether the user has interacted with one or more selectable suggestion elements rendered according to operation 314. When it is determined that the user has interacted with one or more selectable suggestion elements, method 300 may proceed from operation 316 to optional operation 318, which may include further training one or more suggestion models based on user interaction. The one or more suggestion models further trained according to operation 318 may correspond to one or more corresponding models for generating one or more suggestion elements with which the user has interacted. When it is determined that the user has not interacted with one or more selectable suggestions, method 300 may proceed to optional operation 322. Optional operation 322 may include further training one or more suggestion models based on the lack of user interaction. The one or more suggestion models further trained according to operation 322 may correspond to one or more corresponding models for generating one or more suggestion elements with which the user has not interacted.

[0050] In some implementations, operation 316 can be an optional operation to determine the correlation between suggestion data generated according to the current suggestion process and additional suggestion data generated using other suggestion models. Subsequently, other suggestion models can be trained based on whether the correlation meets a threshold. Additionally or alternatively, other suggestion models can be trained based on whether the user interacts with selectable suggestion elements generated using other suggestion models. In this way, other suggestion models can be trained to provide suggestions otherwise provided via the current suggestion process, and can also be trained to provide suggestions that the user is more likely to interact with, thereby encouraging the user to perform computationally efficient operations using additional applications. Method 300 can proceed to operation 302 via continuation element "B" to further determine whether a suggestion request has been received from the application.

[0051] Figure 4 This is a block diagram 400 of an example computer system 410. The computing device 410 typically includes at least one processor 414 that communicates with a plurality of peripheral devices via a bus subsystem 412. These peripheral devices may include a storage subsystem 424 (including, for example, a memory subsystem 424 and a file storage subsystem 426), a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices allow users to interact with the computing device 410. The network interface subsystem 416 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0052] User interface input device 422 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen integrated into a display, a voice recognition system, a microphone, and / or other types of input devices. Generally, the term "input device" is used to encompass all possible types of devices and methods for inputting information onto computing device 410 or a communication network.

[0053] User interface output device 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating visual images. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of devices and methods for outputting information from computing device 410 to a user or another machine or computing device.

[0054] Storage subsystem 424 stores the program and data constructs that provide functionality for some or all of the modules described herein. For example, storage subsystem 424 may include selected aspects of performing method 300, and / or logic implementing system 200, computing device 112, messaging application 102, additional application 120, and / or any other application, device, apparatus, and / or module discussed herein.

[0055] These software modules are typically executed by processor 414 alone or in combination with other processors. Memory 425 used in storage subsystem 424 can include multiple memories, including main random access memory (RAM) 430 for storing instructions and data during program execution and read-only memory (ROM) 432 for storing fixed instructions. File storage subsystem 426 provides permanent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of certain embodiments may be stored by file storage subsystem 426 in storage subsystem 424 or in other machines accessible to processor(s) 414.

[0056] Bus subsystem 412 provides a mechanism for allowing various components and subsystems of computing device 410 to communicate with each other as intended. Although bus subsystem 412 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0057] The computing device 410 can be of various types, including workstations, servers, computing clusters, blade servers, server groups, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, in order to illustrate some implementation methods, Figure 4 The description of the computing device 410 depicted is intended only as a specific example. Many other configurations of the computing device 410 may have... Figure 4 The computing device depicted in the text has more or fewer components.

[0058] In situations where the systems described herein collect or may use personal information about users (or, generally referred to herein as "participants"), users may be given the opportunity to control whether a program or function collects user information (e.g., information about a user's social networks, social behaviors or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. Furthermore, certain data may be processed in one or more ways before being stored or used, thereby removing personally identifiable information. For example, a user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (e.g., generalized to a city, zip code, or state level) when geographic location information is obtained, making it impossible to determine the user's specific geographic location. Therefore, users can control how information about themselves is collected and / or used.

[0059] While several embodiments have been described and illustrated herein, various other means and / or structures may be utilized for performing functions and / or obtaining results and / or one or more advantages described herein, and each such variation and / or modification is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications to which the teachings are addressed. Those skilled in the art will recognize or be able to determine many equivalents of the particular embodiments described herein using only conventional experimentation. Therefore, it should be understood that the foregoing embodiments are presented as examples only, and embodiments may be implemented in ways different from those specifically described and claimed within the scope of the appended claims and their equivalents. Embodiments of this disclosure pertain to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure if such features, systems, articles, materials, kits, and / or methods do not contradict each other.

[0060] In some embodiments, a method implemented by one or more processors is described as including the following operations: receiving a request to generate suggestion data using application content, wherein the application content is generated by an application accessible via a computing device. The method may further include the following operations: in response to receiving the request, processing the application content using a first suggestion model to generate the suggestion data. The method may further include the following operations: determining, based on processing the application content, whether the suggestion data is insufficient for rendering at an interface of the computing device. The method may further include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device: causing a second suggestion model to be used to process the application content to facilitate the generation of additional suggestion data, wherein the first suggestion model is specified as a default suggestion model relative to generating selectable suggestions associated with the application. The method may further include the following operations: determining that the additional suggestion data is sufficient for rendering at the interface of the computing device, and causing the computing device to render one or more selectable suggestions based on the additional suggestion data.

[0061] In some embodiments, the method may further include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device: processing feedback data characterizing the interaction between the user and the one or more selectable suggestions based on the additional suggestion data, and, based on processing the feedback data, determining whether the interaction between the user and the one or more selectable suggestions results in the selection of a particular selectable suggestion of the one or more selectable suggestions. In some embodiments, the method may further include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device, and when the interaction between the user and the one or more selectable suggestions results in the selection of the particular selectable suggestion: specifying a second suggestion model as the default suggestion model relative to the generation of selectable suggestions associated with the application.

[0062] In some embodiments, the method may further include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device, and when the interaction between the user and the one or more selectable suggestions results in the selection of the specific selectable suggestion: based on the user's selection of the specific selectable suggestion, deselecting the first suggestion model from being designated as the default suggestion model. The method may also include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device, and when the interaction between the user and the one or more selectable suggestions results in the selection of the specific selectable suggestion: assigning both the first suggestion model and the second suggestion model as to be used when providing subsequent selectable suggestions for the application. In some embodiments, the method may further include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device: processing feedback data characterizing the interaction between the user and the one or more selectable suggestions based on the additional suggestion data; and, based on processing the feedback data, determining whether to designate the second suggestion model instead of the first suggestion model as the default suggestion model, or determining whether to combine the second suggestion model with the first suggestion model as an additional default suggestion model.

[0063] In some embodiments, the method may further include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device, and when it is determined that the second suggestion model is designated as the default suggestion model or the additional default suggestion model: relative to generating selectable suggestions associated with the application, designate the second suggestion model as the default suggestion model or the additional default suggestion model. In some embodiments, the method may further include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device, and when it is determined that the second suggestion model is designated as the default suggestion model or the additional default suggestion model: provide the second suggestion model from the server computing device to the computing device, wherein the first suggestion model is stored at the computing device. In some embodiments, the method may further include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device, and when the feedback data fails to meet a threshold for designating the second suggestion model as the default suggestion model: based on the feedback data that fails to meet the threshold, modify model data characterizing the suitability of the second suggestion model for use when generating selectable suggestions associated with the application.

[0064] In some embodiments, causing the application content to be processed using the second suggestion model includes: transmitting the application content to a server computing device, wherein the server computing device provides access to the second suggestion model. In some embodiments, the method may further include: when it is determined that the suggestion data is sufficient for rendering at the interface of the computing device: causing the computing device to render one or more other selectable suggestions based on the suggestion data, and causing the application content to be processed using the second suggestion model to facilitate the generation of the additional suggestion data. In some embodiments, the method may further include: when it is determined that the suggestion data is sufficient for rendering at the interface of the computing device: determining the degree of correlation between the additional suggestion data generated using the second suggestion model and the suggestion data generated using the first suggestion model.

[0065] In some embodiments, the method may further include the following operations: when it is determined that the suggestion data is sufficient for rendering at the interface of the computing device: modifying model data characterizing the applicability of the second suggestion model relative to other models based on the relevance, for use when generating selectable suggestions associated with the application. The method may further include the following operations: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device: determining a corresponding relevance between the application content and one or more additional suggestion models of other suggestion models accessible to the computing device, and identifying the second suggestion model based on its relevance relative to other relevance between the application content and the other suggestion models.

[0066] In some embodiments, the method may further include the following operation: when it is determined that the suggestion data is insufficient for rendering at the interface of the computing device: determining a corresponding relevance between the application and one or more additional suggestion models of other suggestion models accessible to the computing device, and identifying the second suggestion model based on its relevance relative to other relevance between the application and the other suggestion models. In some embodiments, the second suggestion model is a multi-domain neural network model. In some embodiments, the method may further include: accessing application content in response to receiving the request, wherein the application content represents one or more operations performed by the application, and wherein the operations of the one or more operations include rendering a graphical user interface containing content generated during an interaction between the user and another user. In some embodiments, the request is received by an automated assistant in response to verbal utterance from the user. In some embodiments, the additional suggestion data represents one or more assistant actions that can be initialized via the automated assistant.

[0067] In another embodiment, a method implemented by one or more processors is described as including operations such as: processing application content by a first application according to a suggestion generation process for generating first suggestion data, wherein the application content is provided at least partially via a second application accessed by a user via a computing device, and wherein the suggestion generation process has previously been used to generate selectable suggestions based on other application content previously provided via the second application. In some embodiments, the method may further include the operation of processing the application content by the first application using a suggestion model for generating second suggestion data, wherein the suggestion generation process differs from processing the application content using the suggestion model, and wherein the suggestion model has not previously been used to process the other application content previously provided via the second application. In some embodiments, the method may further include the operation of causing the interface of the computing device to render: a first selectable suggestion element based on the first suggestion data and the suggestion generation process, and a second selectable suggestion element based on the second suggestion data and the suggestion model.

[0068] In some implementations, processing the application content according to the suggestion generation process includes: accessing an image of graphical content being rendered at the graphical user interface of the computing device, wherein the first selectable suggestion element is different from the second selectable suggestion element.

[0069] In another embodiment, a method implemented by one or more processors is described as including operations such as: an application causing the interface of a computing device to render content based on the interaction between a user and the application. The method may further include: receiving suggestion data from an additional application representing one or more actions that can be performed by the additional application, wherein the suggestion data is generated using a suggestion generation process previously used to generate selectable suggestions using prior application content of the application. The method may further include: receiving additional suggestion data from the additional application representing one or more other actions that can be performed by the additional application, wherein the additional suggestion data is generated using a suggestion model that was previously unavailable to the additional application for generating the selectable suggestions from the prior application content. The method may further include: the application causing the interface of the computing device to render: a first selectable suggestion element based on the suggestion data, wherein the user's selection of the first selectable suggestion element causes the additional application to initialize the execution of the one or more actions; and a second selectable suggestion element based on the additional suggestion data, wherein the user's individual selection of the second selectable suggestion element causes the additional application to initialize the execution of the one or more other actions.

[0070] In some embodiments, the method may further include the following operation: after rendering the first selectable suggestion element and the second selectable suggestion element on the interface of the computing device: in response to the user interacting with the second selectable suggestion element, causing the additional application to continue using the suggestion model when generating subsequent suggestion data for the application. In some embodiments, the method may further include the following operation: after rendering the first selectable suggestion element and the second selectable suggestion element on the interface of the computing device: in response to the user not interacting with the second selectable suggestion element within a threshold time period, causing the additional application to stop using the suggestion model when generating other suggestion data for the application.

Claims

1. A method implemented by one or more processors, the method comprising: Receive requests for generating suggestion data using application content. The application content is generated by an application accessible via a computing device, and the request is received by an additional application. In response to receiving the request, the application content is processed at the computing device using a first suggestion model to generate the suggestion data. The first proposed model is stored at the computing device; Based on processing the application content, determine whether the suggested data is insufficient for rendering on the interface of the computing device; and When it is determined that the suggested data is insufficient for rendering at the interface of the computing device: The application content is processed at a remote computing device using a second recommendation model stored at the remote computing device to facilitate the generation of additional recommendation data. The first suggestion model is specified as the default suggestion model relative to generating selectable suggestions associated with the application. Determine that the additional suggested data is sufficient for rendering at the interface of the computing device, and The computing device renders one or more selectable suggestions based on the additional suggestion data; Provide feedback data, which characterizes the interaction between the user and the one or more selectable suggestions based on the additional suggestion data; After providing the feedback data: The second suggested model is received at the computing device based on the provided feedback data. The second suggested model is received at the computing device, the second suggested model is stored at the computing device, and is used at the computing device as the default suggested model in place of the first suggested model.

2. The method according to claim 1, further comprising: When it is determined that the suggested data is insufficient for rendering at the interface of the computing device: Processing feedback data that characterizes the interaction between the user and one or more selectable suggestions based on the additional suggestion data, and Based on processing the feedback data, it is determined whether the interaction between the user and the one or more selectable suggestions results in the selection of a particular selectable suggestion from the one or more selectable suggestions.

3. The method according to claim 2, further comprising: When it is determined that the suggested data is insufficient for rendering at the interface of the computing device, and When the interaction between the user and the one or more selectable suggestions results in a selection of the specific selectable suggestion: The second suggestion model is specified as the default suggestion model relative to the generated suggestions that can be selected and associated with the application.

4. The method according to claim 2, further comprising: When it is determined that the suggested data is insufficient for rendering at the interface of the computing device, and When the interaction between the user and the one or more selectable suggestions results in a selection of the specific selectable suggestion: Based on the user's selection of the specific selectable suggestion, the first suggestion model is no longer designated as the default suggestion model.

5. The method according to claim 2, further comprising: When it is determined that the suggested data is insufficient for rendering at the interface of the computing device, and When the interaction between the user and the one or more selectable suggestions results in a selection of the specific selectable suggestion: Both the first suggestion model and the second suggestion model are assigned to be used when providing subsequent optional suggestions for the application.

6. The method according to claim 1, further comprising: When it is determined that the suggested data is insufficient for rendering at the interface of the computing device: Processing feedback data that characterizes the interaction between the user and one or more selectable suggestions based on the additional suggestion data, and Based on processing the feedback data, determine whether to designate the second suggestion model instead of the first suggestion model as the default suggestion model, or determine whether to combine the second suggestion model with the first suggestion model as an additional default suggestion model.

7. The method according to claim 6, further comprising: When it is determined that the suggested data is insufficient for rendering at the interface of the computing device, and When it is determined that the second suggested model is specified as the default suggested model or the additional default suggested model: Instead of generating selectable suggestions associated with the application, the second suggestion model is designated as either the default suggestion model or the additional default suggestion model.

8. The method according to claim 6, further comprising: When it is determined that the suggested data is insufficient for rendering at the interface of the computing device, and When the feedback data fails to meet the threshold for designating the second suggested model as the default suggested model: Based on the feedback data that fails to meet the threshold, the model data characterizing the suitability of the second suggestion model is modified for use in generating selectable suggestions associated with the application.

9. The method according to claim 1, wherein, Using the second suggested model to process the application content includes: The application content is transmitted to the server computing device. The server computing device provides access to the second proposed model.

10. The method according to claim 1, further comprising: When it is determined that the suggested data is sufficient for rendering at the interface of the computing device: The computing device renders one or more other selectable suggestions based on the suggestion data, and The second suggestion model is used to process the application content to facilitate the generation of the additional suggestion data.

11. The method of claim 10, further comprising: When it is determined that the suggested data is sufficient for rendering at the interface of the computing device: Determine the degree of correlation between the additional suggestion data generated using the second suggestion model and the suggestion data generated using the first suggestion model.

12. The method of claim 11, further comprising: When it is determined that the suggested data is sufficient for rendering at the interface of the computing device: The model data characterizing the applicability of the second recommendation model relative to other models is modified based on the degree of relevance for use in generating selectable recommendations associated with the application.

13. The method according to claim 1, further comprising: When it is determined that the suggested data is insufficient for rendering at the interface of the computing device: Determine the corresponding relevance between the application content and one or more additional suggestion models of other suggestion models accessible to the computing device, and The second suggestion model is identified based on its relevance to the application content and other suggestion models.

14. The method according to claim 1, further comprising: When it is determined that the suggested data is insufficient for rendering at the interface of the computing device: Determine the corresponding relevance between the application and one or more additional proposed models of other proposed models accessible to the computing device, and The second suggested model is identified based on its relevance to other suggested models relative to the degree of relevance between the application and the other suggested models.

15. The method according to claim 1, wherein, The second proposed model is a multi-domain neural network model.

16. The method according to any one of claims 1-15, further comprising: Access the application content in response to receiving the request. Wherein, the application content represents one or more operations performed by the application, and The operation of one or more of the operations includes rendering a graphical user interface that includes content generated during the interaction between the user and another user.

17. The method according to claim 1, wherein, The request is received by an automated assistant in response to the user's spoken words.

18. The method according to claim 17, wherein, The additional suggestion data represents one or more assistant actions that can be initialized by the automated assistant.

19. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 18.

20. A computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 18.

21. A system comprising one or more processors for performing the method of any one of claims 1 to 18.

Images