Virtual assistant configured for recommended actions in facilitating existing conversations

Abstract

Virtual assistants configured with recommended actions at facilitating existing conversations are disclosed. A system is described that determines, based on information associated with a user of a computing device, an event for initiating an interaction between the user and an assistant executing at the computing device. The system selects, based on the event, at least one action associated with the event from a plurality of actions performed by the assistant. The system determines, based on the at least one action, whether to output a notification of the event that includes an indication of the event and a request to perform the at least one action associated with the event. In response to determining to output the notification of the event, the system sends the notification of the event to the assistant for output during the interaction between the user and the assistant.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201880029807.4, filed on May 2, 2018. Technical Field

[0003] This application relates to a virtual assistant that configures recommended actions when facilitating existing conversations. Background Technology

[0004] Some computing platforms offer a user interface through which users can chat, speak, or otherwise communicate with a virtual computing assistant (e.g., also known as an "intelligent personal assistant" or simply an "assistant") to output information, respond to user needs, or otherwise perform certain actions to help users complete tasks. While some assistants may be quite complex and configured to perform a wide variety of real-world or virtual tasks, users may not always be aware of all the capabilities offered by the assistant. Users may only request information or instruct the assistant to perform certain tasks that the user knows the assistant can handle. Summary of the Invention

[0005] Typically, the technology disclosed herein enables an assistant to output information about one or more additional actions that the assistant can initiate in facilitating an existing conversation with a user, without requiring the user to request such information from the assistant. That is, even without an explicit user request for this information, the assistant can proactively guide the user to additional capabilities of the assistant related to the user's existing conversation. For example, once the assistant has received permission to maintain or access information associated with the user, the assistant can determine, based on user information, that the user's flight booking departure time has been delayed. In addition to generating output informing the user of the flight delay, without receiving user input, the assistant can generate additional output offering to reschedule dinner reservations or ground transportation to adjust for the delay. In this way, the assistant is configured to output conversation initiation, output additional information, or output relevant proposals to perform additional actions (e.g., rescheduling dinner or ground transportation) to facilitate an existing conversation with the user. Therefore, the described technology can improve the usability of the assistant by reducing the amount of user input required for the user to discover various actions and for the assistant to perform them.

[0006] In one example, this disclosure relates to a method comprising: determining, based on information associated with a user of a computing device, an event for initiating an interaction between the user and an assistant executing on the computing device, wherein the assistant is configured to perform a plurality of actions; selecting, based on the event, at least one action associated with the event from the plurality of actions; and determining, based on the at least one action, whether to output a notification of the event, wherein the notification includes an indication of the event and a request to perform the at least one action associated with the event. The method further comprises, in response to determining the notification to output the event, sending a notification of the event to the assistant for output during the interaction between the user and the assistant.

[0007] In another example, this disclosure relates to a computing device including an output device; at least one processor; and at least one memory. The memory includes instructions, when executed, to cause the at least one processor to execute an assistant configured to: determine, based on information associated with a user of the computing device, an event for initiating an interaction between the user and the assistant, wherein the assistant is configured to perform a plurality of actions; select, based on the event, at least one action associated with the event from the plurality of actions; determine, based on the at least one action, whether to output a notification of the event, wherein the notification includes an indication of the event and a request to perform the at least one action associated with the event; and, in response to determining the notification to output the event, output the notification of the event via the output device during the interaction between the user and the assistant.

[0008] In another example, this disclosure relates to a computer-readable storage medium including instructions that, when executed, cause at least one processor of a digital assistant system to: determine, based on information associated with a user of a computing device, an event for initiating an interaction between the user and an assistant executing on the computing device, wherein the assistant is configured to perform a plurality of actions; select, based on the event, at least one action associated with the event from the plurality of actions; and determine, based on the at least one action, whether to output a notification of the event, wherein the notification includes an indication of the event and a request to perform the at least one action associated with the event. When executed, the instructions further cause at least one processor of the digital assistant system to send a notification of the event to the assistant for output during the interaction between the user and the assistant in response to determining the notification to output the event.

[0009] In another example, this disclosure relates to a system including means for determining an event for initiating an interaction between a user and an assistant executing on the computing device based on information associated with a user of the computing device, wherein the assistant is configured to perform a plurality of actions; means for selecting at least one action associated with the event from the plurality of actions based on the event; and means for determining whether to output a notification of the event based on the at least one action, wherein the notification includes an indication of the event and a request to perform at least one action associated with the event. The system further includes means for sending a notification of the event to the assistant for output during the interaction between the user and the assistant in response to determining the notification of the event to be output.

[0010] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of this disclosure will become apparent from the specification, drawings, and claims. Attached Figure Description

[0011] Figure 1 This is a conceptual diagram illustrating an exemplary system that performs an exemplary virtual assistant according to one or more aspects of this disclosure.

[0012] Figure 2 This is a block diagram illustrating an exemplary computing device configured to perform an exemplary virtual assistant according to one or more aspects of this disclosure.

[0013] Figure 3 This is a flowchart illustrating exemplary operations performed by one or more processors executing an exemplary virtual assistant in accordance with one or more aspects of this disclosure.

[0014] Figure 4 This is a block diagram illustrating an exemplary digital assistant configured to perform an exemplary virtual assistant according to one or more aspects of this disclosure. Detailed Implementation

[0015] Figure 1 This is a conceptual diagram illustrating an exemplary system that performs an exemplary virtual assistant according to one or more aspects of this disclosure. Figure 1 System 100 includes a digital assistant server 160 that communicates via network 130 with search server system 180 and computing devices 110A and 110B (collectively, computing device 110). Although system 100 is shown distributed among digital assistant server 160, search server system 180, and computing device 110, in other examples, features and technologies belonging to system 100 may be performed internally by local components of computing device 110. Similarly, digital assistant server 160 may include certain components and perform various technologies otherwise attributed to search server system 180 and / or computing device 110 as described below. Figure 1 Two computing devices 110 are shown, but any suitable number or one or more computing devices 110 can be used.

[0016] Network 130 refers to any public or private communication network used for transmitting data between computing systems, servers, and computing devices, such as cellular, Wi-Fi, and / or other types of networks. Digital assistant server 160 can exchange data with computing device 110 via network 130 to provide virtual assistant services accessible to computing device 110 when computing device 110 is connected to network 130. Digital assistant server 160 can exchange data with search server system 180 via network 130 to access search services provided by search server system 180. Computing device 110 can exchange data with search server system 180 via network 130 to access search services provided by search server system 180.

[0017] Network 130 may include one or more network hubs, network switches, network routers, or any other network devices operatively coupled to each other, thereby providing information exchange between server systems 160 and 180 and computing device 110. Computing device 110, digital assistant server 160, and search server system 180 may use any suitable communication technology to send and receive data through network 130. Computing device 110, digital assistant server 160, and search server system 180 may each be operatively coupled to network 130 using a corresponding network link. The links coupling computing device 110, digital assistant server 160, and search server system 180 to network 130 may be Ethernet or other types of network connections, and these connections may be wireless and / or wired connections.

[0018] Digital assistant server 160 and search server system 180 represent any suitable remote computing system (such as one or more desktop computers, laptop computers, mainframes, servers, cloud computing systems, etc.) capable of sending information to and receiving information from a network (such as network 130). Digital assistant server 160 hosts a virtual assistant service (or at least provides access to a virtual assistant service). Assistant server system 180 hosts a search service (or at least provides access to a search service). In some examples, digital assistant server 160 and search server system 180 represent cloud computing systems that provide access to their respective services via the cloud.

[0019] Computing device 110 refers to a mobile or non-mobile computing device of an individual configured to access a virtual assistant service provided via network 130. Examples of computing devices 110 include mobile phones, tablets, laptops, desktop computers, servers, mainframes, set-top boxes, televisions, wearable devices (e.g., computerized watches, computerized glasses, computerized gloves, etc.), home automation devices or systems (e.g., smart thermostats or security systems), personal digital assistants (PDAs), gaming systems, media players, e-book readers, mobile TV platforms, car navigation or infotainment systems, or any other type of mobile, non-mobile, wearable, and non-wearable computing device configured to perform or access a virtual assistant and receive information via a network (such as network 130).

[0020] The computing device 110 can communicate with the digital assistant server 160 and / or the search server system 180 via the network 130 to access the virtual assistant service provided by the digital assistant server 160 and / or the search service provided by the search server system 180. In providing the virtual assistant service, the digital assistant server 160 can communicate with the search server system 180 to obtain the information needed to complete the task.

[0021] exist Figure 1 In the example, digital assistant server 160 includes assistant module 122B and user information data storage 124B. Computing device 110A includes user interface component (UIC) 112, user interface (UI) module 120, assistant module 122, and user information data storage 124A. Search server system 180 includes search module 182. Although not shown, computing device 110B may include components and / or modules similar to those in computing device 110A.

[0022] Modules 120, 122A, 122B, and 182 can perform operations using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in or executing on one of the computing devices 110A, digital assistant server 160, or search server system 180. The computing devices 110A, digital assistant server 160, and search server system 180 can execute modules 120, 122A, 122B, and 182 having multiple processors or multiple devices. The computing devices 110A, digital assistant server 160, and search server system 180 can execute modules 120, 122A, 122B, and 182 as virtual machines executing on underlying hardware. Modules 120, 122A, 122B, and 182 can execute as one or more services of an operating system or computing platform. Modules 120, 122A, 122B, and 182 can execute as one or more executable programs at the application layer of the computing platform.

[0023] The UIC 112 of computing device 110 can be used as an input and / or output device of computing device 110A. UIC 112 can be implemented using various technologies. For example, UIC 112 can act as an input device using a presence-sensitive input screen, microphone technology, infrared sensor technology, camera, or other input device technologies used when receiving user input. UIC 112 can also act as an output device, configured to present output to the user using any one or more display devices, speaker technology, haptic feedback technology, or other output device technologies used when outputting information to the user.

[0024] UI module 120 can manage user interaction with UIC 112 and other components of computing device 110A, and can interact with digital assistant server 160 to provide assistant services via UIC 112. UI module 120 can cause UIC 112 to output a user interface when a user of computing device 110A views output and / or provides input at UIC 112. For example, as... Figure 1 As shown, UI module 120 can send instructions to UIC 112, which in turn cause UIC 112 to display user interface 114 on its display screen.

[0025] User interface 114 includes text of a conversation between the user of computing device 110A and an assistant provided by assistant module 122A. Figure 1 In the examples, user interface 114 is a graphical user interface. In some examples, user interface 114 may be an audible user interface. User interface 114 may include various forms of virtual assistant information, such as audible sounds, vibrations, text, graphics, content cards, images, etc.

[0026] When a user interacts with the user interface 114 at different times and when the user and computing device 110A are in different locations, UI module 120 and UIC 112 can receive one or more indications from the user regarding input (e.g., voice input, touch input, non-touch input or presence-sensitive input, video input, audio input, etc.). UI module 120 and UIC 112 can interpret the input detected at UIC 112 and can relay information about the input detected at UIC 112 to assistant module 122A and / or to one or more other associated platforms, operating systems, applications, and / or services executed at computing device 110A, such as causing computing device 110A to perform functions.

[0027] UI module 120 can receive information and instructions from one or more associated platforms, operating systems, applications, and / or services executing on computing device 110A and / or one or more remote computing systems (such as server systems 160 and 180). Additionally, UI module 120 can act as a medium between one or more associated platforms, operating systems, applications, and / or services executing on computing device 110A and various output devices of computing device 110A (e.g., speakers, LED indicators, audio or haptic output devices, etc.) to generate output (e.g., graphics, flashing lights, sound, haptic response, etc.) through computing device 110A. For example, UI module 120 can cause UIC 112 to output user interface 114 based on data received by UI module 120 from digital assistant server 160 via network 130. UI module 120 can receive information (e.g., audio data, text data, image data, etc.) and instructions as input from digital assistant server 160 and / or assistant module 122A for presentation as user interface 114.

[0028] Assistant modules 122A and 122B can jointly maintain user information data stores 124A and 124B as part of a virtual assistant service accessed via computing device 110. Assistant module 122B and user information data store 124B represent a server-side or cloud implementation of the exemplary assistant, while assistant module 122A and user information data store 124A represent a client-side or local implementation of the exemplary assistant. In some examples, some or all of the functionality belonging to assistant module 122A can be performed by assistant module 122B, and vice versa.

[0029] Collectively referred to as Assistant Module 122, Assistant Modules 122A and 122B can perform actions configured to automatically and without user interaction proactively guide the user of the computing device to meaningful or other useful information that the user may currently be interested in. Assistant Module 122 can be configured to output information about one or more additional actions that the assistant is configured to perform when facilitating an existing session with the user, without the user requesting such information from the assistant. For example, even without an explicit user request for such information, the assistant provided by Assistant Module 122 can proactively guide the user to additional capabilities of the assistant related to the user's existing session.

[0030] Each of the assistant modules 122 may include a software agent configured to perform tasks or services for an individual user, such as a computing device 110. Assistant modules 122 may perform these tasks or services based on: (e.g., user input detected at UIC 112), context awareness (e.g., location-based), and / or the ability to access other information (e.g., weather or traffic conditions, news, stock prices, sports scores, user schedules, traffic schedules, retail prices, etc.) from various information sources (e.g., locally stored on computing device 110, obtained via search services provided by digital assistant server 160 or through search server system 180). Assistant modules 122 may employ artificial intelligence and / or machine learning techniques to automatically identify and complete one or more tasks on behalf of the user. Assistant modules 122 may rely on other applications (e.g., third-party applications), services, or other devices (e.g., televisions, cars including autonomous vehicles, watches, home automation systems, entertainment systems, etc.) to perform one or more tasks.

[0031] Upon receiving explicit user consent, assistant module 122 may automatically maintain user-associated information, or pointers to locations of user-associated information (e.g., stored by an information source located at computing device 110, server system 160 or 180, or by any other server or device located elsewhere on network 130), for example, during the execution of operations supporting a session with the user of computing device 110. Assistant module 122 may maintain user-associated information or pointers to that information as user information data stores 124A and 124B (collectively, “User Information Data Store 124”). Data store 124 enables the assistant, executed by assistant module 122, to quickly access the user information required by the assistant to complete real-world tasks, virtual tasks, or otherwise respond to the immediate and / or future needs of the user of computing device 110.

[0032] For example, information associated with a user may be information obtained with the user's permission during a session between the user and the assistant, calendar information, contact information, user interests, user preferences, and any other information associated with the user that the assistant module 122 can use to proactively guide the user to meaningful information. Examples of information sources associated with a user may include, but are not limited to, intelligence built into an email system, user interactions with the assistant module 122, user interactions and application usage on devices associated with the user (e.g., computing device 110), or other information sources.

[0033] As used throughout this manual, the term "session" refers to the regular interaction between the user of the computing device and the assistant that performs or can be accessed from the computing device, compared to interactions that may occur during initial setup, installation, first use, etc. In other words, a session occurs during the user's daily use of the assistant while the user is performing tasks other than setting up the assistant. As used herein, a session does not refer to the user answering questions as part of the setup application or to providing information in pre-defined fields that populate the user interface.

[0034] In operation, assistant module 122 can provide an assistant that executes on or is accessible from computing device 110. The assistant can be configured to initiate a session or conversation with the user, ask the user for information, and suggest actions that the assistant can perform on behalf of the user, even if the user is unaware of these actions. The assistant can be configured to recommend valuable actions related to the user's existing session without the user explicitly requesting them. For example, assistant module 122B can determine whether to start a conversation with the user and select that conversation to begin. Assistant module 122B can enable assistant module 122A to initiate a conversation and obtain user authorization for assistant module 122A to perform actions related to that conversation.

[0035] Assistant module 122 can determine events that trigger an interaction (e.g., a session or conversation) between the user of computing device 110 and the assistant performing at computing device 110, based on information associated with the user of computing device 110. Assistant module 122 can analyze only the information associated with computing device 110 and / or the user of computing device 110, if the user has explicitly consented to the use or collection of such information. Assistant module 122 can further provide the user with an opportunity to withdraw consent, and in this case, assistant module 122 can stop collecting or otherwise retaining the information associated with computing device 110 or the user of computing device 110. Assistant module 122 can analyze the user's personal information (e.g., data storage 124), including but not limited to: search history and online interactions, previous interactions with the assistant (e.g., "call mom", "call Kathy"), and interactions with other applications and third-party entities performing at or accessible from computing device 110. The assistant module 122B may know which "surface" or device a user is more likely to interact with the assistant from at any given time, and send an event notification to that specific surface.

[0036] For example, after assistant module 122 has received consent from the user to maintain information associated with that user or has access to information associated with that user, assistant module 122B can maintain user information in data storage 124B, including information about the user's upcoming travel plans on computing device 110. The travel plans may indicate that the user has booked a seat on a specific flight originally scheduled to depart approximately twelve hours from the current time. An agent executing at assistant module 122B can be configured to obtain updates associated with the travel plans from search server system 180. For example, the agent of assistant module 122B can send a query to search module 182 for status information about a specific flight, and in response, the agent of assistant module 122B can receive information indicating that the specific flight has been delayed by thirty minutes from its originally scheduled departure time.

[0037] Search module 182 can perform searches (e.g., internet searches) from various information sources (e.g., locally or remotely stored in search server system 180) for information related to the search query (e.g., weather or traffic conditions, news, stock prices, sports scores, user schedules, traffic schedules, autonomous vehicle reservations, retail prices, etc.). After performing a search, search module 182 can output the information returned from the search (e.g., search results, links to search results, etc.) to digital assistant server 160 or computing device 110. Search module 182 can perform searches on information determined to be related to the search query generated by search module 182 (e.g., based on context information associated with computing device 110A). Search module 182 can also perform searches based on search queries received by search module 182 from digital assistant server 160 or computing device 110A (e.g., as part of a task being performed by a virtual assistant on behalf of a user of computing device 110).

[0038] Assistant module 122B can determine that flight delay is an event that initiates interaction between the user and computing device 110. Assistant module 122B can send a message to assistant module 122A to trigger a notification of this event.

[0039] In response to receiving an event trigger from assistant module 122B, assistant module 122A can select at least one action associated with or related to the event from a plurality of actions configured to be performed by assistant module 122A. For example, assistant module 122A can be configured to perform a plurality of actions, including but not limited to: scheduling services (e.g., transportation, including driverless vehicle booking, home or car-related repair / maintenance, food delivery, pet care, pet walking, housekeeping, babysitting, delivery, travel booking, and any or all other services), exchanging communications (e.g., calendar invitations, email, text-based messaging, social media network messaging, telephone or video calls, and any or all other communications), purchasing goods (e.g., online shopping, flower delivery, purchasing event tickets, and any or all other purchasing-related actions), and other actions. Assistant module 122A may rely on other applications or agents executed at computing device 110A or accessible by computing device 110A to perform the plurality of actions. The assistant module 122A can rely on other applications or agents that are executed at or accessible from the computing device 110A to perform multiple actions.

[0040] Assistant module 122A can select any one or more actions from a plurality of actions as relevant to an event. For example, assistant module 122A can score its capabilities or actions for a specific type of event based on feedback obtained regarding what type of actions other assistants perform on behalf of users on other computing devices for a specific type of event. Assistant module 122A can score each of the plurality of actions and select one or more actions with higher scores as potentially relevant to the event. For example, based on information stored at data storage 124A, assistant module 122A can determine that, in addition to flight bookings, the user has also booked ground transportation, such as autonomous vehicles, at the final destination of the flight. Assistant module 122A can determine, based on how other users interact with other assistants, that scheduling or rescheduling ground transportation is an action typically performed by an assistant related to a flight delay event.

[0041] Not all events are necessarily worth notifying the user of. Therefore, the assistant module 122A can determine whether to output an event notification before outputting one at the current time. The assistant module 122A can restrict the notification of certain events or customize the notification of certain events for specific scenarios to ensure that the user will find value in the notification. As an example, the assistant module 122A can determine whether to output an event notification based at least in part on a combination of how close the previous interaction between the user and the assistant was in time and the frequency of the previous interaction between the user and the assistant.

[0042] For example, based on information stored in user information data storage 124A, assistant module 122A can determine that when the user wakes up, the user interacts with the assistant at an average interval or frequency of a threshold (e.g., approximately once per hour per day). Since the last session between the user and the assistant occurred within a threshold interval prior to the current time (e.g., only fifteen minutes ago), assistant module 122A can determine to prohibit the output of a half-hour flight delay notification for flights not scheduled to depart within another twelve hours or longer. In some examples, because the last session between the user and the assistant occurred within a threshold interval greater than or equal to the current time (e.g., one hour ago), assistant module 122A can determine to output the half-hour flight delay notification for flights not scheduled to depart within another twelve hours or longer.

[0043] In response to a notification indicating an output event, assistant module 122A may output a notification of the event. This notification may include an indication of the event and a request to perform at least one of the actions selected above associated with the event. For example, assistant module 122B may send the event notification to assistant module 122A for output via UIC 212 during an interaction between the user and the assistant. Assistant module 122A may output the notification as part of user interface 114 and include... Figure 1 The diagram shows a portion of the conversation text. Assistant module 122A can make the notification include an instruction to perform one or more actions on behalf of the user in relation to the event. For example, the instruction for the event could correspond to the displayed or spoken text “Your flight tonight to Topeka was cancelled due to bad weather,” and the instruction to perform at least one action could correspond to the displayed or spoken text “Would you like me to reschedule the ground transportation you had reserved?”

[0044] To prevent unintentional output of personal or sensitive information, assistant module 122A can generate notification signaling instead of actually outputting the notification. For example, in response to receiving a notification from assistant module 122B, assistant module 122A can cause UIC 112 to output an alert to indicate that the assistant has information for you. This alert might correspond to the text “I have something for you. Are you here?” shown in user interface 114. The output of UIC 112 can proactively inform the user that meaningful information is available.

[0045] In some examples, assistant module 122 may suppress output notifications until assistant module 122 determines that the user has received and consented to such signaling. For example, assistant module 122B may determine that the user has consented to the session when assistant module 122A receives an indication of consent from the user (e.g., the microphone of UIC 112 indicates the phrase "Yes, I'm here"), an indication that the user has activated the display of UIC 112, and / or data indicating that the user has interacted with computing device 110A in some other way (e.g., lifting the device, pressing a button, etc.) to indicate consent.

[0046] In any case, after notification of an output event and a request to perform at least one action, the assistant module 122A can detect user input when the user responds to the request. Figure 1 In the example, assistant module 122A can receive spoken or written user input corresponding to the text "Yes, please reschedule my ride" via the camera or microphone of UIC 112. In response to confirmation that the spoken or written user input corresponds to the request, the agent of assistant module 122A can reschedule ground transportation reservations to adjust for flight delays.

[0047] In some examples, an action may be associated with a third device that is different from any of the devices in the assistant modules 122. Assistant module 122A can implement or execute an action by sending a message to the third device to cause it to perform the action. For example, a ground traffic service may be associated with an application running at computing device 110B. Assistant module 122A may send a message to the application via network 130 requesting a rescheduling service. In response to the application confirming that ground traffic has been scheduled, computing device 110B may output a notification indicating the new time for the ground traffic.

[0048] In this way, assistant module 122 can proactively guide the user to other actions or capabilities of the assistant without overwhelming the user with trivial information or requiring the user to provide input related to such actions or capabilities. Because the exemplary assistant proactively guides the user to meaningful information, the user does not need to spend time using specific queries to guide the assistant. Therefore, the exemplary assistant according to this disclosure can complete tasks and / or respond to users more quickly than other assistants. By proactively guiding the user to meaningful information, the assistant may appear more natural, less clumsy, and more agreeable to the user.

[0049] Among the several benefits provided by the above method are: (1) by actively guiding the user to the assistant's actions or capabilities without relying on specific queries from the user or the user spending time learning the actions or capabilities via data or other means, the processing complexity and time of device actions can be reduced. Explicit user requests for information about the assistant's capabilities, such as through voice input, can be avoided. This may result in the computer device on which the assistant is implemented not needing to handle such requests, and thus, particularly with time and multiple sessions, the computer device benefits from significant savings in computer resources associated with implementing the assistant; (2) Meaningful information and user-related information can be stored locally, thereby reducing the need for complex and memory-intensive secure protocols for transmitting privacy data on the user device; (3) Because the exemplary assistant guides the user to actions or capabilities, the user can request fewer specific queries, thereby reducing the need for query rewriting and other computationally complex data retrieval on the user device; and (4) Since the number of specific queries is reduced, the data required for the assistant module to respond to specific queries is also reduced, thus reducing network usage. For example, data traffic between the client computer device 110 from which the assistant can output information and the remote server 180 from which the client device can seek information can be reduced. Besides, or alternatively, this reduction in network data traffic may stem from a decrease in the overall amount of time required to keep the communication channel between a specific client device 110 and the remote server 180 open for a specific session. Those skilled in the art will recognize that, by automatically facilitating a session with the user without requiring the user to request information from an assistant, the overall efficiency of the session can be improved from a technical resource perspective, as the user is not required to explicitly search for information through a separate session operation. Furthermore, the duration of the session can be reduced, allowing the communication channel to be closed earlier than in other cases, thereby freeing up network resources.

[0050] Figure 2 This is a block diagram illustrating an exemplary computing device configured to perform an exemplary virtual assistant according to one or more aspects of this disclosure. In the following, it will be... Figure 2 The computing device 210 is described as follows Figure 1 Examples of computing devices 110A and / or 110B. Figure 2 Only one specific example of computing device 210 has been shown, and many other examples of computing device 210 may be used in other instances, and may include a subset of the components included in exemplary computing device 210 or may include... Figure 2 Additional components not shown.

[0051] like Figure 2As shown in the example, computing device 210 includes a user interface component (UIC) 212, one or more processors 240, one or more communication units 242, one or more input components 244, one or more output components 246, and one or more storage components 248. UIC 212 includes output component 202 and input component 204. The storage component 248 of computing device 210 includes a UI module 220, an assistant module 222, a user information data storage 224, one or more application modules 226, and a context module 230.

[0052] Communication channel 250 can interconnect each of components 212, 240, 242, 244, 246, and 248 to enable inter-component communication (physically, communicatively, and / or operationally). In some examples, communication channel 250 may include a system bus, a network connection, an inter-process communication data structure, or any other method for transmitting data.

[0053] One or more communication units 242 of computing device 210 can communicate with external devices via one or more wired and / or wireless networks by sending and / or receiving network signals on one or more networks. Examples of communication units 242 include network interface cards (e.g., such as Ethernet cards), optical transceivers, radio frequency transceivers, global positioning satellite (GPS) receivers, or any other type of device capable of sending and / or receiving information. Other examples of communication units 242 may include shortwave radios, cellular data radios, wireless network radios, and universal serial bus (USB) controllers.

[0054] One or more input components 244 of computing device 210 can receive input. Examples of input are tactile, audio, and video input. In one example, input component 244 of computing device 210 includes presence-sensitive input devices (e.g., touchscreen, PSD), mice, keyboards, voice response systems, cameras, microphones, or any other type of device for detecting input from a person or machine. In some examples, input component 244 may include one or more sensor components, one or more location sensors (GPS components, Wi-Fi components, cellular components), one or more temperature sensors, one or more motion sensors (e.g., accelerometers, gyroscopes), one or more pressure sensors (e.g., barometers), one or more ambient light sensors, and one or more other sensors (e.g., microphones, cameras, infrared proximity sensors, hygrometers, etc.). Other sensors (to name just a few other non-limiting examples) may include heart rate sensors, magnetometers, glucose sensors, olfactory sensors, compass sensors, and pedometer sensors.

[0055] One or more output components 246 of computing device 110 can generate output. Examples of output are haptic, audio, and video outputs. In one example, the output component 246 of computing device 210 includes a PSD, sound card, video graphics adapter card, speaker, cathode ray tube (CRT) monitor, liquid crystal display (LCD), or any other type of device for generating output to a person or machine.

[0056] The UIC 212 of computing device 210 may be similar to the UIC 112 of computing device 110 and includes an output component 202 and an input component 204. The output component 202 may be a display component, such as a screen on which information is displayed by the UIC 212, and the input component 204 may be a sensitive input component that detects the presence of an object at and / or near the display component 202. The output component 202 and the input component 204 may be a speaker and microphone pair or any other combination of one or more input and output components, such as input component 244 and output component 244. Figure 2 In the examples, UIC 212 can render user interfaces (such as...) Figure 1 User interface 114).

[0057] Although illustrated as an internal component of computing device 210, UIC 212 can also represent an external component that shares a data path with computing device 210 for sending and / or receiving input and output. For example, in one example, UIC 212 represents a built-in component of computing device 210 located within and physically connected to the outer packaging of computing device 210 (e.g., a screen on a mobile phone). In another example, UIC 212 represents an external component of computing device 210 located outside its packaging or housing and physically separated from it (e.g., a monitor, projector, etc., sharing a wired and / or wireless data path with computing device 210).

[0058] One or more storage components 248 within computing device 210 may store information for processing during operation of computing device 210 (e.g., computing device 210 may store data accessed by modules 220, 222, 230, and 226 and data storage 224 during execution at computing device 210). In some examples, storage component 248 is temporary memory, meaning that the primary purpose of storage component 248 is not long-term storage. Storage component 248 on computing device 210 may be configured as volatile memory, storing information for short periods, and therefore the stored contents are not retained if power is lost. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0059] In some examples, storage component 248 also includes one or more computer-readable storage media. In some examples, storage component 248 includes one or more non-transitory computer-readable storage media. Storage component 248 can be configured to store a greater amount of information than information typically stored by volatile memory. Storage component 248 can be further configured to serve as non-volatile storage space, storing information for a long period and retaining information after power-on / power-off cycles. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM). Storage component 248 can store program instructions and / or information (e.g., data) associated with modules 220, 222, 230, and 226 and data storage 224. Storage component 248 may include memory configured to store data or other information associated with modules 220, 222, 230, and 226 and data storage 224.

[0060] One or more processors 240 may implement functions and / or execute instructions associated with computing device 210. Examples of processors 240 include application processors, display controllers, auxiliary processors, one or more sensor hubs, and any other hardware configured to act as a processor, processing unit, or processing device. Modules 220, 222, 230, and 226 may be operated by processor 240 to perform various actions, operations, or functions of computing device 210. For example, processor 240 of computing device 210 may retrieve and execute instructions stored in storage component 248 that cause processor 240 to perform operations on modules 220, 222, 230, and 226. When executed by processor 240, the instructions may cause computing device 210 to store information within storage component 248, for example, at data storage 224.

[0061] UI module 220 may include Figure 1 The UI module 220 contains all the functions of the UI module 120 and can perform operations similar to the UI module 120 to make the assistant function as part of the computing device 210. The UI module 220 can manage the user's interaction with the UIC 212 and other components of the computing device 210. When the user of the computing device 210 views output and / or provides input at the UIC 212, the UI module 220 can make the UIC 212 output a user interface.

[0062] User information data storage 224 is Figure 1 An example of user information data storage 124A. Assistant module 222 may include... Figure 1The assistant module 222 provides all the functions of the computing device 210 and can perform similar operations to the assistant module 122 to execute the assistant as part of the computing device 210 and maintain and access user information in the user information data storage 224. In some examples, the assistant module 222 can execute locally (e.g., at the processor 240) to provide assistant functionality. In some examples, the assistant module 222 can act as an interface to remote assistance services accessible by the computing device 210. For example, the assistant module 222 could be... Figure 1 The interface or application programming interface (API) of the assistant module 122B of the digital assistant server 160.

[0063] One or more application modules 226 represent all the various individual applications and services that execute at and are accessible from computing device 210, which can be accessed by the assistant to provide information to the user and / or perform tasks or actions. Numerous examples of application modules 226 may exist and include fitness apps, calendar apps, search apps, map or navigation apps, travel apps (e.g., planning, booking, ticketing, etc.), social media apps, gaming apps, email apps, chat or messaging apps, internet browser apps, or any other app that can execute on computing device 210. An assistant executing on computing device 210 (such as assistant module 222) may cause application 226 to execute instructions for performing multiple actions associated with the assistant. In other words, the assistant may rely on application module 226 to perform multiple actions on its behalf.

[0064] After receiving explicit consent from the user to store and use personal information (e.g., information stored in data storage 224), the context module 230 can process and analyze context information associated with computing device 210 to define the context of computing device 210 or the context of the user of computing device 210. The context module 230 can encrypt or otherwise process the information being analyzed and / or stored to remove the user's true identity before storing or using the individual. For example, the information can be processed by the context module 230 such that any personally identifiable information is removed when it is stored or sent to a remote computing device (e.g., digital assistant server 160) for processing. If the user explicitly consents to the use or collection of such information, the context module 230 can analyze only the information associated with computing device 210 and / or the user of computing device 210. The context module 230 can further provide the user with an opportunity to withdraw consent, and in this case, the context module 230 can stop collecting or otherwise retain information associated with computing device 210 or the user of computing device 210.

[0065] The context of computing device 210 can be specified at various locations and times, including one or more characteristics associated with computing device 210 and / or its user, as well as his or her physical and / or virtual environment. For example, context module 230 can determine the physical location associated with computing device 210 at a specific time as part of the context of computing device 210 based on context information associated with computing device 210 at that specific time. When the context information (e.g., based on sensor information indicating motion over time) changes, context module 230 can update the determined physical location within the context of computing device 210.

[0066] The types of information defining the context of a computing device at a specific location and / or time are too numerous to list. As some examples, the context of a computing device may specify: acoustic fingerprints, video fingerprints, location, movement trajectories, directions, speeds, institution names, street addresses, location types, buildings, weather conditions, and traffic conditions for different locations and times. The context of a computing device may further include calendar information defining meetings or events associated with each location and time, web page addresses viewed at each location and time, text input (e.g., search or browsing history) in data fields of web pages viewed at each location and time, and other application usage data associated with each location and time. The context of a computing device may further include: information about audio and / or video streams accessed or broadcast at each location and time when the computing device is present, television or cable / satellite broadcasts accessed or broadcast at each location and time when the computing device is present, and information about other services accessed by the computing device at each location and time.

[0067] The context module 230 can share the context of the computing device 210 with the assistant module 222, from which the assistant module 222 can determine whether to output an event notification. The context module 230 can respond to requests from the assistant module 222 regarding the current context or current state associated with the computing device 210 by outputting data to the assistant module 222, which specifies the current state or context of the user.

[0068] In operation, assistant module 222 can determine events for initiating interaction between the user and assistant based on information associated with the user of the computing device. For example, assistant module 222 can monitor data sources, such as data storage 224, to detect changes in information that may indicate events worthy of notification to the user. Examples of data sources include messaging accounts, calendars, social media accounts, and other information sources. Examples of events include appointment additions, cancellations, or changes (e.g., invitations, meetings, anniversaries, etc.), events (e.g., shipping notifications, flight delays), announcements (e.g., tour dates, etc.), and other types of events.

[0069] Data storage 224 may include references to private or public calendars. Assistant module 222 can detect calendar-related events when appointments are added, deleted, modified, fulfilled, or missed. For example, assistant module 222 can determine an event in response to a private calendar setting that a dentist has sent a cancellation notification to cancel an appointment. Assistant module 222 can determine an event in response to a public calendar setting that a holiday, birthday, or anniversary is approaching.

[0070] Data storage 224 may include references to emails or text messages, voice messages, video messages, etc. Assistant module 222 may determine an event based on information included in such communications. For example, assistant module 222 may determine an event in response to an email from a service provider (e.g., an airline, cable provider, etc.) confirming that a scheduled appointment has been canceled or delayed. Alternatively, assistant module 222 may determine an event in response to an email from a retailer confirming that a package has been mailed.

[0071] Assistant module 222 can select at least one action associated with an event from a plurality of actions. For example, the model of assistant module 222 can receive an event or event type as input and output one or more actions associated with that event. The model can be a machine learning type model (e.g., a neural network), a rule-based model, or other type of model that matches actions with events.

[0072] The model can assign a degree of likelihood (e.g., score, probability, etc.) to the relationship between an action and an event and / or the closeness of the relationship between the action and the event. For example, the model can be trained on observations of user activity regarding various events and actions performed by the user that cause the computing device to respond to the events. With the user's explicit consent, the model can analyze user device interactions to develop rules that instruct the user what action to take and when to take such action for a specific event or event type.

[0073] For example, a model can be trained on user interaction data indicating that, in 90% of cases, a user receives a flight delay notification and will message a close friend. It can be further trained on user interaction data indicating that, in 80% of cases, the user will cause the device to send a message to a spouse or significant other. Therefore, the model can create rules indicating that whenever a flight delay notification is output, the recommended action might be to message a spouse or significant other.

[0074] As another example, the model can be trained on user interaction data indicating that, after receiving a score update for the currently playing game, most users on other devices check their Fantasy Sports Score. Furthermore, the model can be trained on user interaction data indicating that, in 80% of cases, users open the popular Fantasy Sports app on their devices, as the primary way for users to check their Fantasy Sports Score. Therefore, the model can create rules indicating that whenever a sports score notification is to be output and the popular Fantasy Sports app is installed on the computing device, the recommended action could be to run the Fantasy Sports app after the score is displayed.

[0075] In some cases, the model can be based on a dependency graph. For example, the model's dependency graph could include a root node that could be a flight. Depending on the root node, it could be a transportation node, an accommodation node, a meeting node, a dining node, etc. The edges between the root node and each of these nodes can be dependencies, such as arrival time, availability of bookings, etc. If it is impossible to traverse all the edges, the model can trigger a suggested action. In some examples, the suggested action itself can be a node in the dependency graph, which has various dependencies (e.g., how late, no alternatives, etc.) and weights, and can provide the user with the highest-weighted possible path (and alternative, lower-weighted actions, if appropriate). The model's graph can be initially presented as a template to which user data can be added, and can be modified by the system (e.g., using machine learning, manual input from the user, etc.). Many other examples of this model exist.

[0076] In any case, when assistant module 222 generates such rules, the input to the rules can be an event or an event type, and the output of each rule can indicate the degree of probability that each available action is associated with a specific event. Assistant module 222 can rank the multiple actions it performs based on the degree of probability assigned to each action to determine the highest-ranked action(s). In other words, assistant module 222 can rank each of multiple actions based on an event and select at least one action with the highest ranking.

[0077] In some examples, if the probability of all actions does not meet a threshold, the assistant module 222 can determine that there are no actions related to the event. For example, if none of the multiple actions that the assistant module 222 can perform have a probability of being associated with a specific event of more than 50%, the assistant module 222 can determine that there are no actions worth recommending during the notification period of the specific event.

[0078] In some examples, if the probability of a single action meets a threshold, the assistant module 222 can determine that a single action is relevant to the event. For example, if only one of several actions that can be performed by the assistant module 222 has a greater than 90% probability of being associated with a specific event, the assistant module 222 can determine that a single action worth recommending exists during the notification of the specific event.

[0079] In some examples, if the probability of two or more actions meets a threshold, the assistant module 222 can determine that there are multiple actions related to the event. For example, if two or more of the multiple actions that can be performed by the assistant module 222 each have a probability greater than 95% that they are associated with a specific event, the assistant module 222 can determine that there are multiple actions worth recommending during the notification of the specific event.

[0080] In some examples, the model used by assistant module 222 to select at least one action associated with an event from multiple actions can be based on a table or graph. For example, assistant module 222 can look up an event or event type in a table or graph and determine which one or more actions associated with the event are included in that table or graph.

[0081] Some assistants may offer a voice-only interface, possibly with minimal documentation, or may otherwise be limited in their ability to teach users what the assistant can do. Discovering the assistant's actions or capabilities can be difficult for some users. Assistant module 222 can be configured to teach the user its capabilities when an action or capability is likely to be beneficial or otherwise advantageous to the user. Assistant module 222 can determine whether to output a notification of an event including a suggested action, in order to increase the likelihood that not only is the notification interesting but the user is also likely to authorize the recommended action suggested by the assistant. In this way, assistant module 222 can suggest truly useful actions when the likelihood of the user rejecting the suggestion is low. Even if the user does not authorize the action, assistant module 222 will still have taught the user the new action.

[0082] For example, assistant module 222 may determine whether to output an event notification based at least in part on the likelihood that the user will accept a request to perform an action associated with the event. Assistant module 222 may determine the likelihood based on the user's context, the context of computing device 222, information stored at user information data storage 224, or a combination thereof, and any other information accessible to assistant module 222.

[0083] For example, assistant module 222 can determine whether to output an event notification based at least in part on how close the previous interaction between the user and the assistant was in time. If the previous interaction occurred within a threshold time period (e.g., five minutes), assistant module 222 can determine that the user is less likely to accept the request to perform an action and is more likely to reject the request. If the previous interaction occurred outside the threshold time period, assistant module 222 can determine that the user is more likely to accept the request to perform an action and is therefore more likely to accept the request.

[0084] Assistant module 222 can determine whether to output an event notification based at least in part on the context of computing device 210. For example, if the context indicates that the user is interacting with computing device 210 (e.g., watching UIC 212 and / or providing input to UIC 212), assistant module 222 can determine that the user is highly likely to accept a request to perform an action, and therefore is more likely to accept the request. If the context indicates that the user is not interacting with computing device 210 (e.g., computing device 210 is coupled to a wireless communication unit of a moving vehicle, computing device 210 is lying flat on a table or in the user's pocket, computing device 210 is in a movie theater, etc.), assistant module 222 can determine that the user is less likely to accept a request to perform an action, and therefore is more likely not to output a notification with that request.

[0085] In some examples, assistant module 222 can assign severity levels to events and increase or decrease the likelihood that a user will accept a request to perform an action based on the severity level. For example, if the event is a flight delay of less than a threshold amount of time (e.g., 1 hour, half an hour, etc.), assistant module 222 can determine that the event is not a serious event based on determining that the event's severity level is low. However, if the flight delay is greater than or equal to the threshold amount of time (e.g., 1 hour, half an hour, etc.), assistant module 222 can determine that the event is a serious event based on determining that the event's severity level is medium or high.

[0086] In some examples, the threshold time amount can be based on the actual event. For example, the threshold time amount could be a percentage of the total flight time (i.e., the duration of the flight itself). If a flight is delayed by 1 hour, but the flight is 10 hours long, the assistant module 222 can assign a low or medium severity level. However, if a flight is delayed by half an hour, and the flight is only 1 hour long, the assistant module 222 can assign a high severity level to the event, making it more likely that a notification including suggested actions (e.g., notifying a friend or spouse that they will be home late) will be output.

[0087] In response to a notification indicating an output event, assistant module 222 may output an event notification via UIC 212 during an interaction between the user and the assistant. The notification may include an indication of the event and a request to perform at least one action associated with the event.

[0088] For example, if assistant module 222 determines that a dental appointment on the calendar has been canceled (e.g., via an email or notification pushed to the user), assistant module 222 can determine to output a cancellation notification. The output could be a voice notification that includes a request to call the dentist to reschedule. In response to receiving user input at UIC 212 instructing the user to have the assistant call the dentist, assistant module 222 can invoke the phone application from application module 226 to call the dentist's office on behalf of the user.

[0089] As another example, if assistant module 222 determines that a flight is delayed by five hours, it can determine to output a cancellation notification. The output could be a voice notification that includes a request to cancel or reschedule ground transportation (such as a driverless car) booked at the destination. In response to receiving user input at UIC 212 indicating that the user wishes the assistant to cancel or reschedule ground transportation, assistant module 222 can send a message to the ground transportation service requesting the cancellation or rescheduling of the booking.

[0090] As another example, if a flight is delayed by five hours, assistant module 222 can determine to output a cancellation notification. The output could be a voice notification, including a phone request, text message, email, or other means of contacting the user's wife. In response to receiving user input at UIC 212 instructing the user to have the assistant contact their wife, assistant module 222 can determine the wife's contact information and send her a message informing her of the delay.

[0091] As another example, if assistant module 222 determines that Valentine's Day is less than a week away, it can determine to output a notification of the event. The output could be a voice notification including a request to order flowers for the user's girlfriend. In response to receiving user input at UIC 212 instructing the user to order flowers, assistant module 222 can determine the girlfriend's contact information and order flowers online for delivery to her address.

[0092] In this way, assistant module 222 can improve the discoverability of features, capabilities, or actions that it can perform, thereby enabling the user of computing device 210 to gain more benefits or uses from existing features. Assistant module 222 teaches actions in the grammar of the user's native language. In some examples, multiple actions are suggested via notification, or a single action is suggested. In response to receiving confirmation from the user that they wish the assistant to take an action, assistant module 222 can perform the action and / or direct the user to a companion application, such as application module 226, from which they can perform the action.

[0093] Figure 3 This is a flowchart illustrating exemplary operations performed by one or more processors executing an exemplary virtual assistant according to one or more aspects of this disclosure. In the following, in Figure 1 The scenario described in the computing device 100 Figure 3 For example, according to one or more aspects of this disclosure, the assistant module 122A may perform operations 300-308 while executing on one or more processors of the computing device 110A.

[0094] In operation, computing device 110A may receive consent from the user for the use and storage of personal information (300). For example, in response to identifying potential personal information, assistant module 122A may cause UI module 120 to request permission from the user to store and use personal information obtained during interactions between assistant module 122A and the user. It should be understood that computing device 110A may not require user consent before each time assistant module 122A wants to use or store personal information. For example, if computing device 110A receives consent once a year, once a day, or even only once (e.g., after initial product purchase, setup, etc.), computing device 110A may consider previous consent as consent for future use and storage of personal information. As an example of how a user can provide consent, the user may provide voice input detected by UIC 212, “yeah that's ok to store information about me,” and in response to this voice input, assistant module 122A may begin maintaining personal records about the user at data storage 122A.

[0095] Computing device 110A can determine the event for initiating an interaction (302) based at least in part on personal information. For example, assistant module 122A can receive a notification (e.g., data) from digital assistant server 160 indicating an event for initiating an interaction between the user and assistant of computing device 110A. The notification may include a message received by the user from a friend via their messaging account. The message may include a question asking the user if he or she wants to go to happy hour.

[0096] The computing device 110A can select at least one action associated with an event (304) based on the event. For example, the model of the assistant module 122A can determine that happy hour is a lateness type of event, and therefore sending a message home to let family members know that the lateness is an action associated with happy hour. In addition, the model of the assistant module 122A can determine that recording live news broadcasts that the user usually watches after get off work is an action associated with happy hour.

[0097] Computing device 110A can determine whether to output an event notification based on at least one action (306). For example, assistant module 122A can disable the generation of event notifications until the context (e.g., time) of computing device 110A when assistant module 122A believes that the user is most likely to want the assistant to perform at least one action. Assistant module 122A can determine, based on context information about computing device 110A, that the user is in a meeting room preparing to start a meeting. Assistant module 122A can determine that the happy hour event is a low-severity event, and because the user is unlikely to want the assistant to perform an action now rather than output an event notification, assistant module 122A should disable the output of the notification until later (308, No).

[0098] At a later time, such as after a meeting, assistant module 122A can determine that the user is in his or her office based on contextual information about computing device 110A. Assistant module 122A can determine that although the happy hour event is a low-severity event, and the user is more likely to want the assistant to perform an action now when alone in his or her office. Therefore, assistant module 122A can determine that the current time is a good time to generate an event notification (308, yes).

[0099] Computing device 110A can output notifications (310). For example, assistant module 122A can determine that the user is interacting with computing device 110B (e.g., a laptop computer) while in his or her office. Computing device 110A can send a happy hour notification including suggestions to send a message home to let family know he or she is late, and to record a live news broadcast that the user would normally watch after get off work. Computing device 110B can display a graphical indication of the notification reminding the user of the happy hour invitation, as well as selectable buttons for the user to choose whether to have the assistant send a message home and / or record the news on a home digital video recorder.

[0100] Following the output notification (310), computing device 110A may receive data (312) from computing device 110B indicating that the user has agreed to the assistant module 122A performing one or two actions suggested by the event notification. For example, computing device 110B may share an instruction received from the user indicating that the user wants the assistant to send a message home and / or record news.

[0101] In response to receiving user input, assistant module 122A can perform an action (314) agreed to by the user. For example, assistant module 122A can invoke an application running on at least one processor configured to perform the action, which is different from the assistant. For example, to send a message home, assistant module 122A can send data to a messaging application that sends the message to the user's family via the user's messaging account.

[0102] In some examples, assistant module 122A can perform this action. For instance, to control a digital video recorder, assistant module 122A can send data to the API of a remote control application configured to control the digital video recorder. Assistant module 122A can directly configure the video recorder to record news broadcasts by sending input to the API.

[0103] Figure 4 This is a block diagram illustrating an exemplary computing system configured to perform an exemplary virtual assistant according to one or more aspects of this disclosure. In the following, it will be referred to as... Figure 4 Digital Assistant Server 460 is described as Figure 1 Example of a digital assistant server 160. Figure 4 The digital assistant server 460 is shown as only one specific example, and many other examples of the digital assistant server 460 may be used in other instances, as well as subsets of the components included in the exemplary digital assistant server 460, or may include... Figure 4 Additional components not shown.

[0104] like Figure 4 As shown in the example, the digital assistant server 460 includes one or more processors 440, one or more communication units 442, and one or more storage devices 448. The storage device 448 includes an assistant module 422, a user information data storage 424, and a search module 482.

[0105] Processor 440 is similar to Figure 2 The processor 240 of the computing system 210. The communication unit 442 is similar to... Figure 2 The communication unit 242 of the computing system 210. The storage device 448 is similar to... Figure 2 The computing system 210 has a storage device 248. The communication channel 450 is similar to... Figure 2 The computing system 210 has a communication channel 250, which can therefore interconnect each of components 440, 442, and 448 for inter-component communication. In some examples, the communication channel 450 may include a system bus, a network connection, an inter-process communication data structure, or any other method for transmitting data.

[0106] The search module 482 of the digital assistant server 460 is similar to the search module 182 of the search server system 180 and can perform integrated search functions on behalf of the digital assistant server 460. That is, the search module 482 can perform search operations on behalf of the assistant module 422. In some examples, the search module 482 can connect to an external search system (such as search system 180) to perform search operations on behalf of the assistant module 422. When invoked, the search module 482 can perform search functions (such as generating search queries and performing searches based on search queries generated on various local and remote information sources). The search module 482 can provide the results of the performed search to the calling component or module. That is, the search module 482 can output search results to the assistant module 422.

[0107] User information data storage 424 is similar to Figure 2 User information data storage 224 is configured to store information about the user about the computing device that the assistant module 422 has learned about the user during a session between the user and the assistant provided by the assistant module 422. The assistant module 422 may rely on the information stored at data storage 424 to determine events and select events-related actions to include in a notification sent by the assistant module 422.

[0108] Assistant module 422 may include Figure 1 Assistant module 122 and Figure 2 The assistant module 222 of the computing device 210 performs some or all of the functions of the assistant module 222. The assistant module 422 can perform operations similar to those of assistant modules 122 and 222, providing assistant services accessible via a network such as network 130. That is, the assistant module 422 can act as an interface to remote assistant services accessible to the computing device and communicating with the digital assistant server 460 on the network. For example, the assistant module 422 could be... Figure 1 The interface or API of the assistant module 122B of the digital assistant server 160.

[0109] In operation, the assistant module 422 can analyze information contained in the user information data repository 424 and information received from the search module 482 for events worthy of notification to the user. When the assistant module 422 detects an event and selects a recommended action related to the event, the assistant module 422 can select the notification to trigger the event based on a combination of various factors, including how often the assistant module 422 initiates interaction with the user and the probability of successful interaction.

[0110] Assistant module 422 can generate "surface-appropriate" notifications that are more likely to attract the user's attention. For example, if the user is associated with a home assistant device and has recently interacted with the home assistant, assistant module 422 can enable the home assistant to generate a vent-based voice notification. The voice notification can be generated in the language of the home assistant device. For example, a voice notification can be sent to the home device to generate something like, "Hello Viki, your flight has been delayed by three hours. Should I cancel the limo reservation?" If the user is associated with a mobile phone and interacts with the assistant via the mobile phone, assistant module 422 can send a notification to the mobile phone that graphically displays a "cancel" button for canceling the limo reservation on the screen. In some cases, such as for high-value events or events with a high severity level, assistant module 422 can send notifications with actions to multiple surfaces simultaneously.

[0111] Clause 1. A method comprising: determining, based on information associated with a user of a computing device, an event for initiating an interaction between the user and an assistant executing on the computing device, wherein the assistant is configured to perform a plurality of actions; selecting, based on the event, at least one action associated with the event from the plurality of actions; determining, based on the at least one action, whether to output a notification of the event, wherein the notification includes an indication of the event and a request to perform the at least one action associated with the event; and, in response to determining the notification to output the event, sending the notification of the event to the assistant for output during the interaction between the user and the assistant.

[0112] Clause 2. The method as described in Clause 1 further comprises: scoring each of the plurality of actions based on the event; and selecting at least one action having a corresponding score that satisfies a threshold.

[0113] Clause 3. The method as described in any of Clauses 1 to 2, wherein determining whether to output a notification of the event comprises determining whether to output a notification of the event based at least in part on a previous interaction that recently occurred between the user and the assistant.

[0114] Clause 4. The method of any one of Clauses 1 to 3, wherein determining whether to output a notification of the event comprises: determining the degree of likelihood that the user will accept a request to perform at least one action associated with the event; and determining whether to output a notification of the event based at least in part on the degree of likelihood that the user will accept a request to perform at least one action associated with the event.

[0115] Clause 5. The method as described in Clause 4 further includes: determining the severity level of the event; and further determining, based on the severity level of the event, whether to output a notification of the event.

[0116] Clause 6. The method of any one of Clauses 1 to 5, wherein determining whether to output a notification of the event comprises determining whether to output a notification of the event based at least in part on a combination of how close in time the previous interaction between the user and the assistant was and the frequency of the previous interaction between the user and the assistant.

[0117] Clause 7. The method as described in any one of Clauses 1 to 6, wherein the at least one action associated with the event is associated with an application configured to perform the at least one action, which is different from the assistant.

[0118] Clause 8. The method as described in any one of Clauses 1 to 7, wherein the at least one action is a single action.

[0119] Clause 9. The method as described in any one of Clauses 1 to 8, wherein the at least one action comprises multiple actions.

[0120] Clause 10. A computing device, comprising: an output device; at least one processor; and at least one memory including instructions, which, when executed, cause the at least one processor to execute an assistant, the assistant being configured to: determine, based on information associated with a user of the computing device, an event for initiating an interaction between the user and the assistant, wherein the assistant is configured to perform a plurality of actions; select, based on the event, at least one action associated with the event from the plurality of actions; and determine, based on the at least one action, whether to output a notification of the event, wherein the notification includes an indication of the event and a request to perform the at least one action associated with the event; and

[0121] In response to a notification that the event has been determined to be output, the notification of the event is output via the output device during the interaction between the user and the assistant.

[0122] Clause 11. The computing device as described in Clause 10 further includes an input device, wherein the assistant is further configured to:

[0123] The system receives user input via the input device, instructing the assistant to perform at least one action.

[0124] In response to receiving the user input, perform the at least one action.

[0125] Clause 12. The computing device as described in Clause 10 or 11, wherein the assistant is further configured to perform the at least one action by invoking an application configured to perform the at least one action, which is different from the assistant, and which is executed at the at least one processor.

[0126] Clause 13. The computing device as described in any one of Clauses 10 to 12, wherein the at least one action comprises a plurality of actions.

[0127] Clause 14. The computing device as described in Clause 13 further includes an input device, wherein the assistant is further configured to: receive, via the input device, user input commanding the assistant to perform a specific action among a plurality of actions; and, in response to receiving the user input, perform the specific action.

[0128] Clause 15. The computing device as described in any one of Clauses 10 to 14, wherein the assistant is further configured to receive events from a remote computing system configured to determine events for initiating an interaction between the user and the assistant.

[0129] Clause 16. A computing device as described in any one of Clauses 10 to 15, wherein the output device is configured to output audio, and the notification is a voice notification.

[0130] Clause 17. A computer-readable storage medium including instructions that, when executed, cause at least one processor of a digital assistant system to: determine, based on information associated with a user of a computing device, an event for initiating an interaction between the user and an assistant executing on the computing device, wherein the assistant is configured to perform a plurality of actions; select, based on the event, at least one action associated with the event from the plurality of actions; determine, based on the at least one action, whether to output a notification of the event, wherein the notification includes an indication of the event and a request to perform the at least one action associated with the event; and, in response to determining the notification to output the event, send the notification of the event to the assistant for output during the interaction between the user and the assistant.

[0131] Clause 18. The computer-readable storage medium as described in Clause 17, wherein the instructions, when executed by the at least one processor, cause the digital assistant system to determine whether to output a notification of the event by at least partially based on how close in time a previous interaction between the user and the assistant is in time.

[0132] Clause 19. A computer-readable storage medium as described in any one of Clauses 17 to 18, wherein the instructions, when executed by the at least one processor, cause the digital assistant system to determine whether to output a notification of the event, at least by determining the degree of probability that the user will accept a request to perform at least one action associated with the event; and by determining whether to output a notification of the event, at least in part based on the degree of probability that the user will accept a request to perform at least one action associated with the event.

[0133] Clause 20. A computer-readable storage medium as described in any one of Clauses 17 to 19, wherein the at least one action is associated with a third device different from the computing device and the digital assistant system.

[0134] Clause 21. A computing device comprising at least one processor and at least one memory including instructions, which, when executed, cause the at least one processor to perform the method as described in any one of Clauses 1 to 9.

[0135] Clause 22. A computer-readable storage medium comprising instructions that, when executed, cause at least one processor of a computing device to perform the method as described in any one of Clauses 1 to 9.

[0136] Clause 23. A system comprising means for performing the method as described in any one of Clauses 1 to 9.

[0137] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium that facilitates the transfer of a computer program from one place to another, for example, according to a communication protocol. In this way, the computer-readable medium may generally correspond to (1) a non-transient tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0138] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection is appropriately referred to as a computer-readable medium. For example, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium if instructions are sent from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave). However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. As used herein, disks and optical discs include optical discs (CDs), laser discs, optical discs, digital universal discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs reproduce data using laser optics. Combinations of the above should also be included within the scope of computer-readable media.

[0139] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, as used herein, the term "processor" can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules. Furthermore, these techniques can be implemented entirely within one or more circuit or logic elements.

[0140] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but they do not necessarily need to be implemented by different hardware units. Rather, as described above, various units can be combined within hardware units, or provided by a collection of interoperable hardware units including one or more processors as described above, in conjunction with appropriate software and / or firmware.

[0141] Various embodiments have been described. These and other embodiments are within the scope of the appended claims.

Claims

1. A method implemented by one or more processors, the method comprising: An event is determined based on information associated with a user of the computing device to initiate an interaction between the user and an assistant that is configured to perform multiple actions. Based on the event, at least one action associated with the event is selected from the plurality of actions; Determining whether to output a notification of the event based on how close the previous interactions between the user and the assistant are in time and based on at least one selected action, including avoiding outputting the notification of the event when it occurs, wherein the notification includes an indication of the event and a request to perform the at least one action associated with the event; and In response to determining that the notification of the event is to be output, the notification of the event is sent to the assistant, wherein sending the notification of the event causes the assistant to proactively output the notification of the event without the user requesting the notification of the event.

2. The method according to claim 1, further comprising: Each of the plurality of actions is scored based on the event; The selection of the at least one action is based on a corresponding score that satisfies a threshold.

3. The method according to claim 1, wherein, The determination of whether to output the notification of the event is further based on: The degree of likelihood that the user will accept the request to perform at least one action associated with the event.

4. The method according to claim 1, wherein, Whether to output the notification for the event is further based on the severity level of the event.

5. The method according to claim 1, wherein, The at least one action associated with the event is associated with an application configured to perform the at least one action, and wherein the application is different from the assistant.

6. The method according to claim 1, wherein, The at least one action is a single action.

7. The method according to claim 1, wherein, The at least one action includes multiple actions.

8. A computing device, comprising: Output devices; At least one processor; as well as At least one memory including instructions, which, when executed, cause the at least one processor to execute an assistant, the assistant being configured to: An event is determined based on information associated with a user of the computing device to initiate an interaction between the user and an assistant executed on the computing device, wherein the assistant is configured to perform multiple actions; Based on the event, at least one action associated with the event is selected from the plurality of actions; Determining whether to output a notification of the event based on how close the previous interactions between the user and the assistant are in time and based on at least one selected action, including avoiding outputting the notification of the event when it occurs, wherein the notification includes an indication of the event and a request to perform the at least one action associated with the event; and In response to determining that the notification of the event should be output, the assistant proactively outputs the notification of the event without the user requesting the notification of the event.

9. The computing device according to claim 8, wherein, The assistant is further configured to: Each of the plurality of actions is scored based on the event; When selecting the at least one action, the assistant is configured to select the at least one action based on a corresponding score that meets a threshold.

10. The computing device according to claim 8, wherein, When determining whether to output the notification of the event, the assistant further determines whether to output the notification of the event based on the following: The degree of likelihood that the user will accept the request to perform at least one action associated with the event.

11. The computing device according to claim 8, wherein, When determining whether to output the notification for the event, the assistant further determines whether to output the notification for the event based on the severity level of the event.

12. The computing device according to claim 8, wherein, The at least one action associated with the event is associated with an application configured to perform the at least one action, and wherein the application is different from the assistant.

13. The computing device according to claim 8, wherein, The at least one action is a single action.

14. The computing device according to claim 8, wherein, The at least one action includes multiple actions.

15. A computer-readable storage medium storing instructions, said instructions causing at least one processor of a digital assistant system to: An event is determined based on information associated with a user of the computing device to initiate an interaction between the user and an assistant operating on the computing device, wherein, The assistant is configured to perform multiple actions; Based on the event, at least one action associated with the event is selected from the plurality of actions; Determining whether to output a notification of the event based on how close the previous interactions between the user and the assistant are in time and based on at least one selected action, including avoiding outputting the notification of the event when the event occurs based on how close the previous interactions between the user and the assistant are in time, wherein the notification includes an indication of the event and a request to perform the at least one action associated with the event; as well as In response to determining that the notification of the event should be output, the assistant proactively outputs the notification of the event without the user requesting the notification of the event.

16. The computer-readable storage medium according to claim 15, wherein, The at least one action associated with the event is associated with an application configured to perform the at least one action, and wherein the application is different from the assistant.

17. A method implemented by one or more processors of a user's computing device, the method comprising: The digital assistant maintains information associated with the user in the data storage, and the information associated with the user can be accessed by the digital assistant; The digital assistant is configured to submit a search query for information associated with the user through one or more databases, wherein the digital assistant is executed at least in part on the user's computing device, and wherein the digital assistant submits the search query on behalf of the user of the computing device through one or more databases. In response to causing the digital assistant to submit the search query through one or more databases in the database, a response content in response to the search query is obtained; Based on context information associated with the computing device, determine whether to allow the digital assistant to participate in a conversation with the user of the computing device to output the response content in response to the search query; and In response to determining that the digital assistant should engage in the session with the user of the computing device to output the response content in response to the search query: This enables the digital assistant to participate in the conversation between the user of the computing device; and During the session: This ensures that the digital assistant outputs at least the response content in response to the search query.

18. The method according to claim 17, wherein, The context information associated with the computing device indicates that the computing device is moving.

19. The method according to claim 17, wherein, The digital assistant is required to output at least the response content in response to the search query, including: The digital assistant outputs the response content in response to the search query at least audibly via one or more speakers of the computing device.

20. The method of claim 17, further comprising: Based on the context information associated with the computing device, and at a first time instance among multiple time instances, it is determined to avoid participating in the session with the user of the computing device in order to output the response content in response to the search query.

21. The method of claim 20, further comprising: Based on the context information associated with the computing device, and in the plurality of time instances and at a second time instance following the first time instance, the session of the user participating with the computing device is determined to output the response content in response to the search query.

22. The method according to claim 17, wherein, The one or more databases include remote databases located away from the user's computing device.

23. The method according to claim 17, wherein, The one or more databases include databases located locally on the user's computing device.

24. A user's computing device, the computing device comprising: At least one hardware processor; as well as A memory storing instructions, which, when executed by the at least one hardware processor, cause the at least one hardware processor to execute an assistant, the assistant being configured to: The digital assistant maintains information associated with the user in the data storage, and the information associated with the user can be accessed by the digital assistant; The digital assistant is configured to submit a search query for information associated with the user through one or more databases, wherein the digital assistant is executed at least in part on the user's computing device, and wherein the digital assistant submits the search query on behalf of the user of the computing device through one or more databases. In response to causing the digital assistant to submit the search query through one or more databases in the database, a response content in response to the search query is obtained; Based on context information associated with the computing device, determine whether to allow the digital assistant to participate in a conversation with the user of the computing device to output the response content in response to the search query; and In response to determining that the digital assistant should engage in the session with the user of the computing device to output the response content in response to the search query: This enables the digital assistant to participate in the conversation between the user of the computing device; and During the session: This ensures that the digital assistant outputs at least the response content in response to the search query.

25. The computing device according to claim 24, wherein, The context information associated with the computing device indicates that the computing device is moving.

26. The computing device according to claim 24, wherein, The instructions that cause the digital assistant to output at least the response content in response to the search query include instructions for the following: The digital assistant outputs the response content in response to the search query at least audibly via one or more speakers of the computing device.

27. The computing device according to claim 24, wherein, The instructions further cause the at least one hardware processor to: Based on the context information associated with the computing device, and at a first time instance among multiple time instances, it is determined to avoid participating in the session with the user of the computing device in order to output the response content in response to the search query.

28. The computing device according to claim 27, wherein, The instructions further cause the at least one hardware processor to: Based on the context information associated with the computing device, and in the plurality of time instances and at a second time instance following the first time instance, the session of the user participating with the computing device is determined to output the response content in response to the search query.

29. The computing device according to claim 24, in, The one or more databases include remote databases located away from the user's computing device, or The one or more databases include databases located locally on the user's computing device.

30. A non-transitory computer-readable storage medium storing instructions, said instructions causing said at least one hardware processor to perform operations when executed by said at least one hardware processor, said operations including: The digital assistant maintains information associated with the user in the data storage, and the information associated with the user can be accessed by the digital assistant; The digital assistant is configured to submit a search query for information associated with the user through one or more databases, wherein the digital assistant is executed at least in part on the user's computing device, and wherein the digital assistant submits the search query on behalf of the user on the computing device through one or more databases. In response to causing the digital assistant to submit the search query through one or more databases in the database, a response content in response to the search query is obtained; Based on context information associated with the computing device, determine whether to allow the digital assistant to participate in a conversation with the user of the computing device to output the response content in response to the search query; and In response to determining that the digital assistant should engage in the session with the user of the computing device to output the response content in response to the search query: This enables the digital assistant to participate in the conversation between the user of the computing device; and During the session: This ensures that the digital assistant outputs at least the response content in response to the search query.

Images