Using large language models in reducing degree of calendar-related interactions

Abstract

Some implementations process structured schedule data of an electronic schedule of a first user to generate a natural language representation of the structured schedule data. Each version of those implementations further launches a large language model (LLM) using a launch input (e.g., processing the launch input using the LLM) in response to receiving a query determined to be related to the electronic schedule, where the launch input is based on a natural language representation of the structured schedule data. After the LLM is launched using the launch input, some of those versions process the query-based query input using the LLM to generate an LLM output, and determine a response to the query based on the LLM output. The response may include a natural language response that may be rendered.

Description

BACKGROUND OF THE INVENTION

[0001] In electronic communications in which a user receives or sends emails and / or messages or makes calls, the user is often requested to provide (or confirm) their availability during a specific time period on a specific day for events such as training courses, appointments, office meetings, holiday parties, gatherings with friends, or dinners at a restaurant. For example, the user may receive an email asking "Are you free next Thursday for a meeting?" To confirm availability and / or reply with one or more preferred time slots, the user may need to check his or her schedule to see if anything is already scheduled for next Thursday. As another example, as a close friend of Sarah, the user may be asked during a call "When is Sarah’s birthday?" However, although vaguely remembering that Sarah's birthday is in May, the user may not be very sure if Sarah's birthday is exactly May 18th or May 19th. In this example, the user can check his or her schedule to see if the date is marked for Sarah's birthday, or if there are old messages showing previous communications of a birthday party planned for Sarah last year, the user can look up the old messages.

[0002] In these and other examples, without accurate availability information to assist in scheduling events, users are unable to formulate an immediate response. Thus, the user's response will be delayed until they have gathered the desired information by searching their schedule and / or messages containing schedule data. In addition to the delay in response, a significant amount of resources of the user's client device will be consumed while gathering the necessary information. For example, when the user is searching for and / or viewing the necessary information, the battery resources of the client device can be consumed, the processor resources of the client device can be consumed in searching for and / or rendering the necessary information, and so on. Further, in these and other examples, when the requesting user is trying to schedule a meeting with the receiving user or otherwise inquire about the receiving user's schedule, the receiving user (or a third party on behalf of the receiving user) must utilize the client device resources in formulating a response to the requesting user. SUMMARY OF THE INVENTION

[0003] The implementations disclosed herein relate to using a large language model (LLM) to respond to queries related to a user's electronic calendar after using calendar data (structured or unstructured) associated with at least one user to initiate the LLM. For example, after using the calendar data to initiate the LLM and then using the initiated LLM to process a query, a response to the query can be generated based on the output from the LLM. Some of those implementations generate a natural language representation of the calendar data and initiate the LLM based on the natural language representation.

[0004] Some of the implementations disclosed herein additionally or alternatively relate to receiving one or more messages (text or audio) and determining whether the one or more messages include a query related to an electronic calendar (e.g., the calendar of the user who created the message and / or the calendar of the user to whom the message is addressed). When the query is determined to be related to the electronic calendar, the LLM can be used to process the query after first initiating the LLM with calendar data from the electronic calendar. Further, a response to the query can be generated based on the LLM output generated after processing the query. For example, the LLM output can include a probability distribution of words, word pieces, and / or actions (e.g., remote procedure call RPC), and the response can be generated based on the probability distribution. The response can be rendered visually and / or auditorily to the user who created the message or the user to whom the message is addressed. For example, the response can be rendered to the user who created the message without any input from the receiving user to whom the query is addressed, thereby avoiding any use of their client device by the receiving user for formulating a response - and thereby saving resources of their client device. As another example, the response can be rendered to the receiving user to whom the query is addressed as an optional suggested reply, and if selected (e.g., via a single tap or other single input), can be sent as a reply to the query. This can avoid any use of their client device by the receiving user for looking up the calendar application and formulating a response - thereby saving resources of their client device.

[0005] The various implementations disclosed herein relate to retrieving and processing structured calendar data of an electronic calendar (e.g., calendar entries, sometimes also referred to as "entries" or "entries of the calendar") to generate a natural language representation for initiating the LLM as described herein. For example, a client device (or a remote server) can access the electronic calendar to obtain one or more calendar entries (e.g., entries within a specified month of a specified user's work calendar), and process the one or more calendar entries to generate a natural language representation of the one or more calendar entries.

[0006] In some implementations or scenarios, the client device may process an entire electronic schedule to generate a natural language representation of the entire electronic schedule. In some other implementations or scenarios, the client device may process the entries of the electronic schedule within a selected date range to generate a natural language representation of the entries of the electronic schedule within the selected date range. In some of those implementations, the selected date range is selected based on the query for which the schedule data is being processed. For example, only Bob's "next week" schedule data may be processed in response to a query from Sue, "Does Bob have 30 minutes for a meeting next week to discuss project X", while Bob's schedule for the next three weeks (or other default time period) may be processed in response to a query from Sue, "Find 30 minutes for a meeting with Bob to discuss project X", which does not specify a time period for the meeting. Optionally, the client device may process each entry within the selected date range separately to generate a corresponding natural language representation. The corresponding natural language representations may optionally be combined (e.g., sequentially) to form a single natural language representation.

[0007] The various implementations disclosed herein relate to receiving one or more messages (text, graphics, animations, or audio) and determining whether one or more messages include a query related to an electronic calendar. For example, a client device can at least selectively monitor (e.g., via an automated assistant) one or more applications (e.g., a messaging application and / or a social media application) for electronic communications (e.g., messages, files, screenshots) with appropriate permissions and determine (e.g., in real time) whether a query is detected from the electronic communications. The client device (or a remote server) can directly determine whether a query related to an electronic calendar is detected from the electronic communications. To reduce or save computing resources, the client device can first determine whether a query is detected from the electronic communications, and if a query is detected, the client device can further determine whether this query is related to an electronic calendar accessible to the client device (or a remote server with which the client device communicates). Determining whether a query is related to an electronic calendar (or sometimes pointing to an electronic calendar) can be performed using one or more machine learning models trained on one or more keywords (e.g., time items) or one or more sentence structures, and / or can be performed using other techniques or processes. Optionally, for an electronic calendar that the client device cannot access, in some cases, the client device can generate a request to access the electronic calendar and send the request to an administrator (either the owner of the electronic calendar or a person authorized to manage the electronic calendar) to obtain permission to access the electronic calendar.

[0008] The various implementations disclosed herein relate to generating a natural language representation of structured schedule data for an electronic schedule before using a natural language representation to initiate an LLM and then using the initiated LLM to process a received query. Generating the natural language representation in advance can reduce the latency of using the initiated LLM to process the received query because the natural language representation has already been generated and can be used immediately to initiate the LLM before processing the query. Optionally, in response to receiving a query after generating a natural language representation of structured schedule data for an electronic schedule, a client device (or remote server) can determine whether to update the electronic schedule with additional structured schedule data after generating the natural language representation for the structured schedule data but before initiating the LLM. If the electronic schedule is determined to be updated with additional structured schedule data, the client device (or remote server) can generate an updated natural language representation of the structured schedule data (including the additional structured schedule data) for the electronic schedule such that the natural language representation used to initiate the LLM is up-to-date. However, if the electronic schedule is determined to not have been updated, the previously generated natural language representation can be used to initiate the LLM. In these and other ways, reduced latency due to pre-generation of the natural language representation (e.g., when it is determined that the electronic schedule has not been updated since the natural language representation was generated) can be at least selectively achieved while still achieving selective updating of the natural language representation and improved output generation due to the update of the natural language representation. It should be noted that determining whether the electronic schedule is updated with additional structured schedule data can be performed with less latency than generating the natural language representation.

[0009] In some implementations, initiating an LLM using a natural language representation of structured schedule data may include: generating an initiation input based on the natural language representation of structured schedule data and using the LLM to process the initiation input, where the initiation input is used as the input to the LLM. Intermediate outputs of the LLM may be generated based on processing the initiation input, but are optionally not directly used to generate a response, as described herein. Instead, the LLM model may be used to further process the intermediate outputs. However, the LLM output generated by the initiated LLM may only be used to generate a response after processing a query. As a non-limiting example, the initiation input may include only structured schedule data. For example, the structured schedule data may include: an electronic schedule; a portion of the electronic schedule (although this portion does not include schedule entries); one or more entries of the schedule; columns in a table representing the schedule, where the columns reflect schedule entries on a particular day; and / or a table reflecting schedule entries for an entire week, etc. Alternatively, the initiation input may include only the natural language representation of structured schedule data (e.g., May 25, 2015 is Memorial Day, a federal holiday). Alternatively, as a supplement to or in place of the natural language representation of structured schedule data, the initiation input may include structured schedule data.

[0010] Alternatively or additionally, the initiation input may include unstructured data. For example, the initiation input may include structured schedule data and / or unstructured data. As another example, the initiation input may include the natural language representation of structured schedule data and / or unstructured data. As a further example, the initiation input may include structured schedule data, the natural language representation of structured schedule data, and / or unstructured data. The unstructured data may indicate a user's activities, preferences, availability, relationships, or plans, etc. The unstructured data may be from an electronic schedule or may be from any other available source, such as a text message (e.g., the text message confirms an online restaurant order to be picked up at 11:45 am), an image (e.g., the image is captioned Sarah's birthday party and the date is May 18, 2020), an email (e.g., the email contains an electronic receipt showing a reservation for movie tickets), or a travel planning application (e.g., the travel planning application shows an upcoming itinerary). When the unstructured data is from an electronic schedule or is stored as part of the metadata of an electronic schedule (but is not stored in a structured data structure such as a table, matrix, one or more entries with a fixed format, etc. of the electronic schedule), such unstructured data may be referred to as "unstructured schedule data".

[0011] As non-limiting examples, unstructured data may be or may include: (1) a message (e.g., “doctor appointment confirmed for this Wednesday 4pm,” “Ben leaves work for vacation the first week of June every year,” etc.) or a conversation history containing messages, a piece of information from a diary (e.g., “Kim is going to Alaska next February”), (2) post content showing activity information at a certain moment or within a certain time period (e.g., “Me in Pittsburgh now visiting family”), (3) website page content (e.g., “tickets for pop star A's concert in Seattle will be on sale starting August 8th, 2pm EST (Tickets for Pop Star A's concert in Seattle will go on sale at 2 p.m. EST on August 8)"), the website page content describes information about the event and is not stored as structured calendar data in the aforementioned electronic calendar or in other structured databases. Alternatively, messages and / or conversation histories may also be referred to herein as "unstructured message data", and post content may also be referred to herein as "unstructured post data", and website page content may also be referred to herein as "unstructured web content".

[0012] Alternatively or in addition, the unstructured data may also be or include, for example, preference data stored in association with an electronic calendar rather than being stored in one or more entries of the electronic calendar. For example, a user may configure the system settings of an electronic calendar to indicate a preference for certain activities / events within a certain (certain) time session (e.g., the CFO prefers meetings at 3 p.m. on Tuesdays), and the user's preferences may be stored in preference data (referred to as "unstructured calendar data") as part of the metadata of the electronic calendar. It should be noted that preference data need not be data stored in association with an electronic calendar. For example, preference data may be part of metadata associated with an application (e.g., a virtual meeting) other than a calendar application (which provides one or more electronic calendars).

[0013] Alternatively or additionally, unstructured data can also be or include: routine data indicating a user's routine practices that is not created as a recurring entry in an electronic calendar. For example, the routine data can be alarm data that triggers a wake-up alarm at 7:00 am every weekday, or smart device routine data configured to turn off the lights in the office at 6 pm every day, etc. Alternatively or additionally, unstructured data can also include user input (such as spoken words captured by an automated assistant specifying "I want to set up the meeting this week, if not, next week, but no later than April 30th"), or conversation history.

[0014] Alternatively or additionally, unstructured data can be or can include: relationship data indicating the relationship between a first user (i.e., the "requesting user") who sends a query and a second user (i.e., the "receiving user") who receives the query. For example, the relationship data can indicate that the first user and the second user are colleagues, family members, close friends, people who rarely communicate, classmates, mentor and trainee, teacher and student, and / or can indicate additional or alternative relationships. The relationship data can be retrieved, for example, from a company directory listing the names, titles, departments, and contact information of individuals working in the company. Such relationship data can be included in the initiation input together with structured calendar data (and / or the natural language representation of structured calendar data). Alternatively, the relationship data can be processed to generate a corresponding natural language representation (e.g., "it is the boss who requests the meeting", or "the request sender is uncle Bob"), and the initiation input can include the natural language representation of the relationship data to replace the relationship data itself (or as a supplement to it).

[0015] In some implementations, the relationship data can optionally be applied to determine whether a query needs to be processed (e.g., using an LLM) to generate a response after the query is determined to be related to the electronic calendar. For example, if the relationship data indicates that the requesting user is not within a network (e.g., the network of colleagues, friends, customers, classmates, etc.), a prompt (such as, for a query requesting access to your calendar from an unknown source, do you want to check the sender?) that requires confirmation or input from the user before using the LLM to process the query (or even before using the aforementioned initiation input to initiate the LLM) can be generated.

[0016] Optionally, the activation input may include a first activation input and a second activation input. As a non-limiting example, the first activation input may include a natural language representation of structured schedule data, and the second activation input may include the structured schedule data itself without the natural language representation of the structured schedule data. The present disclosure is not limited thereto. For example, the first activation input may include a natural language representation of structured schedule data, and the second activation input may include unstructured data (or a natural language representation of unstructured data). For example, the activation input may further include a third activation input, where the first activation input includes a natural language representation of structured schedule data, the second activation input includes the structured schedule data itself, and the third activation input includes a natural language representation of unstructured data. Optionally, the first activation input may include a natural language representation of structured schedule data corresponding to a first time range (e.g., next week), and the second activation input may include a natural language representation of structured schedule data corresponding to a second time range (e.g., next month).

[0017] In some implementations, after starting the LLM using the natural language representation of structured schedule data, the LLM can be used to process the aforementioned queries determined to be related to the electronic schedule to generate an LLM output. Turning to the aforementioned case where the start input includes a first start input and a second start input different from the first start input. The LLM model can first be started using the first start input. The queries determined to be related to the electronic schedule can then be processed using the started LLM (started using the first start input) to generate a first LLM output. The LLM model can then be reset and started using the second start input, or the LLM model can be started using the second start input without being reset. Then, the queries can be processed using the started LLM (started using the second start input) to generate a second LLM output. For example, the first start input can include the natural language representation of the electronic schedule corresponding only to week 5 (e.g., "next week, Bob is available 2-3pm on Friday"), and the second start input can include the natural language representation of the electronic schedule corresponding to weeks 5-7 (e.g., "next week, Bob is available 2-3pm on Friday, for the week after next week, Bob is available on Monday 9-10am but will be away afterwards for the rest of the two weeks"). The first LLM output and the second LLM output can be different, even though they are generated based on processing the same queries using the LLM. This is because the first LLM output is generated after starting the LLM using the first start input, while the second LLM output is generated after alternatively or additionally starting the LLM using the second start input.

[0018] In the above example, the LLM can be launched using the first launch input, and the LLM launched using the first launch input can be applied to process the query to generate a first LLM output. Based on the first LLM output, a first response can be generated (e.g., "invite Bob for a meeting next Friday at 2-3pm?"). Optionally, the first response can be rendered to the user in response to the query (e.g., the user audibly asks the automated assistant, "when is Bob available for a meeting to discuss project X?"), and the user can confirm or select the first response. Optionally, in response to the user confirming or selecting the first response (e.g., "invite Bob for a meeting next Friday at 2-3pm?"), the automated assistant can generate a link that causes a calendar entry for a meeting at 2-3pm next Friday to be created, or can generate an email containing such a link and a description (e.g., "Jerry invites you for a meeting next Friday at 2-3pm") that points to or is sent to Bob.

[0019] Additionally, in the above examples where the LLM is to additionally or alternatively be initiated using a second initiation input, the LLM can be reset (or calibrated) after being initiated using the first initiation input. The reset LLM can be initiated using the second initiation input and then process a query to generate a second LLM output, based on which a second response (e.g., "invite Bob for a meeting next Friday at 2-3pm, or the week after on Monday 9-10am?") can be generated. Optionally, instead of the first response, the second response can be rendered to the user in response to the query (e.g., the user audibly asks the automated assistant, "when is Bob available for a meeting to discuss project X?"), where the second response can include two selectable elements (e.g., the first selectable element corresponds to "next Friday at 2-3pm" and the second selectable element corresponds to "the week after on Monday 9-10am"). The user can select one of the two selectable elements, which triggers the generation and / or sending of a link, invitation, or email containing the content of the selected selectable element. In some implementations, the second response and the first response can be ranked based on one or more ranking factors, and the higher-ranked response (e.g., the second response) can be rendered to the user for further interaction (user selection, sending via invitation / email, creating a calendar entry, etc.). Alternatively, in some cases, the first response and the second response can be rendered simultaneously as separate interactive elements for the user to interact with. Alternatively, in some cases, the first response and the second response can be used to generate a final response rather than being directly rendered, where the final response is rendered to the user visually or audibly.

[0020] Alternatively, queries determined to be related to an electronic schedule can be processed by an LLM that has been launched using a first launch input and a second launch input in succession to generate a single LLM output. In such a case, the LLM may not need to be reset after being launched using the first launch input. That is, the LLM can be launched using the second launch input immediately after being launched using the first launch input. After being launched using both the first launch input and the second launch input, the LLM can process the query to generate a single LLM output. Optionally, different launch weights that affect the single LLM output can be assigned to the first launch input and the second launch input.

[0021] In some implementations, based on the LLM output, a response to the query can be generated for user interaction, where the response can be rendered visually and / or auditorily. Optionally, the response to the query can be modified or altered based on unstructured data such as a message or an electronic note.

[0022] The above is provided only as an overview of some implementations. Those implementations and / or other implementations are disclosed in more detail herein.

[0023] Each implementation can include a non-transitory computer-readable storage medium storing instructions that can be executed by a processor to perform methods, such as one or more of the methods described herein. Each of the other implementations can include a system that includes a memory and one or more hardware processors operable to execute instructions stored in the memory to perform methods, such as one or more of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the drawings. In the drawings:

[0025] Figure 1A and Figure 1B Accordingly, block diagrams are depicted that illustrate various aspects of the present disclosure and example environments in which the implementations disclosed herein can be implemented.

[0026] Figure 2A An example user interface is depicted in accordance with various implementations.

[0027] Figure 2B Depicted is a transition from Figure 2A to another example user interface in accordance with various implementations.

[0028] Figure 3A Another example user interface is depicted in accordance with various implementations.

[0029] Figure 3B depicts another example user interface transitioning from a Figure 3A user interface, according to various implementations.

[0030] Figure 3C depicts another example user interface transitioning from a Figure 3A or Figure 3B user interface, according to various implementations.

[0031] Figure 4 shows a flowchart depicting an example method for generating an automated response in response to a message (or query), according to various implementations.

[0032] Figure 5 is a flowchart showing an additional example method for generating an automated response in response to a message (or query), according to various implementations.

[0033] Figure 6 shows an example architecture of a computing device, according to various implementations.

[0034] Figure 7 depicts an example of a user interaction involving a query related to calendar data, according to various implementations;

[0035] Figure 8A , Figure 8B and Figure 8C together depict another example of a user interaction involving a query related to calendar data, according to various implementations. Detailed Description

[0036] The following description with reference to the accompanying drawings is provided to understand various implementations of the present disclosure. It should be understood that different features from different embodiments may be combined with and / or exchanged with each other. Additionally, those of ordinary skill in the art will recognize that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. For clarity and conciseness, descriptions of well-known or repetitive functions and configurations may be omitted.

[0037] The terms and words used in the following description and claims are not limited to their bibliographical meanings and are used only by the inventors to achieve a clear and consistent understanding of the present disclosure. Thus, it will be apparent to those skilled in the art that the following description of the various embodiments of the present disclosure is provided for illustrative purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0038] Figure 1Ais a block diagram of an example environment 100A that illustrates various aspects of the present disclosure and in which implementations disclosed herein may be implemented. As Figure 1A shown, the environment 100A may include a client computing device 11 and a server 12 that communicates with the client computing device 11 via one or more networks 15. The client computing device 11 may be, for example, a cellular phone, a laptop computer, a desktop computer, a notebook computer, a tablet computer, a smart TV, a messaging device, or a personal digital assistant (PDA), and the present disclosure is not limited thereto. The server computing device 13 may be, for example, a web server, a proxy server, a VPN server, or any other type of server as needed. The one or more networks 15 may include, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, and / or any other suitable network.

[0039] In some implementations, the client computing device 11 may be installed with or otherwise access a messaging application 111, where the messaging application 111 may be a test messaging application, an email application, an instant messaging application, a social media application, etc. The messaging application 111 may be an application locally installed at the client computing device 11, or may be a web-based application accessible via the web browser of the client computing device 11.

[0040] The messaging application 111 can receive one or more messages via electronic communication 16, where the one or more messages can include queries such as "Are you free next Thursday?", "Do you have time tomorrow?", "When is Sarah’s birthday?", etc. A user of the client computing device 11 can use the messaging application 111 to receive or send one or more messages containing unstructured schedule data, where the unstructured schedule data can be time data indicating a schedule of an individual that has not been entered as an entry in an electronic schedule. For example, a user can use the messaging application 111 to send a message containing unstructured schedule data for which no corresponding entry has been created in any electronic schedule, i.e., "Hi Doctor Han, this is to confirm my visit to Eastside animal hospital Friday noon". As another example, the messaging application 111 can receive an auto-reply message containing unstructured schedule data for which no corresponding entry has been created in the electronic schedule accessible via the client computing device 11, i.e., "I am out of office between June 25th and July 5th".

[0041] In some implementations, the client computing device 11 may be installed with or otherwise access a calendar application 115, where the calendar application may be integrated with the messaging application 111 or may be a separate application separate from the messaging application 111. A user of the client computing device 11 may use the calendar application 115 to access one or more electronic calendars. For example, the user may use the calendar application 115 to access the user's first electronic calendar to obtain structured calendar data, where the structured calendar data may include one or more entries created in the first electronic calendar for one or more corresponding events that the user is interested in or plans to attend. One or more entries in the first electronic calendar may include a first entry automatically created by the calendar application 115 in the first electronic calendar for a federal holiday or other recognized holiday, and / or a second entry created by the user (or created by another user but entered by this user) in the first electronic calendar for an event (such as a training course, meeting, party, birthday, wedding, class, reunion, concert, deadline, etc.). The event may last for several minutes, several hours within a day, several hours over several days, or may span several days, and the present disclosure is not intended to be limiting.

[0042] In some implementations, the first entry and the second entry in the structured calendar data associated with the first electronic calendar can each include multiple fields (e.g., a fixed number of fields). As a non-limiting example, the first entry can have a time field with a value of "July 4th, 2018", an event field with a value of "IndependenceDay", an attendee field that is left blank (a value indicating N / A), a location field that is left blank, and / or a description field with a value such as "riverside fireworks at 9pm", etc. Similarly, for example, the second entry may have a time field with a value of "February 22, 2015, 9pm-10pm," an event field with a value of "Jazz concert," an attendee field with a value of "Helen & me," a location field with a value of "Concert Hall," and / or a description field with a value such as "parking at parking lot A across the Concert Hall, need to arrive 15 min earlier." Optionally, the first entry and the second entry, while displayed as being remote from each other (e.g., separately located on a calendar month or calendar year), may be structured using a standard format (e.g., a table) or data model.

[0043] Optionally or additionally, a user of the client computing device 11 may use the calendar application 115 to access a second electronic calendar shared by an attached user to obtain structured calendar data. Similar to the structured calendar data from the first electronic calendar, the structured calendar data from the second electronic calendar may include one or more entries created for one or more events in the second electronic calendar, where the one or more events respectively correspond to the one or more entries. For example, the second electronic calendar may include a third entry for an event (such as a team meeting) that the attached user has confirmed to attend, and / or a fourth entry created for the birthday of the attached user's best friend. In some cases, the user may be a colleague of the attached user or a family member of the attached user, such that the attached user grants the user temporary (or permanent, or partial) permission to access the second electronic calendar. In some other cases, the second electronic calendar may be a publicly accessible public calendar or a private calendar shared among a group of people, where members of the group do not need to request access permission whenever they want to access the second electronic calendar after being added as members of the group.

[0044] In some implementations, the client computing device 11 may further include one or more data storage devices 117, where the one or more data storage devices 117 may store structured calendar data 1171, unstructured data such as preference data 1173, and / or other data 1175 (such as electronic notes) described in the present disclosure.

[0045] In some implementations, the client computing device 11 may include or otherwise access a content generation system 113 that communicates with one or more machine learning (ML) models 14. The one or more ML models 14 may include, for example, a large language model (LLM) 142, where the LLM 142 may be T5, GPT-3, or other language models. The content generation system 113 may include, for example, a query recognition engine 1131, where the query recognition engine 1131 may process natural language content parsed from a text message (or translated from a voice message) to determine whether the text message (or voice message) includes a query related to any of the previously mentioned electronic calendars (such as the first electronic calendar). Alternatively or additionally, the query recognition engine 1131 may directly process a voice message (or spoken utterance) to determine whether the voice message (or spoken utterance) includes a query related to any of the previously mentioned electronic calendars (such as the first electronic calendar or the second electronic calendar).

[0046] As a non - limiting example, a user of the client computing device 11 can receive an email containing content (i.e., "Are you free next Thursday") via the messaging application 111, and the query recognition engine 1131 can process the email to determine that this email includes a query related to the first electronic schedule based on the content (i.e., "Are you free next Thursday") that includes a time item (i.e., "Thursday") and a noun (i.e., "you") referring to the user of the client computing device 11 and associated with the first electronic schedule. Optionally or in addition, the query recognition engine 1131 can identify the query as a candidate for use in a subsequent invocation of the LLM 142 based on an entry created for the user next Thursday in the first electronic schedule.

[0047] As another non - limiting example, a user of the client computing device 11 can receive an oral utterance (e.g., "Hey Assistant, do I have time to go to the dentist tomorrow?") via an automated assistant (see Figure 1B ), where the oral utterance is processed as natural - language content (e.g., "do I have time to go to the dentist tomorrow" in natural - language form). The query recognition engine 1131 can process the natural - language content (e.g., "do I have time to go to the dentist tomorrow") to determine that the oral utterance includes a query related to the first electronic schedule.

[0048] Optionally, the query recognition engine 1131 can determine that the oral utterance (e.g., "Hey Assistant, do I have time to go to the dentist tomorrow?") includes a query related to the second electronic schedule based on natural - language content (e.g., "do I have time to go to the dentist tomorrow") processed from the oral utterance that includes: (1) one or more time items (i.e., "tomorrow"); and (2) a noun (i.e., "I") associated with the first electronic schedule (e.g., the private schedule of the user who provided the oral utterance).

[0049] As a further non - limiting example, a user of the client computing device 11 may receive an oral utterance (e.g., "Hey Assistant, when is Sarah’s birthday?") via an automated assistant (see Figure 1B ), where the oral utterance is processed into natural language content (e.g., "when is Sarah’s birthday" in natural language form). The query recognition engine 1131 may process the natural language content (e.g., "when is Sarah’s birthday") to determine that the oral utterance includes a query related to a second electronic schedule (e.g., a semi - private or public schedule for birthdays, parties, and outdoor activities shared among a group of friends). In this example, the oral utterance (e.g., "Hey Assistant, when is Sarah’s birthday?") may be determined to include a query related to the second electronic schedule based on natural language content (e.g., "when is Sarah’s birthday") processed from the oral utterance that includes one or more time items (i.e., "when" and "birthday") and a noun (i.e., "Sarah") that appear in one or more entries of the second electronic schedule. Optionally or additionally, the query recognition engine 1131 may identify the query as a candidate for use in a subsequent invocation of the LLM 142 based on identifying an entry in the second electronic schedule corresponding to Sarah’s birthday.

[0050] As an additional non - limiting example, referring to Figure 7 , a client computing device 720 (e.g., a smart speaker) of user 710 may receive an oral utterance via an automated assistant installed at the client computing device 720 (see Figure 1B , Figure 7not shown) receives an oral utterance 710a (e.g., "Hey Assistant, I’d like to meet with the VPsometime this week please") from a user 710, where the oral utterance may be processed as natural language content (e.g., in natural language form, "I want to meet with the VP sometime this week"). A query recognition engine 1131 may process the natural language content (e.g., "I want to meet with the VP sometime this week") to determine that the oral utterance includes a query related to the electronic calendar of the VP (Vice President). In some implementations, the query recognition engine 1131 may, for example, use a first ML model among one or more ML models 14 to determine that the oral utterance (e.g., "Hey Assistant, I’d like to meet with the VPsometime this week please") includes a query (e.g., in natural language form, "does the VP have time this week for a meeting"), and further use a second ML model among one or more ML models 14 to determine that the query is related to the electronic calendar of the VP (Vice President). The second ML model may determine that the query is related to the electronic calendar of the VP (Vice President) based on one or more terms (e.g., "time this week" and "VP") in the query (e.g., "does the VP have time this week for a meeting").

[0051] In some implementations, the content generation system 113 may further include, for example, an LLM engine 1133 that communicates with the LLM 142. The LLM engine 1133 may use one or more startup inputs to start the LLM 142. The one or more startup inputs may include, for example, a first startup input generated based on one or more entries of structured schedule data from the first electronic schedule (or the second electronic schedule or other electronic schedules). Continuing with the above example, where the query recognition engine 1131 processes natural language content (e.g., "I want to meet with the VP sometime this week") from an oral utterance (e.g., "Hey Assistant, I’d like to meet with the VPsometime this week please") to determine that this natural language content includes a query related to the VP's electronic schedule (e.g., "does the VP have time this week for a meeting"), the LLM engine 1133 may use entries for other times this week in the VP's electronic schedule to generate the first startup input.

[0052] For example, the portion of the VP's electronic schedule with entries for other times this week may include a first entry (e.g., "recruiting interview, October 9th, 3pm-4pm"), a second entry (e.g., "out of office, from October 10 th(all day), to October 11th, 1pm (not in the office, from October 10th (all day), to October 11th, 1pm)”, and the third entry (e.g., “group meeting, October 11th, 1:30pm to 3pm (group meeting, October 11th, 1:30pm to 3pm)”). The LLM engine 1133 can process the first, second, and third entries of the electronic schedule of the VP (Vice President) to generate a first start input, where the first start input can be in the form of natural language content including a description such as “VP is available October 9th before 3pm or after 4pm, and available on October 11th between 1pm and 1:30pm or after 3pm (The vice president is available before 3 pm or after 4 pm on October 9th, and available between 1 pm and 1:30 pm or after 3 pm on October 11th)”. The LLM engine 1133 can use the first start input (e.g., in natural language form “VP is available October 9th before 3pm or after 4pm, and available on October 11th between 1pm and 1:30pm or after 3pm (The vice president is available before 3 pm or after 4 pm on October 9th, and available between 1 pm and 1:30 pm or after 3 pm on October 11th)”), or in natural language form “VP is available the rest of week on October 9th before 3pm or after 4pm, or on October 11 thbetween 1pm and 1:30pm or after 3pm” in natural language (The vice president is available at other times this week before 3pm on October 9th or after 4pm, or between 1pm and 1:30pm or after 3pm on October 11th)”) to start the LLM model 142. After starting with the first start input, the LLM engine 1133 can use the started LLM model 142 to process a query (e.g., “does the VP have time this week for a meeting (Does the vice president have time for a meeting this week)”) to generate an LLM output. In this case, the LLM output can be “on October 9th, the VP has time for a meeting before 3pm or after 4pm, and on October 11th, the VP has time for a meeting between 1pm and 1:30pm, or after 3pm (On October 9th, the vice president has time for a meeting before 3pm or after 4pm, and on October 11th, the vice president has time for a meeting between 1pm and 1:30pm, or after 3pm)”.

[0053] Optionally or alternatively, in the above example, in addition to the part of the VP (vice president)'s electronic schedule for other times this week, the LLM engine 1133 can also process the part of the user 710's electronic schedule for other times this week to generate the first start input. The part of the user 710's electronic schedule for other times this week can for example include an entry for October 9 (e.g., out of office, October 9 th , all day (Out of office all day on October 9)) and an additional entry for October 11 (e.g., meeting with manager A, October 11 th , 3-5pm (Meeting with manager A from 3 to 5pm on October 11)). Correspondingly, the LLM engine 1133 can generate a first start input such as “both you and VP are available on October 11 th, between 1pm and 1:30pmthis week (You and the vice president are both available on October 11th between 1pm and 1:30pm). Optionally or additionally, in addition to the portion of the user 710's electronic calendar for the rest of the week and the portion of the user 710's electronic calendar for the rest of the week, the LLM engine 1133 may also process user characterization data (e.g., user data indicating the position, title, or category label of the user 710 (i.e., the director of engineering), and / or relationship data indicating the relationship between the user 710 and the VP (vice president), e.g., the user 710 as the director of engineering directly reports to the VP (vice president)). In this case, the LLM engine 1133 may generate a first launch input, such as “director of engineering and VP are both available on October 11th, between 1pm and 1:30pmthis week”. th , between 1pm and 1:30pm this week (the head of engineering and the vice president are both available between 1pm and 1:30pm on October 11th)".

[0054] As another non-limiting example, the first launch input may include a natural language representation generated based on a schedule entry of the first electronic calendar (e.g., "Helen & me | Jazz concert | Concert Hall | February 22, 2015, 9pm to 10pm | parking at parking lot A across the Concert Hall, need to arrive 15 min earlier").

[0055] Alternatively or additionally, the first initiation input may include a calendar entry in addition to the first natural language representation. Referring to the non-limiting examples mentioned above, the first launch input may include: (1) a calendar entry (e.g., “Helen & me | Jazz concert | Concert Hall | February 22, 2015, 9pm to 10pm | parking at parking lot A across the Concert Hall, need to arrive 15 min earlier”), and / or (2) a first natural language representation (e.g., Helen and I will go to Concert Hall attending the Jazz concert on February 22, 2015 from 9pm to10pm, need to arrive 15 min earlier to park at parking lot A across the Concert Hall).

[0056] Optionally or additionally, in addition to the first natural language representation, the first activation input may further include unstructured message data (e.g., a message containing "Hey, could I join you and Helen for the Jazz concert? ——Lily"). In this case, the first activation input may include: (1) natural language content generated based on the message (e.g., "Lily wants to join Helen & I for the Jazz concert"); and / or (2) the first natural language representation (e.g., Helen and I will go to Concert Hall attending the Jazz concert on February 22, 2015 from 9pm to 10pm, need to arrive 15 min earlier to park at parking lot A across the Concert Hall).

[0057] As another non-limiting example, one or more activation inputs can include a second activation input, where the second activation input can include a second natural language representation (e.g., "On September 12, 2013 next Thursday, one-on-one meeting with Bill at Meeting Room between 10am and 10:30am and take Frank to First Animal Hospital for vet visit - rabies vaccine between 4:30pm and 5pm"), and the second natural language representation is generated based on a first schedule entry and a second schedule entry of a first electronic schedule (e.g., "Bill and I | one-on-one meeting | Meeting Room | September 12, 2013, 10am to 10:30am | N / A" and "me | vet visit | First Animal Hospital | September 12, 2013, 4:30pm to 5pm | rabies vaccine for Frank").

[0058] After being launched using, for example, a second launch input, the LLM 142 can be used to process a query input generated based on a query (e.g., an email containing "Are you free next Thursday" received from David via the messaging application 111 at 3:23 PM on September 6, 2013), to generate an LLM output. In this case, the LLM output can be, for example, "not available 10am~10:30 am and 4:30pm~5pm next Thursday".

[0059] In some implementations, the content generation system 113 may further include a response generation engine 1135. The response generation engine 1135 may, for example, process or modify the aforementioned LLM output (e.g., "not available 10am~10:30am and 4:30pm~5pm next Thursday") to generate a response to the aforementioned query (e.g., "I am free next Thursday except for 10am~10:30 am and 4:30pm~5pm", or "on Thursday, I am free between 10:30am and 4:30pm"). The response may be rendered in a natural language form (or auditorily) to answer the query in an email containing "Are you free next Thursday", where the email is received via the messaging application 111 from a requesting user (e.g., a user named David). The response may be rendered as a selectable element to the user of the client computing device 11, where the user may select the selectable element to enter a response (e.g., "I am free next Thursday except for 10am~10:30 am and 4:30pm~5pm") as a reply to David's email containing "Are you free next Thursday".

[0060] Turning to the example mentioned above, where the LLM engine 1133 has initiated the LLM model 142 with a first startup input (e.g., "VP is available October 9th before 3pm or after 4pm, and available on October 11th between 1pm and 1:30pm or after 3pm") to generate an LLM output (e.g., "on October 9th, the VP has time for a meeting before 3pm or after 4pm, and on October 11th, the VP has time for a meeting 1pm~1:30pm, or after 3pm"), the response generation engine 1135 can be configured to process the LLM output to generate a response to a query (e.g., "does the VP have time this week for a meeting"). For example, the response generation engine 1135 can directly output the LLM output as the response.

[0061] Alternatively or additionally, the response generation engine 1135 can search for and / or identify unstructured data (which is associated with the LLM output), such as an email message from the VP (Vice President) to everyone that says "starting from October 1, the company will offer free lunch between 12pm to 1pm every workday for full-time employees". The response generation engine 1135 can process the LLM output and the identified unstructured data to generate a response that modifies the LLM output to "on October 9th, the VP has time for a meeting before 12pm, 1pm - 3pm, or after 4pm, and on October 11th, the VP has time for a meeting between 1pm and 1:30pm, or after 3pm".

[0062] Alternatively or in addition, the response generation engine 1135 can retrieve unstructured calendar data (e.g., user preference data, also referred to as "preference data") and process the LLM output and the unstructured calendar data (e.g., user preference data) to generate a response. The response generation engine 1135 can, for example, retrieve user preference data of the user from which the verbal utterance of the query was identified and / or user preference data of a vice president (VP). As a non-limiting example, the response generation engine 1135 can retrieve the user preference data of the VP and can determine that the user preference data of the VP indicates that the VP has a preference for meetings between 8 a.m. and 10 a.m. In this example, the response generation engine 1135 can process the LLM output (e.g., "on October 9th, the VP has time for a meeting before 3pm or after 4pm, and on October 11th, the VP has time for a meeting October 11th between 1pm and 1:30pm or after 3pm") and the VP's user preference data indicating that the VP has a preference for meetings between 8am and 10am to generate a response (e.g., Figure 7 Response 720a), which is "invite the VP for a meeting on October 9th, 8am to 9am or 9am to 10am".

[0063] The response 720a can be aurally rendered to the user 710, and the user 710 can further interact with the client computing device 720 by providing additional spoken utterances 710b received by an automated assistant installed at (or otherwise accessible to) the client computing device 720 (e.g., "I’d like to meet the VP 9am to 10am"). In response to receiving the additional spoken utterance 710b, the automated assistant can render an additional response 720b via the microphone of the client computing device 720b (e.g., "Okay, would you like to send an invite for the meeting over to the VP?"). And in response to further spoken utterance 710c from the user 710 (e.g., "yes, please"), the automated assistant can generate an invitation and send the invitation to the VP (either via an audio message describing the invitation or via a text message / email containing a link to the invitation), and then output a further response 720c (e.g., "Invite sent. Waiting for response.").

[0064] Optionally or additionally, the automated assistant can output a follow-up response in response to the VP accepting the invitation (e.g., by selecting a link), where the follow-up response is a reminder sound that alerts the user 710 that there is a notification to read (or listen to), or the follow-up response can be an audio message 720d that is automatically rendered in response to the VP accepting the invitation (e.g., "VP accepts the invite, would you like to create the meeting event in calendar?"). If the user 710 confirms with an input 710e (e.g., "Yes, in my calendar"), the automated assistant can create an entry in the user 710's work calendar for "meeting with VP on October 9th, 9am to 10am". If the user 710 confirms with a different input (e.g., "yes, in my calendar and the VP’s calendar"), the automated assistant can create an entry in the user 710's work calendar for "meeting with VP on October 9th, 9am to 10am", and create an entry in the VP's work calendar for "meeting with VP on October 9th, 9am to 10am" (if permitted, and if not permitted, permission to create this entry can first be sought from the VP). The conversation between the user 710 and the automated assistant via the client computing device 720 can continue with additional communication regarding user permissions, user confirmations, modification of entries in the calendar, etc., and the present disclosure is not intended to be limiting. Optionally or alternatively, in response to the VP accepting the invitation, the automated assistant can create an entry in the VP's work calendar for "meeting with employee David on October 9th, 9am to 10am".

[0065] Additionally or alternatively, rather than auditorily rendering the response 720a to the user 710, one or more selectable elements corresponding to the generated response (e.g., having the same or similar content as the response 720a in natural language form) may be rendered to the user 710 via a display of the client computing device 720. For example, the one or more selectable elements may be a single selectable element corresponding to "meeting with the VP on October 9th, 9am to 10am". Alternatively, the one or more selectable elements may include a first selectable element corresponding to "8am to 9am" and a second selectable element corresponding to "9am to 10am". When a selectable element among the one or more selectable elements is selected, an invitation for a meeting from 8am to 9am on October 9th (and / or from 9am to 10am, depending on the design of the one or more selectable elements and the user selection of the one or more selectable elements) may be sent to the VP along with a description including, for example, "Sam asks: I’d like to meet with the VP sometime this week please". Optionally, the VP may accept the meeting invitation by selecting "October 9th, 8am to 9am" as the proposed time for the meeting, and optionally, in response to the VP accepting the invitation, a calendar entry for "Meeting (Sam and VP), October 9th, 8am to 9am" may be created in the VP's electronic calendar and in the electronic calendar of Sam (the user who provided the oral statement (i.e., "I’d like to meet with the VP sometime this week please")).

[0066] In some implementations, the content generation system 113 may further include a schedule processing engine 1137 that processes structured schedule data of an electronic schedule (e.g., entries in the electronic schedule) to generate a natural language presentation of the structured schedule data. As a non-limiting example, the schedule processing engine 1137 may process an entry of a first electronic schedule (e.g., "Helen & me | Jazz concert | Concert Hall | February 22, 2015, 9pm to 10pm | parking at parking lot A across the Concert Hall, need to arrive 15 min earlier") to generate a corresponding natural language presentation (e.g., "Helen and I will go to Concert Hall attending the Jazz concert on February 22, 2015 from 9pm to 10pm, need to arrive 15 min earlier to park at parking lot A across the Concert Hall").

[0067] In some implementations, the server 12 may include a query recognition engine 121, an LLM engine 123, a response generation engine 125, and / or a schedule processing engine 127. The query recognition engine 121 may be the same (or similar) to the query recognition engine 1131 that can be locally accessed at the client computing device 11. For example, the query recognition engine 121 may determine whether a message or an oral utterance includes a query related to an electronic schedule. In the case where it can be accessed via the server 12, the query recognition engine 121 may perform this determination in a more efficient manner than the query recognition engine 1131. Thus, to prevent this determination from consuming too much (or unnecessary) computing resources of the client computing device 11, the client computing device 11 may offload this determination process to the query recognition engine 121.

[0068] Similarly, LLM engine 123 can be the same as or similar to LLM engine 1133, response generation engine 125 can be the same as or similar to response generation engine 1135, and schedule processing engine 127 can be the same as or similar to schedule processing engine 1137. In other words, LLM engine 123 can be the cloud equivalent of LLM engine 1133 at client computing device 11 (e.g., providing services in a cloud computing environment), response generation engine 125 can be the cloud equivalent of response generation engine 1135, and schedule processing engine 127 can be the cloud equivalent of schedule processing engine 1137. In this case, environment 100A can be a cloud computing environment in which multiple computing devices, which can be on the order of hundreds or thousands or more, share resources via one or more networks 15. Repeated descriptions can be found in the description of this specification, and thus these repeated descriptions are omitted here.

[0069] Figure 1B A block diagram depicting another example environment 100B that demonstrates various aspects of the present disclosure and in which the implementations disclosed herein can be implemented. As Figure 1B shown, environment 100B can include a client computing device 11 having a local automation assistant 119. Environment 100B can further include a cloud-based automation assistant 13 that communicates with client computing device 11 via one or more networks 15. Optionally, the client computing device can include a messaging application 111, a calendar application 115, and a data storage device 117. Local automation assistant 119 can access one or more ML models 14 and can include the aforementioned LLM engine 1131 that communicates with LLM 142 in one or more ML models 14. Local automation assistant 119 can receive user input 17 via an input device of client computing device 11. For example, user input 17 can be an oral utterance and is received by local automation assistant 119 via a microphone of client computing device 11. User input 17 can also be a touch input, a camera input, or a keyboard input received by local automation assistant 119 via a touch screen, a camera, or a keyboard of client computing device 11, and the present disclosure is not limited thereto.

[0070] The cloud-based automated assistant 13 may include an automatic speech recognition (ASR) engine 131, a natural language understanding (NLU) engine 133, a text-to-speech (TTS) engine 135, and a content generation system 137. The ASR engine 131 may process audio data capturing an oral utterance to generate a recognition of the oral utterance. The NLU engine 133 may determine the semantic meaning of the audio and / or the text converted from the audio by the ASR engine, and decompose the determined semantic meaning to determine the intent and / or parameters of an assistant action. For example, the NLU engine 133 may determine the intent and / or parameters of an assistant action based on the foregoing recognition of the oral utterance generated by the ASR engine 131.

[0071] In some implementations, the NLU engine 133 may parse intent and / or parameters based on a single utterance of a user, and in other cases, may generate a prompt based on unparsed intent and / or parameters, those prompts rendered to the user, and user responses to those prompts utilized by the NLU engine 133 in parsing the intent and / or parameters. In those cases, the NLU engine 133 may optionally work in concert with a dialogue manager engine (not shown) that determines unparsed intent and / or parameters and / or generates corresponding prompts. The NLU engine 133 may utilize one or more NLU machine learning models in one or more of the ML models 14 to determine intent and / or parameters.

[0072] The TTS engine 135 may convert text to synthesized speech and may rely on one or more speech synthesis neural network models to do so. For example, the TTS engine 135 may be utilized to convert a text response to audio data that includes a synthesized version of the text, and the synthesized version may be auditorily rendered via a hardware speaker of the client computing device 11 or another device. The content generation system 137 may be the same as or similar to the foregoing content generation system 113, and a repeated description is not provided herein.

[0073] In some implementations, an automated assistant (i.e., local automated assistant 119) can, for example, use NLU 133 (which can be accessed locally or remotely) to determine that a text message received via an instant messaging application (e.g., "Liam, do you have time tomorrow? just curious") includes a query (e.g., "Do you have time tomorrow" or "Liam, do you have time tomorrow"), where the text message can be dated February 5, 2017, 2:45 PM. The automated assistant (i.e., local automated assistant 119) can process the query locally (if the automated assistant includes or otherwise has access to content generation system 113), or forward the query to cloud-based automated assistant 13 for remote processing. The query can be processed locally or remotely to determine that the query is related to an electronic calendar, for example, by the automated assistant identifying an entry in the electronic calendar (e.g., "vacation to Puerto Rico from February 3rd to February 16th") in response to the query. In such a case, the automated assistant can use the entry in the electronic calendar (e.g., "vacation to Puerto Rico from February 3rd to February 16th") to initiate LLM model 142, and then use LLM model 142 initiated with the entry (e.g., "vacation to Puerto Rico from February 3rd to February 16th") to process the query (e.g., "Do you have time tomorrow"), where the initiated LLM model 142 processes the query (e.g., "Do you have time tomorrow") to generate an LLM output, such as "I don't have time tomorrow".

[0074] Optionally, the automated assistant can use the content generation system 113 and further determine a response to the query based on the LLM output. For example, the automated assistant can determine based on the LLM output (e.g., “I don't have time tomorrow”) and based on the entry (e.g., “vacation to Puerto Rico from February 3rd to February 16th”) that the response includes a natural language response, such as “I don't have time tomorrow. I am away from February 3rd to February 16th” or “I don't have time tomorrow, my whole calendar tomorrow is full”. The automated assistant can cause the natural language response (e.g., “I don't have time tomorrow, my whole calendar tomorrow is full.”) to be rendered as a selectable element via the interface of the instant messaging application. When selected, a natural language response (e.g., "I don't have time tomorrow, my whole calendar tomorrow is full.") can be entered into a text input field at the interface of the instant messaging application as a reply (or part of a reply) to the aforementioned text message (e.g., "Liam, do you have time tomorrow? just curious."). The natural language response can be sent directly, or can be edited before being sent. In this way, the user does not need to check his or her calendar to formulate a response to a query related to his or her calendar.

[0075] Figure 2A Depicted is an example user interface 200A for providing one or more selectable elements for selection as a response to a query from an email, according to various implementations. Figure 2B Depicted is another example user interface 200B transitioning from user interface 200A and showing a user selection of a response in accordance with various implementations.

[0076] Figure 4 An example method 400 for generating an automatic reply in response to a message according to various implementations is shown. For convenience, the operations of method 400 are described with reference to a system that performs the operations. The system of method 400 includes one or more processors and / or other components of a client device and / or a server device. In addition, although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0077] At block 401, the system receives a query. As a non-limiting example, the query may be a verbal utterance such as "Hey Assistant, I'd like to meet with the VP sometime this week please" (see Figure 7 ) from audio data (e.g., “When VP is available this week?”). As another non-limiting example, the query may be a query identified from a message (e.g., an email received by Kate asking “are you free next Thursday for project discussion?”, see Figure 2A ) (e.g., "Is Kate free next Thursday? " or "What is Kate's schedule next Thursday?"). As a further non-limiting example, the query can be a text query detected from text input via a touch screen, keyboard, or other input unit.

[0078] Optionally, prior to receiving the query, the system may receive a message or audio data capturing the spoken utterance and determine whether the message or spoken utterance includes the query. Figure 2A, when receiving an email message 210 (e.g., subject 211: “Are you free next Thursday”; sender / receiver 213: Tom / Kate and Jerry; email body 215: “Kate and Jerry, Are you both available next Thursday for some discussion about project X?”), the email message 210 may be processed. In this example, based on detecting a question mark and / or one or more terms representing a query (e.g., “are you”) from the email message 210 (e.g., the email body 215 and / or the email subject 211), it may be determined that the email message 210 includes a query. Here, the query may be determined as “Kate and Jerry, Are you both available next Thursday for some discussion about project X?” or “Are Kate and Jerry available next Thursday for project discussion?”, and the present disclosure is not limited thereto.

[0079] At block 403, the system determines whether the query is directed to an electronic calendar. As a non-limiting example, the system can determine whether the query is directed to an electronic calendar based on whether the query includes a first keyword representing a human entity (e.g., "my daughter") and / or whether the query includes a second time-based keyword (e.g., "Friday"). The first keyword representing a human entity can include, for example, a human name (e.g., "Mary"), a title in an organization / family (e.g., "professor for the economy class" or "niece"), a noun (e.g., "she"), and / or other human entity keywords. The second time-based keyword can be, for example, a time-related item (e.g., "this weekend"), or one or more time-related words (e.g., "when" and / or "birthday"), and / or other time-related terms. For example, when the query is identified from a message as "Is Kate free next Thursday for discussion of project X", the query can be determined to be directed to Kate's electronic calendar based on including the keyword "Kate" (human entity) and "next Thursday" (time term). Optionally, determining whether the query includes the first keyword and determining whether the query includes the second keyword can be performed simultaneously. Optionally, determining whether the query includes the first keyword can be performed before determining whether the query includes the second keyword. Optionally, determining whether the query includes the second keyword can be performed before determining whether the query includes the first keyword.

[0080] In some implementations, at block 403, the system can determine that the query is related to more than one calendar. Refer to Figure 2AAs a non-limiting example, a client device may receive an email message having a title 211 (i.e., “Are you free next Thursday?”), a message body 215 (i.e., “Kate and Jerry, Are you both available next Thursday for some discussion about project X?”), and sender / receiver information 213 (i.e., the message is sent by Tom to Kate and Jerry). In this example, a query may be identified from the email message as: “what are Kate's and Jerry's schedules next Thursday”. Based on having human entity keywords (i.e., “Kate” and “Jerry”) and time terms (i.e., “next Thursday”), this query may be determined to be relevant to two schedules (i.e., Kate's schedule and Jerry's schedule). In this case, in addition to the human entity keywords and time terms, metadata may also be applied to determine the schedule to which the query is directed. For example, based on metadata (eg, reply message 230 is being edited by Kate to Tom and Jerry), the query may be determined to be directed to Kate's calendar.

[0081] In box 405, the system starts the LLM model using a start input generated based on schedule data associated with the electronic schedule. In some implementations, when starting the LLM model (405), the system can perform one or more of sub-boxes 405-1, 405-3 and / or sub-box 405-5. In sub-box 405-1, the system processes the structured schedule data of the electronic schedule to generate a natural language representation of the structured schedule data of the electronic schedule. For example, the electronic schedule can be a private schedule of a first user that can be accessed by the first user via a schedule application (or an application with a schedule module). In this example, the first user can create one or more private schedule entries in the private schedule, or the first user can receive a schedule invitation and accept the schedule invitation to create one or more shared schedule entries in the private schedule (shared only between the first user and the inviter who sent the schedule invitation). As another example, the electronic calendar may be a public calendar shared among a group of users (eg, a group of coworkers or a group of friends), wherein each user within the group may create one or more public calendar entries in the public calendar that are shared within the group.

[0082] In some implementations, the structured schedule data may include one or more entries of an electronic schedule, and when processing the structured schedule data of the electronic schedule to generate a natural language representation of the structured schedule data of the electronic schedule (sub-box 405-1), the system may: process one or more entries of the electronic schedule data to generate a natural language representation of the one or more entries. As a non-limiting example, the electronic schedule of a first user (e.g., User A) may include a schedule entry (Title: Sue’s birthday; Time: July 8, 2012, all day; Visibility: shared), and this schedule entry may be processed to generate a natural language representation of the schedule entry, i.e., “Sue’s birthday is July 8th” or “Sue’s birthday is July 8th, next Thursday”.

[0083] Optionally, when processing one or more entries of electronic calendar data to generate a natural language representation of the one or more entries, the system can: process each of the one or more entries separately to generate a corresponding natural language representation of the corresponding entry in the one or more entries. As a non-limiting example, the electronic calendar of a first user (e.g., User A) can include a first entry (Title: Sue’s birthday; Time: July 12, 2012, all day; Visibility: shared) and a second entry (Title: pick up Bob at the airport; Time: July 15, 2012, 3:00pm to 6:00pm EST; Visibility: private). In this example, processing one or more entries of the electronic calendar can include: processing the first entry to generate a first natural language representation of the first entry (e.g., Sue’s birthday is July 12th), and processing the second entry to generate a second natural language representation (e.g., user A will pick up Bob at the airport between 3 to 6pm on July 15, 2012).

[0084] Optionally, when processing one or more entries of an electronic calendar to generate a natural language representation of the one or more entries, the system can: process multiple entries during the same time period to generate a single corresponding natural language representation of the multiple entries. As a non-limiting example, the electronic calendar of a first user (e.g., User B) can include a first entry (Title: Meeting; Time: July 6, 2012, 2:00pm to 3:00pm EST; Location: Conference room; Visibility: private; Description: N / A) and a second entry (Title: dental appointment; Time: July 6, 2012, 8:00am to 10:00am EST; Location: 365 Old Town Street; Visibility: private; Description: dental cleaning) that have the same format. In this example, the first and second entries can be processed to generate a natural language representation such as, "User A is not available July 6, 2012, 8:00am to 10:00am EST and 2:00pm to 3:00pm EST" (or, "On July 6th, 2012, User B is away for dental appointment 8:00am to 10:00am EST and away for a meeting 2:00pm to 3:00pm EST", etc.).

[0085] In sub - box 405 - 3, the system can generate a launch input based on the generated natural - language representation of structured schedule data. As a non - limiting example, the launch input can include a natural - language rendering generated in sub - box 405 - 1 using one or more entries of an electronic schedule (e.g., "Sue’s birthday is July 8th, next Thursday"). As another non - limiting example, the launch input can include: (1) the natural - language representation of the structured schedule data of the electronic schedule generated in sub - box 405 - 1 (e.g., "Sue’s birthday is July 8th, next Thursday"); and (2) the structured schedule data of the electronic schedule (e.g., schedule entry: "Title: Sue’s birthday; Time: July 8, 2012, all day; Visibility: shared"). As an additional non - limiting example, the launch input can include: (1) the natural - language representation of the structured schedule data of the electronic schedule (e.g., "Sue’s birthday is July 8th, next Thursday"); and (2) unstructured schedule data from a source other than the electronic schedule (e.g., an earlier electronic reminder that says "I need to get up early next Thursday"). As a further non - limiting example, the launch input can include: (1) the natural - language representation of the structured schedule data of the electronic schedule (e.g., "Sue’s birthday is July 8th, next Thursday"); (2) the structured schedule data of the electronic schedule (e.g., schedule entry: "Title: Sue’s birthday; Time: July 8, 2012, all day; Visibility: shared"); and / or (3) unstructured schedule data from a source other than the electronic schedule (e.g., an earlier text message that says "buy a birthday cake and go to Sue’s home at 5pm next Thursday").Optionally, multiple startup inputs can be used to start the LLM model.

[0086] In sub - box 405 - 5, the system uses the startup input generated in sub - box 405 - 3 to start the LLM model. Here, starting the LLM model (also referred to as the "LLM") using the startup input can be performed via multiple techniques. As a non - limiting example, after the system generates a startup input by processing the structured schedule data of an electronic calendar to generate a natural language representation of the structured schedule data of the electronic calendar, the natural language representation and / or the structured schedule data can be used as the startup input to start the LLM. As another non - limiting example, after the system retrieves unstructured data including preference data indicating one or more preferred time slots of the requesting user and / or the receiving user, the preference data can be included as part of the startup input (e.g., included in the startup input together with the natural language representation and / or the structured schedule data) to start the LLM model. As a further non - limiting example, after the system retrieves metadata (also referred to as "auxiliary data") including relationship data indicating the relationship between, for example, the user sending the query (i.e., the "requesting user") and the user receiving the query (box 4055) (i.e., the "receiving user"), the relationship data can be included in the startup input together with one or more other types of data described herein (natural language representation of structured schedule data, structured schedule data, unstructured data, etc.).

[0087] At block 407, the system uses the launched LLM model to process a query input generated at least based on the foregoing query to generate an LLM output. For example, for the query (“Kate and Jerry, Are you both available next Thursday for some discussion about project X?”), a query input such as “Kate, are you available next Thursday to discuss project X?” can be generated based on: (1) the query and (2) the fact that the client computing device receiving the message (e.g., email message 210) does not have access to Jerry's electronic calendar. As another example, the query input can also be “search Kate’s and Jerry’s electronic calendars for events planned next Thursday, July 8th, 2012”. The query input can also be in other formats, and the present disclosure is not limited thereto. In these examples, the LLM output can be, for example, “Kate’s electronic calendar shows next Thursday is Sue’s birthday”, “Kate’s electronic calendar shows next Thursday all day is Sue’s birthday”, or “Kate’s electronic calendar shows next Thursday is Sue’s birthday, and Kate mentions in a text message to buy a birthday cake and go to Sue’s home at 5pm next Thursday”.

[0088] Optionally, at block 407, the system generates multiple query inputs and uses the launched LLM model to separately process each query input to generate multiple LLM outputs. The multiple LLM outputs can include a first LLM output (e.g., "Kate's electronic calendar shows next Thursday is Sue's birthday"), and a second LLM output (e.g., "Kate's electronic calendar shows next Thursday is Sue's birthday, and Kate mentions in a text message to 'buy a birthday cake and go to Sue's home at 5pm next Thursday'").

[0089] At block 409, the system determines a response to the query based on the LLM output. For example, based on the aforementioned first LLM output (e.g., "Kate's electronic calendar shows next Thursday is Sue's birthday"), a response 233 can be generated that includes first response content 233A (e.g., "Yes, I am available next Thursday"). The first response content 233A can be generated based on the first LLM output and further based on metadata (e.g., user data) indicating that a person typically does not take time off for a friend's birthday.

[0090] Alternatively or additionally, the response 233 can include second response content 233B (e.g., Yes, I am available next Thursday before 5pm). In some implementations, the second response content 233B can be generated based on the first LLM output (e.g., “Kate’s electronic calendar shows next Thursday is Sue’s birthday”) and unstructured data such as a text message (e.g., “buy a birthday cake and go to Sue’s home at 5pm next Thursday”). In some embodiments, the second response content 233B can be generated based on the first response content 233A (e.g., “Yes, I am available next Thursday”) modified with a text message (e.g., “buy a birthday cake and go to Sue’s home at 5pm next Thursday”). In some embodiments, the second response content 233B can be generated based on a second LLM output (Kate’s electronic calendar show next Thursday is Sue’s birthday, and Kate mentions in a text message to “buy a birthday cake and go to Sue’s home at 5pm next Thursday”).

[0091] Alternatively or additionally, response 233 may include third response content 233C (e.g., "Sorry, I cannot make it next Thursday") generated based on the first LLM output (e.g., "Kate’s electronic calendar shows next Thursday all day is Sue’s birthday"). Alternatively or additionally, response 233 may include fourth response content 233D (e.g., "Sorry, I cannot make it next Thursday, but I am free all day Friday") generated based on the first LLM output (e.g., "Kate’s electronic calendar shows next Thursday all day is Sue’s birthday") and metadata indicating that the structured schedule data of the user's (i.e., Kate's) electronic calendar does not include a schedule entry for next Friday.

[0092] In other words, response 233 may include one or more response contents (e.g., Figure 2A 233A, 233B, 233C, and / or 233D in Figure 2A ), where each of the one or more response contents may be a selectable element displayed in the user interface 200A of the client computing device for user selection. Optionally, as

[0093] shown, the user interface 200A may further include a reply message 230 that includes: a recipient address field 231 showing the recipient names (such as Tom and Jerry). Optionally, response 233 may include instructions such as "Type, or choose one of the following as a reply", which disappear once user input is received to formulate the reply message 230 of the email message 210. Figure 2A, the response 233 can be rendered at the user interface 200A of the client computing device, where the response 233 includes first response content 233A displayed using a first selectable element, second response content 233B displayed using a second selectable element, third response content 233C displayed using a third selectable element, and / or fourth response content 233D displayed using a fourth selectable element. Refer to Figure 2B , a user (e.g., Kate) of the client computing device can select, for example, the second selectable element to enter the second response content 233B (e.g., "Yes, I am available next Thursday before 5pm") in the text body of the reply message 230. Then, the user can select the icon "Send" at the user interface 200B to send the reply message 230 to recipients such as Tom and Jerry.

[0094] Optionally, in some implementations, given a user's electronic calendar, before any query directed to the electronic calendar is received, the electronic calendar can be processed (e.g., at regular or irregular intervals) to generate a natural language representation of the electronic calendar. For example, Kate's electronic calendar can be processed at 10 pm every day to generate a natural language representation of the electronic calendar. In this example, the entries entered in Kate's electronic calendar before 10 pm every day can be processed to generate a natural language representation of the electronic calendar. If the next day (e.g., June 23), Kate speaks to the automated assistant and asks "Assistant, do I have any meeting tomorrow", the electronic calendar can be scanned or processed to detect if any new entries have been created since 10 pm (June 22). If no new entries have been created in Kate's electronic calendar, the natural language representation of the electronic calendar generated on June 22 can be used to initiate the LLM model when preparing to generate a response to the query. If one or more entries have been created in Kate's electronic calendar, the one or more entries can be processed to modify the natural language representation of the electronic calendar generated on June 22 such that a modified language representation of the electronic calendar is obtained and used to initiate the LLM model.

[0095] Figure 3A depicts according to various implementations Figure 1A or Figure 1B another example user interface 300A. Figure 3B depicts an example user interface 300B transitioning from the user interface 300A according to various implementations. Figure 3CDepicts an example user interface 300C transitioning from the user interface 300B according to various implementations.

[0096] Figure 5 Shows another example method 500 for generating selectable response suggestions according to various implementations. For convenience, the operations of method 500 are described with reference to the system performing the operations. The system of method 500 includes one or more processors and / or other components of a client device and / or a server device. Additionally, although the operations of method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0097] As Figure 5 shown, at block 501, the system may generate a first launch input (and / or a second launch input different from the first launch input) based at least on an electronic calendar.

[0098] In some implementations, the system may process the electronic calendar to generate the first launch input (501) by: processing the structured calendar data of the electronic calendar to generate a natural language representation of the structured calendar data of the electronic calendar; and generating the first launch input based on the generated natural language representation of the structured calendar data of the electronic calendar. As a non-limiting example, the electronic calendar may be the calendar of a first user (e.g., the user of a cellular phone having the user interface 300A, see Figure 3A ), where the calendar of the first user includes a calendar entry titled "Sarah’s birthday" on May 18, 2020. In this example, the structured calendar data of the calendar of the first user (i.e., the calendar entry for May 18, 2020) may be processed to generate a natural language representation of the structured calendar data (e.g., "Sarah’s birthday is May 18th"). Here, the natural language representation (e.g., "Sarah’s birthday is May 18th") may be used as the first launch input, or may be processed or modified to generate the first launch input.

[0099] In various implementations, at block 503, the system uses a first startup input to start the LLM model, where the first startup input is generated based on a natural language representation of structured schedule data of an electronic schedule (e.g., “Sarah’s birthday is May 18th”) of (a first user). In various implementations, at block 505, the system uses the LLM model (which is started using the first startup input) to process a query input generated based on a received query to generate a first LLM output. For example, the LLM model can process a query input (e.g., “find Sarah’s birthday in the first user’s calendar”) generated based on a query received by the first user (e.g., “When is Sarah’s birthday”), to generate a first LLM output such as “May 18th, 2020”.

[0100] In some implementations, the received query is determined to be related to an electronic schedule. For example, referring to Figure 3A , the first user can receive a text message 331 from a sender 311 (e.g., Molly) via the conversation interface 330 of the user interface 300A of the client device. The text message 331 can include a query (i.e., “When is Sarah’s birthday”), and the query (i.e., “When is Sarah’s birthday”) can be determined to be related to the first user’s schedule based on meeting any one of one or more matching conditions. The one or more matching conditions can include, for example, a first matching condition under which the query (e.g., “When is Sarah’s birthday”) matches a schedule entry (e.g., a schedule entry titled “Sarah’s birthday” on May 18th, 2020) in the first user’s schedule. If the matching score of matching the query with the schedule entry of the first user’s schedule exceeds the matching threshold, the first matching condition can be considered to be satisfied.

[0101] Alternatively or additionally, one or more matching conditions can include, for example, a second matching condition under which the query is processed to determine whether the query includes a noun and / or a time item associated with the calendar of the first user. If it is determined that the query includes a noun and / or a time item associated with the calendar of the first user, then the second matching condition is satisfied. For example, a query (i.e., "When is Sarah’s birthday") can be determined to include a noun (i.e., "Sarah") indicating a person or a name of a person and a time item (i.e., "birthday"), such that it is determined that the second matching condition is satisfied. In response to satisfying the second matching condition, a query (i.e., "When is Sarah’s birthday") related to the electronic calendar of the first user (the electronic calendar includes a calendar entry: "Time: May 18th, 2020; Title: Sarah’s birthday") can be determined to be received.

[0102] Optionally, instead of from the text message 331, a query (e.g., "When is Sarah’s birthday") can be received from the spoken utterance of the first user (e.g., "Hi Assistant, when is Sarah’s birthday") captured by the microphone of the client device. The spoken utterance can be transmitted to the automated assistant installed at the client device to be processed as natural language content (e.g., "When is Sarah’s birthday" in natural language form), for example, using the aforementioned ASR engine 131 and / or NLU engine 133. The automated assistant can determine that the spoken utterance includes a query based on the natural language content of the spoken utterance. Based on satisfying one of the aforementioned matching conditions, the automated assistant can further determine that the query from the spoken utterance is related to the electronic calendar of the first user. A repeated description is omitted here.

[0103] In various implementations, at block 507, the system uses the first LLM output to determine a first response to the query. For example, refer to Figure 3A, the first response 333A can be "I believe Sarah’s birthday is May 18th", or "Sarah’s birthday is May 18th" (not shown), which is generated by modifying the first LLM output by expanding the first LLM output "May 18th, 2020" into a complete sentence and removing redundant (or incorrect) information such as "2020". Optionally, at block 515, the system causes the final output generated based on the first response to be rendered. For example, referring to Figure 3A , the final output can include (optionally only include) the first response 333A (e.g., "I believe Sarah’s birthday is May 18th"). The first response 333A can be selectable and, when selected, is displayed in the text input field of the dialogue interface 330. Optionally, the first response 333A can be displayed as an overlay 333 shown over a part of the dialogue interface 330 (see Figure 3A ), and if the first response 333A is not selected by the user for entry into the text input field of the dialogue interface 330 (see the dialogue interface 330 of the user interface 300B that does not display "I believe Sarah’s birthday is May 18th" in Figure 3B ), then the first response can disappear.

[0104] In some implementations, continuing with the example mentioned earlier, where the query (e.g., "When is Sarah’s birthday") is from an oral utterance (e.g., "Hi Assistant, when is Sarah’s birthday") that is captured and transmitted to the automated assistant, the final output can be rendered auditorily via the automated assistant. In this example, the first response 333A (e.g., in natural language form "I believe Sarah’s birthday is May 18th") can be converted to audio data (e.g., in speech form "I believe Sarah’s birthday is May 18th") using the aforementioned TTS engine 135 of the automated assistant to be rendered auditorily as the final output.

[0105] Optionally, referring to Figure 5 , at block 509, the system uses a second startup input to start the LLM model. The second startup input can be generated based on the natural language representation, unstructured data, or a combination thereof of the structured schedule data of the (first user's) electronic schedule. As a non-limiting example, the second startup input can include (or be generated using) the following items here: (1) the natural language representation of a schedule entry in the first user's electronic schedule (e.g., "Time: May 18th, 2020; Title: Sarah’s birthday") (e.g., "Sarah’s birthday is May 18th"); and (2) unstructured data (e.g., an email) from a source different from the electronic schedule (e.g., an email application), where the unstructured data can include content such as Jane sent an email saying "Sarah’s birthday is May 19th, let’s prepare a surprise".

[0106] Optionally, in some implementations, referring to Figure 5, at block 511, the system uses an LLM model launched with a second start input to process the query input to generate a second LLM output. Given the same example above, the second start input can include here: (1) a natural language representation (e.g., "Sarah’s birthday is May 18th"); and (2) an email sent by Jane saying "Sarah’s birthday is May 19th, let’s prepare a surprise". In this example, the query input (e.g., "find Sarah’s birthday in the first user’s calendar") can be processed using the LLM model launched with the second start input to generate a second LLM output such as "May 18th and May 19th", or "the calendar says Sarah’s birthday is May 18th, and the email from Jane says Sarah’s birth is May 19th".

[0107] Optionally, in some implementations, referring to Figure 5 , at block 513, the system determines a second response to the query based on the second LLM output. As a non-limiting example, given that the second LLM output is "the calendar says Sarah’s birthday is May 18th, and the email from Jane says Sarah’s birth is May 19th", the second response can be generated by modifying the second LLM output using one or more of the techniques described previously as: "I remember it being in May, not sure about the exact date" (see the selectable element 333B in Figure 3A ), "(Let me ask Jane)"; see Figure 3AThe selectable element 333C), and / or "I believe it’s May 18th or May 19th (I think it's May 18th or May 19th)" Figure 3A is not shown in, but if the user selects Figure 3A the selectable element 333D titled "More options (More options)" in, then it can pop up), etc. The selectable element 333C can be displayed on the user interface 330A together with the selectable element 333A, the selectable element 333B, and / or the selectable element 333D to receive user selection.

[0108] Optionally, in some implementations, the system can cause the final output generated based on the first response to be rendered (block 515) by: causing the final output generated based on the first response and / or the second response to be rendered. For example, the second response can include: (1) "I remember it being in May, not sure about the exact date (I remember it was in May, not sure about the exact date)", and (2) "Let me ask Jane (Let me ask Jane)". Before being rendered, the final output can be generated by combining the second response and the first response, and the final output can be rendered to include multiple selectable elements (e.g., 333A to 333C), each selectable element corresponding to the first response (or a part thereof) or the second response (or a part thereof).

[0109] Optionally, the user interfaces 330A to 330C can each include a bottom portion 350, where the bottom portion can receive user input by reminding the user with an instruction such as "Type your response here (Type your response here)". For example, the user can select the selectable element 333B at the user interface 300B to enter the natural language content from the second response (i.e., "I remember it being in May, not sure about the exact date (I remember it was in May, not sure about the exact date)") as a reply message, and can continue to enter more information (e.g., "Let me ask Jane… (Let me ask Jane...)") via the manual input field 351 of the bottom portion 350 of the user interface 300B. Refer to Figure 3C and the user can manually enter a follow-up response based on the selectable element 333C, i.e., "Let me ask Jane to confirm (Let me ask Jane to confirm)".

[0110] Optionally, block 509 (i.e., starting the LLM model using the second startup input) can be executed after block 507 (processing the query-based query input using the LLM model started with the first startup input). Optionally, block 509 can be executed before block 507 or before block 505. The present disclosure is not limited to this, and depending on the computing environment, the order of different blocks can be changed or rearranged to improve computing efficiency and save computing resources.

[0111] Figure 6 is a block diagram of an example computing device 610 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client computing device, a cloud-based automated assistant component, and / or other components can include one or more components of the example computing device 610.

[0112] The computing device 610 generally includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices can include a storage subsystem 624 that includes, for example, a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow a user to interact with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0113] The user interface input device 622 can include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touchscreen integrated into a display, an audio input device (such as a speech recognition system, a microphone), and / or other types of input devices. Generally speaking, the use of the term "input device" is intended to include all possible types of devices and ways for inputting information into the computing device 610 or onto a communication network.

[0114] The user interface output device 620 can include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem can include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem can also provide a non-visual display, such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include all possible types of devices and ways for outputting information from the computing device 610 to a user or another machine or computing device.

[0115] The storage subsystem 624 stores the programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected aspects of the methods disclosed herein and for implementing the various components depicted in FIGS. 1 and 2.

[0116] These software modules are typically executed by the processor 614 alone or in conjunction with other processors. The memory 625 used in the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical disk drive, or a removable media cartridge. The modules implementing the functionality of certain implementations may be stored by the file storage subsystem 626 in the storage subsystem 624 or in other machines accessible to the processor 614.

[0117] The bus subsystem 612 provides a mechanism for enabling the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0118] The computing device 610 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 610 depicted in Figure 6 is only intended as a specific example for demonstrating some implementations. Many other configurations of the computing device 610 are possible, which have more or fewer components compared to the computing device depicted in Figure 6

[0119] Figure 8A Figure 8B Figure 8C depict another example of user interaction involving queries related to calendar data implemented using, for example, the computing device 610. As shown in Figures 8A to 8C the computing device 610 can be a laptop computer 820 on which an automated assistant (not shown) is installed. Referring to Figure 8A, the first user 810 can initiate a conversation with the automated assistant by providing an oral utterance 810a (e.g., "Assistant, figure out when Alex is available to meet to discuss the draft patent application"). The laptop 820 can capture the audio data containing the oral utterance 810a and transmit the audio data containing the oral utterance 810a to the automated assistant, where the automated assistant can process the oral utterance 810a and determine that the oral utterance 810a includes a query related to the electronic calendar of a second user named "Alex" (and / or related to the electronic calendar of the first user 810). As a non-limiting example, in response to determining that the oral utterance 810a includes a query related to the electronic calendar of a second user named "Alex" (e.g., "when Alex is available to meet"), the automated assistant can process the structured calendar data of the second user's (i.e., Alex's) electronic calendar (e.g., Event: out of office; When: August 3 rd / Wednesday to August 9 th / Tuesday; Who: Alex; Description: N / A (Event: out of office; When: August 3 / Wednesday to August 9 / Tuesday; Who: Alex; Description: N / A)) to generate its natural language representation (e.g., Alex will be out of office next Wednesday until Tuesday August 9 th(Alex will be out of the office until Tuesday, August 9th next week.)) Then, the generated natural language representation of the structured schedule data of the second user's electronic schedule can be applied to generate a startup input for the LLM, where the startup input is used to start the LLM. After starting the LLM, a query input (e.g., "when is Alex available based on Alex’s electronic calendar?") generated based on a query (e.g., "when Alex is available to meet?") can be processed using the started LLM to generate an LLM output, such as "based on Alex’s calendar, he will be away starting next Wednesday for a week". The automated assistant can further process the LLM output (and / or the spoken utterance 810a) to generate a response 820a, such as "Alex should have time next Monday and Tuesday to meet and discuss. He will be away starting next Wednesday for a week."

[0120] Optionally, refer to Figure 8B, in response to receiving response 820a, the first user 810 can continue the conversation with the automated assistant by providing additional verbal utterance 810b (e.g., "Great, send a calendar invite for next Monday 2-3pm, please"). The automated assistant can process the additional verbal utterance 810b to perform one or more actions, where the one or more actions can include: constructing an invitation to a second user named Alex, sending the invitation to the second user named Alex via an applicable or designated application, and / or causing an additional response 820b (e.g., "Invite sent to Alex. You can find the invite in your email. Do you want to see it?") to be rendered to the first user 810. An example of the invitation can be found in Figure 8C , which will be described later in this disclosure. In some implementations, the additional response 820b can include confirmation that the invitation has been sent (e.g., "Invite sent to Alex"), and can identify the application via which the invitation was sent (e.g., your email). In some implementations, the additional response 820b can further include a suggestion, such as "Do you want to see the invite itself?"

[0121] Optionally, referring to Figure 8C , the first user 810 can respond to the additional response 820b by providing further verbal utterance 810c (e.g., "Yes, please"). In response to receiving the further verbal utterance 810c, the automated assistant can cause a further response 820c to be rendered auditorily, and cause the invitation 800 to be rendered via the display of the laptop 820. As Figure 8CAs shown, the invitation 800 may include a title 801 of the invitation 800 ("Invite for Meeting") and information 803 of the participants, and the information of the participants includes the requesting user (here the first user 810, with or without a title or identifier, such as "Partner") and the receiving user (here the second user named Alex, with or without a title or identifier, such as "Co-Partner"). The invitation 800 may further include time information 805, such as the proposed time of the meeting. As in this example, the proposed time of the meeting is August 1 (Monday), 2 pm to 3 pm. Optionally, as Figure 8C shown, the content (i.e., August 1 (Monday), 2 pm - 3 pm) may be selectable, and when selectable, it may be formulated and entered into the schedule entry upon user confirmation. Optionally, the invitation 800 may further include a description 807, such as "Alex, are you available to meet to discuss the draft patent application, say, next Monday 2 - 3 pm?" The description 807 may be generated by an automated assistant based on the previous spoken words of the first user 810 (e.g., spoken word 810a and additional spoken word 810b). Optionally, the invitation 800 may include a status icon 809 indicating whether the invitation 800 has been accepted by the second user (here the receiving user Alex). The status icon 809 may be clickable or selectable to send a reminder of the invitation 800. It should be noted that all examples in this disclosure are for illustrative purposes only and are not intended to be restrictive. Different features of the examples may be combined or interchanged unless they are not combinable or interchangeable.

[0122] Although several implementations have been described and shown in this document, many other means and / or structures can be utilized for performing the functions and / or obtaining the results and / or one or more of the advantages described herein, and each of these variations and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation many equivalents to the implementations described herein. Accordingly, it is to be understood that the foregoing implementations are presented by way of example only, and that implementations may be practiced otherwise than as specifically described and claimed within the scope of the appended claims and their equivalents. Implementations of the present disclosure relate to each individual feature, system, and / or method described herein. Additionally, any combination of two or more such features, systems, and / or methods, where such features, systems, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: Processing structured schedule data of an electronic schedule of a first user to generate a natural language representation of the structured schedule data of the first user; And In response to receiving a query determined to be related to the electronic schedule of the first user: Using a start input based on the natural language representation of the structured schedule data of the first user to start a large language model (LLM), wherein using the start input to start the LLM includes using the LLM to process the start input, After at least using the start input to start the LLM: Using the LLM to process a query input based on the query to generate an LLM output, and Determining a response to the query based on the LLM output, wherein the response includes a natural language response; and Causing the response to be rendered.

2. The method according to claim 1, wherein The start input includes: the natural language representation of the structured schedule data of the first user and includes the structured schedule data of the electronic schedule of the first user.

3. The method according to any one of the preceding claims, further comprising: Retrieving unstructured schedule data of the first user, the unstructured schedule data including preference data of the first user indicating preferences for one or more time slots.

4. The method according to claim 3, wherein The start input includes the natural language representation of the unstructured schedule data of the first user and the structured schedule data of the first user indicating the user availability of the first user.

5. The method according to claim 3, wherein, Determining the response to the query based on the LLM output includes: Modifying the LLM output based on the unstructured schedule data of the first user to determine the response to the query.

6. The method according to claim 3, further comprising: Using an additional start input based on the unstructured schedule data of the first user indicating the user availability of the first user to start the LLM, wherein using the additional start input to start the LLM includes using the LLM to process the additional start input.

7. The method according to claim 6, further comprising: After using the additional start input to start the LLM: Using the LLM to process the query input based on the query to generate an additional LLM output, and Determining an additional response to the query based on the additional LLM output, wherein the additional response includes an additional natural language response.

8. The method according to claim 7, wherein, Causing the response to be rendered includes: Causing the response and the additional response to be rendered simultaneously.

9. The method according to claim 7, wherein, Causing the response to be rendered includes: Combining the response with the additional response to generate a combined response, and Causing the combined response to be rendered.

10. The method according to claim 3, wherein: The unstructured schedule data of the first user includes: message data indicating the availability status of the first user during one or more time periods, and preference data indicating one or more preferred time slots of the first user.

11. The method according to any one of the preceding claims, wherein, When the natural language response indicates that the first user is available during a first time period, the method further includes: Based on the natural language response, creating an entry corresponding to the first time period in the electronic schedule of the first user.

12. The method according to claim 11, wherein, Before creating the entry in the electronic schedule of the first user, the method further includes: Requesting authorization from the first user to create the entry.

13. The method according to claim 12, wherein, Requesting authorization from the first user to create the entry includes: Forwarding the query and the content describing the entry to the first user, and Requesting user input from the first user regarding whether to create the entry, wherein the entry is created in the electronic schedule of the first user in response to the user input indicating the user permission from the first user to create the entry.

14. The method according to any one of the preceding claims, wherein, Before starting the LLM, the method further includes: Receiving a query; Determining whether the query is related to the electronic schedule of the first user, wherein the query is determined to be related to the electronic schedule of the first user in response to the query including (1) a noun indicating the first user and (2) at least one time keyword; and Starting the LLM in response to determining that the query is related to the electronic schedule.

15. The method according to any one of the preceding claims, wherein: The query is received from a second user different from the first user via a messaging application of a computing device.

16. The method according to claim 15, wherein, Causing the response to be rendered includes: Causing selectable suggestions to be rendered via the messaging application for user selection, the selectable suggestions including the natural language response.

17. The method according to any one of the preceding claims, wherein: The query is received from the first user or a third user different from the first user in spoken words captured by an automated assistant.

18. The method according to claim 17, wherein, Causing the response to be rendered includes: Causing the natural language response to be rendered aurally via the automated assistant.

19. A method implemented by one or more processors, the method including: Processing the structured schedule data of the electronic schedule of a first user to generate a natural language representation of the structured schedule data of the first user; And In response to receiving a query via an automated assistant and in response to the query being determined to be related to the electronic schedule of the first user: Using a startup input based on the natural language representation of the structured schedule data of the first user to start a large language model (LLM), wherein using the startup input to start the LLM includes using the LLM to process the startup input, After at least using the startup input to start the LLM: Using the LLM to process a query input based on the query to generate an LLM output, and Determining a response to the query based on the LLM output, wherein the response includes a natural language response; and Causing the response to be rendered via the automated assistant.

20. A method implemented by one or more processors, the method comprising: Processing structured schedule data of one or more electronic schedules to generate a natural language representation of the structured schedule data for each electronic schedule; Receiving a query; Determining whether the query is related to the one or more electronic schedules; And In response to determining that the query is related to a first electronic schedule among the one or more electronic schedules: Using a start input based on the natural language representation of the structured schedule data of the first electronic schedule to start a large language model (LLM), wherein using the start input to start the LLM includes using the LLM to process the start input, After at least using the start input to start the LLM: Using the LLM to process a query input based on the query to generate an LLM output, and Determining a response to the query based on the LLM output, wherein the response includes a natural language response; and Causing the response to be rendered.

21. A client device, comprising: At least one processor; And A memory storing instructions that, when executed, cause the at least one processor to perform operations corresponding to any one of claims 1 to 20.

22. A system, comprising: At least one processor; And A memory storing instructions that, when executed, cause the at least one processor to perform operations corresponding to any one of claims 1 to 20.

23. At least one non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 20.

Images