Biasing meeting audio and / or generating meeting documents based on meeting content and / or related data

Abstract

Embodiments relate to applications in which data that may be associated with a meeting / meeting participants may be used to bias automatic speech recognition for a meeting. Transcriptions of inputs provided during a meeting may additionally / alternatively be processed to determine whether the inputs should be incorporated into a meeting document to provide a summary of the meeting. In some examples, entries into the meeting document may be designated as action items, and those action items may have conditions that remind meeting participants about the action items and / or conditions that determine whether the action items have been fulfilled. In this manner, tasks that are typically performed manually by meeting participants, such as creating a meeting summary, may be more accurately automated. This saves resources that are wasted in video conferences, in-person meetings, and / or other gatherings.

Description

[Background technology]

[0001] As the availability of videoconferencing software increases, many users are finding that the current state of the technology has several shortcomings. For example, similar to in-person meetings, videoconferencing can involve participants discussing a variety of topics. Some participants may manually take notes during the meeting, which may be later referenced by a particular participant who may be tasked with completing action item(s) identified during the meeting. When participants take notes during a meeting in this manner, they may miss certain topics that were discussed during the meeting, and as a result, some action items may not be accurately addressed. Although some applications may function to provide a meeting transcript (e.g., a text version of most of the spoken input provided during the meeting), such a transcript may not reflect that certain topics are more important than others (e.g., a discussion of "ordering lunch" versus a discussion of an experiment conducted by a meeting participant). Thus, relying on some available transcription applications may not provide any additional efficiency with respect to streamlining the creation of related meeting documents and / or action items.

[0002] For applications that facilitate audio transcription of meetings, speech recognition may be limited in situations where a variety of subjects are discussed and / or otherwise referenced during the meeting. For example, the spoken words and phrases that occur during a meeting may be specific to an organization and / or may have been created relatively recently by the organization. As a result, typical speech recognition applications may not be able to accurately transcribe words and / or phrases that have recently been adopted into a particular industry and / or organization's vocabulary. Thus, meeting participants who rely on such transcripts may not be working with an accurate transcription when attempting to follow through on specific action items from the meeting. Summary of the Invention

[0003] Embodiments described herein relate to techniques for automating certain aspects of group conferences involving multiple users. Such aspects may include, for example, note taking and / or biasing automatic speech recognition (ASR) based on relevant conference documents and / or other content to generate conference notes. Such aspects may additionally or alternatively include generating a summary of the meeting based, for example, on a conference transcript, notes taken by at least one participant during the meeting, visual and / or audio cues captured during the meeting (with prior permission from the participant(s)), and / or explicit requests from participants during the meeting regarding content to include in the summary. Such aspects may further additionally or alternatively include generating action items from the meeting such that those action items can be linked to specific portions of the summary and / or used to create reminders that can be rendered for specific participants based on conditions that can be determined from the content of the meeting.

[0004] In some embodiments, the ASR may be biased toward transcribing content spoken by participants during a meeting to facilitate more accurate creation of meeting notes and / or other content. The ASR may be biased using content from documents and / or other files that may be selected by the meeting application and / or other applications (e.g., automated assistant applications) based on the relevance of each particular document to the meeting. For example, an upcoming meeting may have a total number of participants M (e.g., M=10), and a portion N (e.g., N=3) of the total participants M may have accessed one or more documents and / or shared one or more documents prior to the meeting. The document(s) may be determined to be associated with the upcoming meeting based on at least a threshold number or percentage of participants (or invitees) who access the document(s) prior to and / or during the meeting, and / or based on a determination that the content of the document(s) is associated with other content in the meeting invitation for the upcoming meeting (e.g., a determination that the content of the document is similar to the title of the meeting invitation).

[0005] Once a document is determined to be relevant to the upcoming meeting, the ASR used to transcribe audio during the upcoming meeting may be biased according to the content of the document. For example, a document accessed by some of the participants in the upcoming meeting may include multiple instances of the term "Cardinal," which may refer to a product that will be discussed at the upcoming meeting. During the upcoming meeting, participants may frequently speak the term "Cardinal" in relation to the product, and using the ASR to process audio data generated during the meeting, candidate interpretations of the audio that embody the spoken term "Cardinal" may be identified. For example, the candidate interpretations may include "garden hole," "card in a," "guard the ball," and "Cardinal." Each candidate interpretation may be assigned a score based on a variety of different factors (e.g., the relevance of the interpretation to other recent audio, context, location, etc.), and the candidate interpretation with the highest score may be incorporated into the meeting document generated by the meeting application and / or other applications. However, in some implementations, one or more respective scores may be weighted according to whether their corresponding candidate interpretations are associated with a document(s) accessed by some of the participants. For example, the score of a candidate interpretation "cardinal" may be increased and / or the scores of other candidate interpretations may be decreased based on the explicit appearance of the term "cardinal" in a document(s) accessed by some of the conference participants.

[0006] In some implementations, the score may be based on the term frequency (TF) of the term "cardinal" appearing in different documents determined to be associated with the conference and / or one or more participants, and / or the inverse document frequency (IDF) of the term "cardinal" in another corpus, such as a global corpus of Internet documents and / or a corpus of training instances used in the training model(s) used in the ASR. In other words, a score for a term may be generated such that it is more heavily biased against a term if it has a higher TF and / or a lower IDF as opposed to a lower TF and / or a higher IDF. For example, a given term may be more heavily biased if it appears frequently in the document(s) determined to be associated with the conference and / or if it is not included in the training example(s) used to train any (or very few) ASR model(s). These and other methods may improve the ASR of speech during a meeting, and may further allow automated note-taking and / or other functions that rely on the ASR results to be performed more accurately. This may conserve computational resources by reducing the number of inputs that participants must manually provide to each device to edit incorrect ASR entries. This may also encourage more participants to actively rely on ASR-based features over other manual features, which may distract participants from engaging with other meeting participants during the meeting.

[0007] In some embodiments, the conference application and / or other applications may additionally or alternatively generate a summary, action item(s), and / or other content for the conference based on various features of the conference and / or interactions that may occur during the conference. Such features may include participant note taking, audio from one or more participants, direct and / or indirect requests from participants, visual content from the conference, gestures from one or more participants, and / or any other features that may indicate content for incorporation into the summary. In some embodiments, some of the content included in the summary may be generated in response to multiple different participants speaking during a portion of the conference related to a particular topic. For example, after a participant has been speaking for a period of time, multiple other participants may provide feedback regarding a particular topic that the one participant was referring to. As a result, topic-based summary items may be generated for the summary document to facilitate creating a summary that includes content that is important to a greater number of conference participants.

[0008] In some embodiments, summary items for a meeting summary may be automatically generated based on associations between the content of the meeting (e.g., audio from one or more participants) and other content related to the meeting (e.g., the title of the meeting invitation, the content of attachments to the meeting invitation, the content of files accessed by the meeting participants before, during, and / or after the meeting). For example, the title of the meeting invitation may be "Meeting on Phase II Cellular Trials" and attachments provided with the meeting invitation may include a spreadsheet of clinical trial data. At the start of the meeting, a participant may ask, "How was everyone's weekend?" and other participants may respond by briefly describing their own weekend (e.g., "It was great. We went to a concert on the waterfront."). However, because terms such as "weekend," "concert," and "waterfront" do not appear in the meeting title or the meeting attachments, a summary may be generated that does not mention all of the content from this portion of the meeting.

[0009] According to the above example, somewhere during the "Meeting on Phase II Cellular Trials" while the second participant is speaking about "Results of Batch T", the first participant may raise his hand (actually and / or virtually via a "raise hand" interface element) and make a request such as "Bill, I don't think the results of Batch T are complete. Can you check it after the meeting?" Image data embodying the first participant's raising of hand may be captured by a video camera (with prior permission from the participant(s)) and processed using one or more trained machine learning models and / or one or more heuristic processes. Alternatively or additionally, the content of the spoken request from the first participant may be captured as audio data (with prior permission from the first participant) and processed using one or more trained machine learning models and / or one or more heuristic processes. Based on these processes, the conference application and / or other applications may generate summary items to be incorporated into the summary being generated for the conference. For example, language processing may be used to determine that terms in a conference title (e.g., "...Phase II cellular trial...") may often be associated with terms such as "results." Based on this determination, content from a request from a first participant to a second participant may be ranked for inclusion in the summary above other conference content (e.g., "How was everyone's weekend?") that may not be deemed relevant enough to be incorporated into the summary.

[0010] In some implementations, a summary item may be incorporated into the meeting summary document based on a threshold number (N) of individuals determined (with prior permission from the participants) to be making notes on a topic during the meeting. A summary item may then be generated to address the discussion of that particular topic. Alternatively or additionally, the attention level(s) of the participant(s) with respect to a particular topic may be determined (with prior permission from the participants) and / or a change in the attention level(s) of the participant(s) during the discussion of the particular topic may be determined. Based on an increase or change in the attention level during the discussion of the particular topic, the particular topic may be made the subject of a summary item to be included in the summary document or other documents related to the meeting. In some implementations, determining the attention level of the participants may be performed using one or more cameras, with prior permission from the participants, during the meeting, which may be a face-to-face meeting, a virtual video conference (e.g., where all participants connect to the meeting via an internet or other network connection), and / or any meeting having a combination of remote and face-to-face participants.

[0011] In some embodiments, summary items generated by the conference application and / or automated assistant may be made into "action items" based on at least the corresponding conference content (e.g., the content that served as the basis for the action item) and / or the context in which the corresponding conference content was presented. Following the above example, content provided by a first participant to a second participant (e.g., "Bill") (e.g., "...I don't think the results of Batch T are complete. Can you check it after the meeting?") may be incorporated into the conference summary as an action item for the second participant. The action item may be included in the conference summary along with an embedded link to any file that the action item may reference (e.g., a "Results of Batch T" document) and / or along with a reminder for the second participant and / or the first participant. In some embodiments, in response to one or more conditions being met, a reminder may be rendered to the first participant, the second participant, and / or any other person. The conditions may be selected, for example, based on the content and / or context of the conference. For example, the second participant may receive notification regarding the action item after the conference in response to the second participant accessing the "Batch T Results" document. Alternatively or additionally, the first participant and / or the second participant may receive notification regarding the action item after and / or during another conference to which the first and second participants are invitees. Alternatively or additionally, the first participant and / or the second participant may receive notification regarding the action item in response to receiving and / or sending a message to another attendee (e.g., a third participant) of the conference from which the action item was derived.

[0012] In some embodiments, the meeting summary is non-static and / or generated in real-time as the meeting progresses, which may enable participants to verify that certain items are included in the summary and / or to further modify the summary before the meeting is concluded. For example, an action item that is automatically included in the meeting summary may be modified by a participant to target one or more additional participants that the action item was not originally intended for. Alternatively, or additionally, a summary of the topics discussed that is automatically included in the meeting summary before the meeting ends (e.g., "Batch B Results") may be editable to add, remove, modify, and / or otherwise change the topic summary. In some embodiments, portions of the automatically generated summary may be automatically edited, for example, when a particular topic is revisited during the meeting, when additional contextual data becomes available, when additional content (e.g., meeting attachments, documents, files, etc.) becomes available and / or is otherwise accessed by a participant, and / or when additional meeting information becomes otherwise available.

[0013] The above description is provided as a summary of some embodiments of the present disclosure. The details of these and other embodiments are described in more detail below.

[0014] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., central processing unit (CPU)(s), graphics processing unit (GPU)(s), and / or tensor processing unit (TPU)(s)) to perform methods, such as one or more of the methods described above and / or elsewhere herein. Still other embodiments may include systems of one or more computers including one or more processors operable to execute the stored instructions to perform methods, such as one or more of the methods described above and / or elsewhere herein.

[0015] It should be understood that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]

[0016] [Figure 1A] 1 shows a view of the audio biasing and document generation being performed for a conference based on data created before and / or during the conference. [Figure 1B] 1 shows a view of the audio biasing and document generation being performed for a conference based on data created before and / or during the conference. [Figure 1C] 1 shows a view of the audio biasing and document generation being performed for a conference based on data created before and / or during the conference. [Figure 1D] 1 shows a view of the audio biasing and document generation being performed for a conference based on data created before and / or during the conference. [Figure 1E] 1 shows a view of the audio biasing and document generation being performed for a conference based on data created before and / or during the conference. [Diagram 2] A system is shown that provides applications, such as automated assistants and / or conferencing applications, that can bias ASR and / or generate meeting documents based on data generated before and / or during the meeting. [Diagram 3] A method is presented for biasing automatic speech recognition according to instances of data determined to be relevant to the meeting and / or meeting participants before and / or during the meeting. [Figure 4]We present a method for automatically incorporating specific content into meeting documents to facilitate generating meeting summaries and / or other types of documents based on the content of the meeting. [Diagram 5] We show how to generate action items based on natural language content provided during a participant meeting, remind specific participants of action items, and / or designate action items as completed based on specific conditions. [Figure 6] 1 illustrates an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] 1A, 1B, 1C, 1D, and 1E show views 100, 120, 140, 160, and 180, respectively, of audio biasing and document generation being performed for a conference based on data created before and / or during the conference. Such operations may be performed to minimize the number of inputs a user must manually enter during a conference, thereby streamlining certain types of conferences and conserving computational resources of conference-related devices. Additionally, the accuracy of certain conference documents may be improved using certain processes that generate conference summary documents and / or conference action items.

[0018] Prior to the conference, as shown in FIG. 1A, a first user 102 (e.g., an invitee and / or participant) may have accessed, via their respective computing device 104, a conference invitation 114 that has been communicated to the first user 102 by a conference application 106. The conference invitation 114 may have a title that is rendered in an application interface 108 of the conference application 106, and the title may include terms that may serve as a basis for biasing the ASR during the conference. Alternatively, or in addition, the content of the conference invitation 114 (i.e., the first document) may include words and / or phrases (i.e., terms) that may indicate whether a particular topic to be discussed during the conference is sufficiently relevant to be included in the conference document and / or designated as an action item. Other users receiving the conference invitation 114, such as a second user 110 operating an additional computing device 112, may also influence whether particular data is considered relevant to the conference.

[0019] For example, the second user 110 may be viewing the collaborative spreadsheet 122 via their respective computing devices 112 as shown in view 120 of FIG. 1B. The collaborative spreadsheet 122 (i.e., the second document) may be accessible to one or more meeting participants and thus may be considered relevant to the meeting by the meeting application and / or other supporting applications. For example, the collaborative spreadsheet 122 may be a cloud-based document, created / owned by one of the meeting invitees and accessible to all of the meeting invitees by being shared with all other meeting invitees. The collaborative spreadsheet 122 may be determined to be relevant to the meeting based on being shared with at least a threshold number or a threshold percentage of the meeting invitees (e.g., all) and / or based on whether the document was presented to one or more participants during the meeting. Optionally, determining that the collaborative spreadsheet 122 is relevant may also be based on determining that the collaborative spreadsheet 122 is shared with less than a threshold number or percentage of individuals who are not meeting invitees. For example, a first document shared only with all meeting invitees may be determined to be relevant to the meeting, while a second document shared with all meeting invitees and also with N additional (e.g., 50 additional) individuals who are not meeting invitees may be determined to be not relevant to the meeting. Alternatively, or additionally, a collaborative spreadsheet 122 may be deemed relevant to the meeting by the meeting application based on the content of the collaborative spreadsheet 122. For example, the collaborative spreadsheet 122 may include content that is associated with the content of the meeting invitation 114, thereby indicating that the collaborative spreadsheet 122 is relevant to the meeting. The content of the collaborative spreadsheet 122 may include, for example, the price of ingredients 124 for "hummus," which may be a term referenced in the meeting invitation (e.g., in the title of the meeting invitation and / or the description section of the meeting invitation).Based on this correspondence, various portions of the document content may be processed (e.g., using inverse document frequency and / or other document review processes) to identify portions that may be relevant for ASR bias and / or other process(es) to be performed during the meeting. These portions may then be utilized as a basis for biasing the ASR during the meeting, as a basis for identifying relevant content (e.g., input from participants) during the meeting, and / or as a basis for identifying conditions for action item reminders and / or conditions for action item fulfillment.

[0020] In some embodiments, the conferencing application, computing device, and / or server device 142 may determine that a conference has started (e.g., as shown in FIG. 1C) based on data available from one or more devices and / or applications. For example, calendar data and / or data from the conferencing application may be utilized to determine that a conference has started and / or that one or more people have joined the conference. Once the conference has started, data 146 from various devices may be processed to facilitate biasing the ASR and generating a conference document. This conference document may include the conference content 144 and may not include any data that is not related to the conference (e.g., other topics such as "small talk" during breaks in the conference). In some cases, the data 146 may include audio data embodying voices from various participants of the conference.

[0021] For example, the data 146 may characterize spoken words 148 provided by a third user 150, such as "Let's think of some peppers to add to the hummus." The data 146 may be processed using ASR biased according to instances of data identified as relevant to the meeting. For example, a meeting invitation 114 having a title that includes the specific term "hummus" may have one or more candidate terms, such as "hummus" and / or "recipe," assigned a higher probability and / or weight value to a transcription of the spoken words 148 than other words and / or phrases that may be pronounced similarly (e.g., "honeys" for "hummus" and "rest in peace" for "recipe"). Alternatively, or additionally, the resulting transcription of the spoken words 148 may be processed to determine whether the content of the transcription is sufficiently relevant to the meeting to be included in the meeting content 144 for the meeting document. For example, if the content of the transcription includes the terms "hummus" and "recipe" and the data accessed before and / or during the meeting includes the terms "hummus" and "recipe," the content of the transcription may be deemed sufficiently relevant to be incorporated into the meeting documents. Following the example above, because the second user 110 viewed hummus ingredients before the meeting and the invitation 114 to the meeting includes "hummus recipe" in the title, the content of the transcription of the spoken words 148 may be deemed sufficiently relevant to be included in the meeting content 144 and / or the meeting documents.

[0022] In some implementations, non-verbal gestures and / or other non-verbal cue(s) captured by one or more sensors during the conference may be utilized to determine relevance of input from participants to the conference. For example, in response to spoken words 148 from the third user 150, the first user 102 may provide a separate spoken word 162, such as "Yes, I'll create a list and send it to Jeff for pricing." The first user 102 may also perform non-verbal gestures 164 while providing the spoken words 162 and / or within a threshold duration of providing the spoken words 162 that may indicate the importance of what they are saying. With prior permission from the participant(s), audio and image data captured by the camera 156 and computing device 154 may be processed at the local computing device and / or server device 142 to determine whether to incorporate responses from the first user 102 into the conference document. Additionally or alternatively, the data may be processed to determine whether to generate an action item 166 based on one or more conditions of the spoken words 162 and / or the action item 166 .

[0023] For example, the audio and / or video data may be processed using one or more heuristic processes and / or one or more trained machine learning models to determine whether the text entry should be included in the conference document. In some embodiments, the determination may be based on whether the spoken words 162 were provided within a threshold duration during which the third user 150 provided the spoken words 148. Alternatively, or additionally, the determination of whether the text entry should be included in the conference document may be based on whether the spoken words 162 are responsive to the conference-related input (e.g., the spoken words 148) and / or whether the spoken words 162 are directed to the person who provided the conference-related input. In some embodiments, the text entry may be assigned a score according to one or more of these and / or other factors, and the score may be compared to a score threshold. If the score meets the score threshold, the text entry may be incorporated into the conference document (e.g., a conference “summary” document).

[0024] Once a text entry is determined to be incorporated into the meeting document, a determination can be made as to whether the text entry is an action item and, if so, whether the action item should have a condition. For example, the text entry corresponding to the spoken words 162 can be designated as an action item based on the first user 102 expressing an intention to take an action (e.g., "make a list"). Alternatively, or in addition, the action item can be assigned one or more conditions based on the content of the text entry and / or the context in which the spoken words 162 were provided. For example, the action item can be stored with conditional data characterizing a reminder that can be rendered the next time the first user 102 communicates with the second user 110 (e.g., "Jeff"). Alternatively, or in addition, the action item can be stored with conditional data indicating that the action item will be fulfilled when the first user 102 communicates "Pepper's list" to the second user 110. In this manner, not only can action items be incorporated into the meeting document to accurately track action items, but reminders can also be established and / or action items can be automatically updated based on user actions (with prior permission from the user(s)).

[0025] FIG. 1E illustrates a view 180 of a summary document 182 that may be automatically created by the conference application and / or other applications based on the content of the meeting and / or data related to the meeting. For example, the summary document 182 may include a list of summary items that summarize various topics discussed during the meeting and action items identified during the meeting. In some implementations, the summary items may embody terms that may or may not have been explicitly stated during the meeting, either verbally or in writing. For example, a summary item may include "The group agreed that ingredients in hummus should include peppers," which may be a sentence that was not explicitly stated in those terms during the meeting. Alternatively, or additionally, the summary document may be generated with a list of action items identified during the meeting. In some implementations, the action items may be generated to include reminders (e.g., reminders before the next meeting) that may be rendered for certain participants when certain conditions are met. Alternatively or additionally, the action items may include embedded links to specific data (e.g., documents, websites, images, contact information, and / or any other data), such as a participant's electronic address (e.g., "@Jeff") and / or a specific reminder (e.g., a reminder before the next meeting). In some embodiments, the summary document 182 may be viewed during the meeting, thereby allowing participants to edit the meeting document 182 as it is being created. Alternatively or additionally, the summary document 182 may have editable embedded data, such as allowing participants to edit when a particular reminder is rendered and / or whether an action item is still outstanding.

[0026] FIG. 2 illustrates a system 200 that provides an application, such as an automated assistant and / or a conferencing application, that can bias ASR and / or generate conferencing documents based on data created before and / or during a meeting. The automated assistant 204 can operate as part of an assistant application provided on one or more computing devices, such as the computing device 202 and / or a server device. A user can interact with the automated assistant 204 through the assistant interface(s) 220, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device that can provide an interface between one or more users and an application. For example, a user can initialize the automated assistant 204 by providing verbal, textual, and / or graphical input to the assistant interface 220 to cause the automated assistant 204 to initialize one or more actions (e.g., providing data, controlling peripheral devices, accessing agents, generating input and / or output, etc.). Alternatively, the automated assistant 204 can be initialized based on processing of the context data 236 using one or more trained machine learning models.

[0027] The context data 236 can characterize one or more features of an environment accessible to the automated assistant 204 and / or one or more features of a user who is predicted to be intended to interact with the automated assistant 204. The computing device 202 can include a display device, which can be a display panel including a touch interface for receiving touch inputs and / or gestures to allow a user to control the application 234 of the computing device 202 via a touch interface. In some embodiments, the computing device 202 may not have a display device and thus provide an audible user interface output without providing a graphical user interface output. Additionally, the computing device 202 can provide a user interface, such as a microphone, to receive spoken natural language content from a user. In some embodiments, the computing device 202 can include a touch interface and may not have a camera, but may optionally include one or more other sensors.

[0028] The computing device 202 and / or other third party client devices may communicate with the server device via a network such as the Internet. Additionally, the computing device 202 and any other computing devices may communicate with each other via a local area network (LAN) such as a Wi-Fi network. The computing device 202 may offload computing tasks to a server device to conserve computing resources at the computing device 202. For example, the server device may host the automated assistant 204 and / or the computing device 202 may send inputs received at one or more assistant interfaces 220 to the server device. However, in some embodiments, the automated assistant 204 (e.g., a conferencing application) may be hosted at the computing device 202, and various processes that may be associated with the automated assistant operation may be executed at the computing device 202.

[0029] In various implementations, all or less than all aspects of the automated assistant 204 may be implemented on the computing device 202. In some of those implementations, aspects of the automated assistant 204 may be implemented via the computing device 202 and interface with a server device that may implement other aspects of the automated assistant 204. The server device may optionally provide services to multiple users and their associated assistant applications via multiple threads. In implementations in which all or less than all aspects of the automated assistant 204 are implemented via the computing device 202, the automated assistant 204 may be an application separate from (e.g., installed "on") the operating system of the computing device 202, or may instead be implemented directly by (e.g., considered an operating system application but integral to the operating system) the operating system of the computing device 202.

[0030] In some embodiments, the automated assistant 204 can include an input processing engine 206 that can use multiple different modules to process input and / or output of the computing device 202 and / or the server device. For example, the input processing engine 206 can include a voice processing engine 208 that can process audio data received at the assistant interface 220 to identify text embodied in the audio data. The audio data can be transmitted from the computing device 202 to a server device, for example, to conserve computational resources at the computing device 202. Additionally or alternatively, the audio data can be processed exclusively at the computing device 202.

[0031] The process for converting audio data to text may include speech recognition algorithms that may use neural networks and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data may be analyzed by the data analysis engine 210 and made available to the automated assistant 204 as text data that may be used to generate and / or identify command phrase(s), intent(s), action(s), slot value(s), and / or any other content specified by the user. In some embodiments, the output data provided by the data analysis engine 210 may be provided to the parameter engine 212 to determine whether the user has provided input that corresponds to a particular intent, action, and / or routine that may be executed by the automated assistant 204 and / or an application or agent accessible via the automated assistant 204. For example, the assistant data 238 may be stored on the server device and / or the computing device 202 and may include data defining one or more actions that the automated assistant 204 may execute, and parameters necessary to execute the action. The parameter engine 212 may generate one or more parameters for the intent, action, and / or slot value and provide the one or more parameters to the output generation engine 214. The output generation engine 214 may use the one or more parameters to communicate with the assistant interface 220 to provide output to the user and / or to communicate with one or more applications 234 to provide output to the one or more applications 234.

[0032] In some embodiments, the automated assistant 204 may be an application that can be installed "on top of" the operating system of the computing device 202 and / or that itself may form part (or the entirety) of the operating system of the computing device 202. The automated assistant application may include and / or access on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, the on-device speech recognition may be performed using an on-device speech recognition module that processes the audio data (detected by the microphone(s)) using an end-to-end speech recognition machine learning model stored locally on the computing device 202. The on-device speech recognition generates recognized text for the spoken words present in the audio data (if any are present in the audio data). In some embodiments, the speech recognition may be biased according to the operation of the ASR bias engine 218, which may actively bias certain instances of audio according to available data prior to and / or during the capture of the audio. Also, for example, on-device natural language understanding (NLU) may be performed using an on-device NLU module that processes recognized text generated using on-device speech recognition, and optionally processes contextual data, to generate NLU data.

[0033] The NLU data may include intent(s) corresponding to the spoken words and, optionally, parameters (e.g., slot values) of the intent(s). On-device fulfillment may be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU), and optionally other local data, to determine action(s) to take to resolve the spoken word intent(s) (and optionally parameter(s) for the intents). This may include determining local and / or remote responses (e.g., replies) to the spoken words, interaction(s) with locally installed application(s) to perform based on the spoken words, command(s) to send (directly or via corresponding remote system(s)) to Internet of Things (IoT) device(s) based on the spoken words, and / or other resolution action(s) to perform based on the spoken words. The on-device fulfillment may then initiate local and / or remote implementation / execution of the determined action(s) to resolve the spoken words.

[0034] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment may be utilized, at least selectively. For example, recognized text may be at least selectively sent to a remote automated assistant component(s) for remote NLU and / or remote fulfillment. For example, recognized text may be optionally sent for remote performance in parallel with on-device performance or in response to a failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized due to the reduced latency they provide at least when resolving spoken words (due to the absence of client-server round-trip(s) to resolve spoken words). Additionally, on-device functionality may be the only functionality available in situations where there is no or limited network connectivity.

[0035] In some embodiments, the computing device 202 may include one or more applications 234 that may be provided by a third party entity different from the entity that provided the computing device 202 and / or the automated assistant 204. The application state engine of the automated assistant 204 and / or the computing device 202 may access the application data 230 to determine one or more actions that may be performed by the one or more applications 234, as well as the state of each application of the one or more applications 234 and / or the state of each device associated with the computing device 202. The device state engine of the automated assistant 204 and / or the computing device 202 may access the device data 232 to determine one or more actions that the computing device 202 and / or one or more devices associated with the computing device 202 may perform. Additionally, the application data 230 and / or any other data (e.g., device data 232) can be accessed by the automated assistant 204 to generate context data 236 that can characterize the context in which a particular application 234 and / or device is running, and / or the context in which a particular user is accessing the computing device 202, the application 234, and / or any other device or module.

[0036] While one or more applications 234 are executing on the computing device 202, the device data 232 can characterize a current operational state of each application 234 executing on the computing device 202. Additionally, the application data 230 can characterize one or more features of the executing applications 234, such as the content of one or more graphical user interfaces rendered at the direction of the one or more applications 234. Alternatively, or in addition, the application data 230 can characterize action schemas, which can be updated by the respective applications and / or by the automated assistant 204 based on the respective applications' current operational state. Alternatively, or in addition, one or more action schemas for one or more applications 234 can remain static, but can be accessed by the application state engine to determine suitable actions to initialize via the automated assistant 204.

[0037] The computing device 202 may further include an assistant invocation engine 222 that may process the application data 230, the device data 232, the context data 236, and / or any other data accessible to the computing device 202 using one or more trained machine learning models. The assistant invocation engine 222 may process this data to determine whether to wait for the user to explicitly speak an invocation phrase to invoke the automated assistant 204, or whether to consider the data as indicative of the user's intent to invoke the automated assistant instead of requiring the user to explicitly speak an invocation phrase. For example, the one or more trained machine learning models may be trained using instances of training data based on scenarios in which a user is in an environment in which multiple devices and / or applications exhibit various operating states. The instances of training data may be generated to capture training data that characterize contexts in which the user invokes an automated assistant and other contexts in which the user does not invoke an automated assistant. When the one or more trained machine learning models are trained according to these instances of training data, the assistant invocation engine 222 may cause the automated assistant 204 to detect or be limited from detecting a voice input from the user based on features of the context and / or environment.

[0038] In some embodiments, the system 200 can include a relevant data engine 216 that can process data from various sources to determine whether the data is relevant to an upcoming meeting and / or other gathering of one or more people. For example, the relevant data engine 216 can utilize one or more heuristic processes and / or one or more trained machine learning models to process the data to determine whether a meeting is expected to occur or is occurring. Based on this determination, the relevant data engine 216 can process data from various sources (e.g., various devices, applications, servers, and / or any other source that can provide data related to meetings) to determine whether the data is relevant to a particular meeting. In some embodiments, the relevance of the data can be characterized by a metric (i.e., a score) that can be compared to a relevance threshold. If the metric meets the relevance threshold, the data can be considered relevant to the meeting. For example, one or more trained machine learning models can be utilized to generate embeddings from data that can be relevant to a meeting. For example, the trained machine learning model(s) may include Word2Vec, BERT, and / or other model(s) that can be used to process data (e.g., text data) and generate semantically meaningful reduced-dimensional embeddings in a latent space. The embeddings may be in a latent space, and the distance between the embeddings and the meeting embeddings (also mapped to the latent space) may be characterized by a metric. The metric may be compared to a relevance threshold in determining whether the data is relevant to the meeting (e.g., if the distance is closer than the threshold, the data may be determined to be relevant). The meeting embeddings may be generated based on processing one or more features of the meeting with one or more trained machine learning models (e.g., models used to generate the data embeddings).For example, the meeting characteristic(s) may include the title of the meeting, a description or notes included in the meeting invitation, the time of the meeting, the time the meeting invitation was scheduled, the number of participants in the meeting, and / or any other characteristics associated with the meeting.

[0039] In some embodiments, once one or more instances of data related to the conference are identified, the ASR bias engine 218 may process the one or more instances of the data to determine whether to bias the ASR based on the content of the data. For example, terms within the content of the data may be identified using one or more heuristic processes and / or one or more trained machine learning models. In some embodiments, an inverse document frequency (IDF) measure of the terms within the data may be identified and utilized to determine whether the ASR should be biased toward that particular term. For example, the IDF measure may be based on the frequency of the term in the spoken language utilized to train the ASR model(s) utilized in the ASR. For example, a first term within the data, such as "garbanzo," may be selected for use to bias the ASR based on having a high IDF measure measure (e.g., very few utterances used to train the ASR model(s) contain "garbanzo"). Meanwhile, a second term within the data, such as "oil," may not be selected for use to bias the ASR based on having a high IDF measure. Further, in some embodiments, the degree of bias of a term may be a function of its IDF measure and / or term frequency (TF) measure (e.g., a function of the frequency with which the term appears in the data). Additional and / or alternative techniques may be utilized in determining whether a term is important to the data and / or meeting instance. For example, where the term is used in the document (e.g., use in the title, first sentence, and / or conclusion may be more important than use in the "middle" of the document), and / or whether the term is also used in the meeting invitation. Once a particular term is determined to be important to the data and / or meeting instance, the particular term may be utilized in biasing the ASR during the meeting. For example, the particular term may be weighted and / or assigned a higher value score or probability during the ASR of input spoken during the meeting accordingly.

[0040] In some embodiments, the system 200 can include a document input engine 226, which can utilize data generated by the ASR bias engine 218 and / or the input processing engine 206 to automatically generate a meeting document(s). The meeting document can be generated, for example, to represent a summary of the meeting and thus can describe points relevant to the meeting while omitting content of the meeting discussion that may not be relevant to the meeting. In some embodiments, embeddings can be generated for text and / or audio data transcribed during the meeting, and the embeddings can be mapped to a latent space that can also include embeddings of the meeting. If it is determined that the embedding distance between the embedding and the meeting embedding meets a threshold for incorporating the text entry into the meeting document, the text entry can be incorporated into the meeting document. In some embodiments, the candidate text entries can be rendered in an interface during the meeting, and participants and / or others can choose to incorporate the text entry into the meeting document regardless of whether the embedding distance meets the threshold.

[0041] In some embodiments, the system 200 can include an action item engine 224 that can determine whether a text entry should be designated as an action item in a meeting document and / or other data and / or whether an action item should have certain conditions. For example, one or more heuristic processes and / or one or more trained machine learning models can be utilized to determine whether an input from a meeting participant should be considered an action item. For example, an action item can refer to an input that describes a task to be completed by at least one participant, or others, following an input provided during the meeting. Thus, an input that embodies a request to another participant and may optionally have a deadline (e.g., "Let's follow up on the budget at the next meeting") can be assigned a higher value action item score than other inputs that may not embody a request or deadline (e.g., "Did you enjoy your lunch?").

[0042] In some implementations, the content of the input on which the action item is based, along with any other relevant data, may be processed to identify conditions to be stored in association with the action item's input. For example, reminder conditions and / or fulfillment conditions may be generated based on the content of the text entries and / or other inputs, and / or any data associated with the input during the meeting. In some implementations, a condition explicitly provided in the input (e.g., "Please send the report after the meeting") may be processed to generate a condition that can be stored with the action item (e.g., actionItem("Send report", nextMeetingTime(), reminderEmail())). Alternatively, or additionally, conditions inferred from data associated with the input (e.g., documents accessed during the meeting) may be utilized to generate the action item's condition (e.g., actionItem("Send report", nextMeetingTime(), fulfillmentCondition(email, "report," "budget," jeff@email.com))). In this way, action items can be generated automatically during a meeting without requiring manual user input that can disrupt a user's participation in the meeting and waste computational resources on a particular computing device and each user's interface.

[0043] FIG. 3 illustrates a method 300 for biasing automatic speech recognition according to instances of data determined to be related to a conference and / or conference participants before and / or during a conference. Method 300 may be performed by one or more computing devices, applications, and / or any other devices or modules that may be associated with an automated assistant. Method 300 may include an operation 302 of determining whether a conference is occurring or is expected to occur. The determination at operation 302 may be performed by an application, such as a conference application and / or an automated assistant application, accessible via a computing device (e.g., a server device, a portable computing device, etc.). The determination at operation 302 may be performed to facilitate a user associated with the application determining whether to participate in a conference in which one or more participants may communicate information to one or more other participants. In some embodiments, the determination may be based on data accessible to the application, such as contextual data (e.g., a schedule stored by the application) and / or other application data (e.g., a conference invitation provided to multiple invitees).

[0044] Method 300 may proceed from operation 302 to operation 304, which may include determining whether any instances of data related to the conference are available. The instances of data may be determined to be associated with the conference using one or more heuristic processes and / or one or more trained machine learning models. For example, data associated with one or more invitees and / or participants of a conference may be processed (with prior permission from a user) to determine whether the data is related to the conference. The data may include files (e.g., documents) that one or more invitees of the conference have permission to access and / or that were accessed within a threshold duration prior to the conference. In some implementations, the duration may be based at least in part on the time that at least one invitee received a conference invitation for the conference. For example, the threshold duration may be directly proportional to the amount of time between the time an invitation to the conference was first sent or received by at least one invitee and the scheduled time of the conference. In this manner, the threshold duration prior to the conference during which relevant files may have been accessed may be longer for conferences that are planned further in advance. Alternatively, the threshold duration may be based on other factors, such as the duration of the meeting, the number of invitees to the meeting, the location of the meeting, and / or any other characteristic that may be specified for the meeting.

[0045] In some implementations, an instance of data may be determined to be relevant to a meeting based on content of the instance of data compared to content of data provided with a meeting invitation (e.g., a title of the meeting invitation, a description within the meeting invitation, attachments to the meeting invitation, etc.). For example, content of a file may be determined to be relevant to a meeting if terms within the file are also present and / or synonymous with terms in the meeting invitation. In some implementations, specific terms determined to be relevant to characterizing a particular file and comparing to a meeting invitation may be identified using an inverse document frequency metric for those specific terms. Alternatively, or additionally, specific terms determined to be relevant to characterizing a particular file and comparing to a meeting invitation may be identified using contextual data associated with the particular file.

[0046] In some embodiments, an instance of data may be determined to be relevant to a conference based on the number of participants with whom a particular instance of data is shared and / or whether a particular instance of data includes a term(s) corresponding to content related to the conference (e.g., a conference invitation, audio captured during the conference with prior permission from the participants, documents created and / or shared by the participants, etc.). For example, a document that does not include terms associated with a conference invitation (e.g., no terms deemed relevant according to IDF) but is shared with 80% of the conference participants may not be considered relevant for purposes of ASR biasing. However, a document that includes one or more terms associated with a conference invitation (e.g., terms deemed relevant according to IDF) and is shared with 60% of the conference participants may be considered relevant for purposes of ASR biasing. In some embodiments, the extent to which a term embodied in an instance of data is considered relevant may be based on a variety of different characteristics of the conference. Alternatively, or in addition, the threshold for the number of participants with which a document is shared before the document is considered relevant may be based on the number of relevant terms in the document (e.g., the percentage threshold may be inversely proportional to the degree of relevance of particular document terms).

[0047] Alternatively, or in addition, an instance of data may be deemed relevant or not relevant based on whether a threshold percentage of participants accessed the data during the meeting. For example, a document may be deemed not very relevant based on only one participant accessing the document during a large portion of the meeting (e.g., the one participant may not be paying full attention and may be distracted by content not related to the meeting). Rather, a document may be deemed relevant if at least a threshold percentage of participants accessed the data (e.g., document) during the meeting and / or if a threshold percentage of participants accessed the data for a threshold duration (e.g., at least a threshold percentage of the total scheduled time of the meeting). In this way, the ASR may be biased according to the terms of documents considered relevant to the meeting during the meeting without considering certain data that may only be relevant to an individual during the meeting.

[0048] Once it is determined that the instance of data is associated with a conference, the method 300 may proceed from operation 304 to operation 306, which may include determining whether the content of the data satisfies a condition(s) for using the content as a basis for automating speech recognition biasing. In some implementations, the content of the data may satisfy the condition for using the content as a basis for biasing the automatic speech recognition when it is determined that the embedding of the content is a threshold distance in the latent space from the embedding of the conference. In other words, the conference data (e.g., the conference invitation, the attachment of the conference, etc.) may be processed using one or more trained machine learning models to generate embeddings of the conference. Furthermore, the content of the data may also be processed using one or more trained machine learning models to generate content embeddings. Each embedding may be mapped to a latent space, and their distance in the latent space may be determined. When the distance between the embeddings meets the distance threshold, the condition for biasing the automatic speech recognition based on the content of the data and / or one or more terms in the content of the data may be satisfied.

[0049] Alternatively or additionally, the data content may satisfy a condition for biasing automatic speech recognition based on a term in the data content if characteristics of the term in both the data content and the conference data satisfy one or more conditions. For example, a condition for biasing automatic speech recognition may be satisfied if a term shared by both the data content and the conference data is determined to have a particular inverse document frequency. Alternatively or additionally, a condition for biasing automatic speech recognition may be deemed satisfied if the shared term appears in a similar section (e.g., title, first sentence, abstract section, etc.) of the respective sources.

[0050] If the content of the data meets one or more conditions for biasing the automatic speech recognition, the method 300 may proceed from operation 306 to operation 308, which may include biasing the automatic speech recognition based on the content of the instance(s) of the data. Alternatively, if the content of the data does not meet the conditions for biasing the automatic speech recognition, the method 300 may proceed from operation 306 to operation 310. The biasing the automatic speech recognition operation 308 may be performed according to one or more different processes. For example, in some implementations, the automatic speech recognition may be performed by assigning probabilities to various hypotheses for parts of speech (e.g., words, phonemes, and / or other hypothetical parts of speech). The probabilities may then be adjusted according to whether any of the parts of speech correspond to any of the contents of the data related to the meeting. For example, the probability assigned to a phoneme of a spoken term such as "Quadratic" may be increased when the term appears in an instance of the data related to the meeting. Alternatively, or additionally, phonemes of a spoken term such as "insurance" may be assigned a higher probability than phonemes of the term "assurance" if one or more participants write the term "insurance" into a meeting notes document during the meeting. In this manner, automatic speech recognition biasing may be performed in real-time during the meeting as additional content related to the meeting is created and / or discovered.

[0051] Method 300 may proceed from operation 308 to operation 310 as shown in FIG. 4 and, optionally, via continuation element "B", to operation 402 of method 400. Operation 310 may include determining whether participants and / or invitees for the conference are gathering for the conference. If it is determined that the conference participants and / or invitees are gathering for the conference (e.g., based on schedule data, geolocation data, conference application data, video data, etc.), method 300 may proceed from operation 310 to operation 312. Otherwise, if the participants and / or invitees have not yet gathered for the conference, method 300 may proceed from operation 310 to operation 302. Operation 312 may include determining whether any conference participants (or others associated with the conference) are accessing any instances of data during the conference. For example, the data may include memo documents being accessed by the participants, portions of a transcript of the conference, one or more different types of media files (e.g., images, videos, etc.), and / or any other data to which a person may have access. If it is determined that at least one participant is accessing an instance of the data during the conference, method 300 may return to operation 306 for further biasing the automatic speech recognition according to the content of the data being accessed. Otherwise, method 300 may proceed from operation 312 to operation 302 for determining whether the conference is still in progress and / or whether another conference is expected to occur.

[0052] FIG. 4 illustrates a method 400 for automatically incorporating specific content into a meeting document to facilitate generating a meeting summary and / or other type of document based on the content of the meeting. Method 400 may be performed by one or more applications, devices, and / or any other apparatus or module that may interact with participants of a meeting. Method 400 may include operation 402, which may optionally be a continuation of method 300, as indicated by continuation element "B" shown in FIG. 3 and FIG. 4. Operation 402 may include determining whether natural language content is provided by a participant (or other person associated with the meeting) during the meeting. The natural language content may be, for example, spoken words from participants of the meeting (e.g., a lunch meeting, a college class, a family meal, and / or any other gathering) regarding a particular topic of the meeting. The spoken words may be, for example, "That's a good idea. We should each think about how we can implement it in our own projects."

[0053] If natural language content is provided by a participant of the conference, method 400 may proceed from operation 402 to operation 404. Otherwise, method 400 may optionally proceed from operation 402 to operation 302 of method 300 via continuation element "A" as shown in FIG. 3 and FIG. 4. Operation 404 may include determining a degree of relevance of the natural language content to the conference. In some embodiments, if the natural language content is a written input to an application, the degree of relevance may be based on whether one or more participants provided text entries and / or voice inputs similar to the written input. Alternatively, or additionally, if the natural language content is an audio input captured by one or more audio interfaces present during the conference (e.g., a video conference in which participants are each using their own laptop in a home office), the degree of relevance may be based on whether one or more other participants provided similar voice inputs and / or written inputs. For example, spoken input that is reflected in a written notes application by another participant (e.g., "See if you can implement Keith's idea into my project.") may be assigned a higher degree of relevance than if the other participant does not reflect the spoken input in their written notes (e.g., the meeting participant's notes do not include a mention of "Keith's idea").

[0054] Alternatively or additionally, the degree of relevance assigned to natural language content provided by a participant may be based on whether the natural language content is associated with any conference documents and / or other instances of data related to the conference. For example, natural language content that embodies terms contained in the title and / or other portions of an invitation to a conference may be assigned a higher degree of relevance than other natural language content that does not specifically have any other terms related to the conference. Alternatively or additionally, natural language content that embodies terms contained in other data related to the conference (e.g., messages between invitees, media accessed by the invitees, places visited by the invitees, and / or any other relevant data with prior permission from the participant(s)) may be assigned a higher degree of relevance than other natural language content that does not embody such terms.

[0055] Method 400 may proceed from operation 404 to operation 406, which may include determining whether a degree of relevance meets a threshold for incorporating text entries characterizing the natural language content into a conference document (e.g., an automatically generated conference summary document). If the degree of relevance assigned to the natural language content meets the threshold, method 400 may proceed to operation 408. If not, method 400 may return to operation 402. In some embodiments, the threshold for incorporating text entries may be based on one or more inputs from one or more participants. Alternatively, or in addition, the threshold may be based on the number of people participating in the conference, the frequency of inputs from users during the conference, the volume of content provided during the conference (e.g., number of words, phrases, pages, etc.), the location of the conference, the format of the conference (e.g., video, in-person, audio only, etc.), and / or any other characteristic of the conference.

[0056] Act 408 may include incorporating and / or modifying the text entry in the meeting document. In some cases, the text entry may be incorporated into the meeting document to summarize spoken input and / or gestures from one or more participants for later reference. In this manner, participants may avoid manually providing typing input into the meeting document during and / or after the meeting to summarize portions of the meeting. This may save resources on each computing device that may normally be utilized to process such input. In some implementations, method 408 may optionally include act 410 of determining whether the text entry corresponds to a meeting action item. For example, a meeting action item may be a task created by one or more participants during a meeting that requires one or more people to take action (e.g., gather certain information before a follow-up meeting). This determination may be based on manual input from participants and / or others to explicitly designate the text entry as an action item. Alternatively, or additionally, the determination may be based on terms contained in the text entry, the tone of the detected text entry (e.g., an inquisitive tone), the context in which the text entry was entered into the meeting document (e.g., the moment in the meeting where a particular participant is stating what needs to be done before the next meeting). Once it is determined that the text entry corresponds to an action item, method 400 may proceed from operation 410, optionally via continuation element "C", to operation 502 of method 500, as shown in Figures 4 and 5.

[0057] If it is determined that the text entry does not correspond to an action item and / or operation 410 is optionally bypassed, method 400 may proceed to operation 412. Operation 412 may include determining whether other meeting content indicates a change in relevance to the text entry. For example, if additional natural language content from another participant and / or other contextual data indicates that the text entry is less relevant, the text entry may be deemed less relevant. For example, emails received by one or more participants during and / or after the meeting may be processed, with prior permission from one or more participants, to determine whether a particular text entry is more or less relevant. If other meeting content indicates that there is a change in the relevance of the text entry, method 400 may return to operation 406 to determine the degree of relevance of the text entry and / or the natural language content that formed the basis of the text entry. Otherwise, if other meeting content does not indicate a change in the relevance of the text entry, method 400 may proceed from operation 412 to operation 402, and optionally proceed to operation 302 via continuation element "A" if no additional natural language content has been provided by any of the participants (e.g., when the meeting has ended).

[0058] FIG. 5 illustrates a method 500 for generating an action item based on natural language content provided during a participant's conference, reminding a particular participant of an action item, and / or designating an action item as completed based on a particular condition. The method 500 may be performed by one or more applications, devices, and / or any other devices or modules that may interact with participants in a conference. The method 500 may include an operation 502 of generating data characterizing an action item for one or more conference participants and / or other persons. In some embodiments, the generated data may be based on natural language content and / or other data provided by one or more participants in the conference, one or more applications associated with the conference, and / or one or more other persons and / or devices associated with the conference. For example, a participant in a video conference may provide a spoken phrase such as "Yes, let's follow up next month" in response to another participant providing another spoken phrase such as "Can we talk about maintenance fees soon?". The audio corresponding to each spoken phrase may be processed to generate a text entry, which may be further processed to generate data on which the action item is based. For example, one or more trained machine learning models may be utilized to process the text entries and generate summary entries from the text entries. The summary entries may be designated as "action items" that may then be incorporated into meeting documents being generated and / or generated by the conferencing application and / or other applications (e.g., assistant applications).

[0059] Method 500 may proceed from operation 502 to operation 504, which may include determining whether data associated with the meeting indicates that the action item should have a particular condition. The condition may be utilized to render one or more reminders to one or more participants to fulfill the action item and / or to determine whether the action item has been fulfilled (i.e., completed). For example, spoken words such as "Let's follow up on that next month" may provide an indication that the action item should have one or more particular conditions. Alternatively, or additionally, spoken words during the meeting such as "Send me that attachment and I'll get to work on this" may provide an indication that upon receipt of the "attachment," a reminder should be triggered for the participant to work on the action item. In other words, a conditional statement occurring within a context that also identifies a particular action item may indicate that the action item should be stored in association with a conditional reminder and / or fulfillment condition.

[0060] If the data indicates that the action item should have a particular condition, the method 500 may proceed from operation 504 to operation 506. Otherwise, if the data does not indicate that the action item should have a particular condition, the method 500 may proceed from operation 504 to operation 510, which incorporates the action item into the meeting document. Operation 506 may include processing data related to the meeting to facilitate identifying the action item condition. For example, one or more spoken words from one or more participants may provide the basis for establishing the condition for the particular action item. Alternatively, or in addition, contextual data related to the meeting may provide the basis for establishing the condition for the particular action item. For example, calendar data correlating a series of meetings and / or reminders for the series of meetings may serve as the basis for the "due date" of the action item and / or the time to remind participants about the action item (e.g., 24 hours before the next meeting of the related series of meetings). Alternatively, or in addition, the gathering of participants after the meeting and / or communication between participants may trigger a reminder for the action item generated based on the meeting. For example, a first participant sending an email to a second participant may trigger a reminder to the first participant to complete an action item that may have been generated during a previous meeting that the second participant was present at.

[0061] From operation 506, method 500 may proceed to operation 508, which generates action item data that conditionally characterizes the action item. The action item data may then be stored in association with one or more participants who may be tasked with completing the action item and / or may otherwise be associated with the action item. For example, the conference application may communicate the action item data to another application (e.g., an automated assistant application if the conference application is separate from the automated assistant), which may utilize the action item data to generate reminders for the participants (with prior permission from the participants) and / or to determine whether the action item has been completed. From operation 508, method 500 may proceed to operation 510, which incorporates the action item into a conference document. In this manner, the conference document may provide a summary of relevant topics discussed during the conference and / or a comprehensive list of action items created during the conference. Each action item may optionally serve as an embedded link to other data that may be useful in completing the respective action item.

[0062] From operation 510, method 500 may optionally proceed to optional operation 512, which determines whether one or more conditions and / or action items have been met. If it is determined that one or more conditions have been met, method 500 may proceed to operation 514, for indicating that the action item has been fulfilled and / or for rendering a reminder of the action item to one or more associated participants and / or others. For example, in response to meeting a condition that the two participants subsequently meet in person and / or via a conference call, the conference application may cause a reminder of the action item to be rendered at a device associated with each of the two participants. Method 500 may then optionally proceed from operation 514 via continuation element "A" to operation 302 of method 300, as shown in FIG. 3.

[0063] 6 is a block diagram 600 of an exemplary computer system 610. The computer system 610 typically includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, a memory 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices enable user interaction with the computer system 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0064] The user interface input devices 622 may include pointing devices such as a keyboard, a mouse, a trackball, a touchpad, a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 610 or a communications network.

[0065] The user interface output devices 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from computer system 610 to a user or to another machine or computer system.

[0066] Storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 624 may include logic for performing selected aspects of method 300, method 400, method 500, and / or for implementing one or more of system 200, computing device 104, computing device 112, computing device 152, server device 142, and / or any other applications, devices, apparatus, and / or modules discussed herein.

[0067] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used by the storage subsystem 624 may include a main random access memory (RAM) 630 for storing instructions and data during program execution, and a number of memories, such as a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage of program files and data files, and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of an embodiment may be stored by the file storage subsystem 626 in the storage subsystem 624 or on other machines accessible by the processor(s) 614.

[0068] Bus subsystem 612 provides a mechanism that allows the various components and subsystems of computer system 610 to communicate with each other as intended. Although bus subsystem 612 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0069] Computer system 610 can be of many different types, such as a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 610 shown in Figure 6 is intended only as a specific example to illustrate some implementations. Many other configurations of computer system 610 can have more or fewer components than the computer system shown in Figure 6.

[0070] In situations where the systems described herein collect or may utilize personal information about users (or "participants" as often referred to herein), the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how it receives content from a content server that may be more relevant to the user. Also, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that personally identifiable information about the user cannot be determined, or if geographic location information is obtained (such as to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific geographic location cannot be determined. Thus, the user may control how information about the user is collected and / or used.

[0071] Although several embodiments have been described and illustrated herein, various other means and / or structures can be utilized to perform the functions and / or obtain the results and / or one or more advantages described herein, and each such variation and / or modification is deemed to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be illustrative, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application(s) in which the teachings are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation many equivalents to the specific embodiments described herein. Thus, it is to be understood that the foregoing embodiments have been presented for illustrative purposes only, and that, within the scope of the appended claims and equivalents thereof, embodiments may be practiced otherwise than as specifically described and claimed. Embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is within the scope of the disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0072] In some embodiments, a method implemented by one or more processors is described as including operations such as determining, by an application, that a conference of a plurality of different participants is occurring or scheduled to occur. The conference provides an opportunity for one or more participants of the plurality of different participants to communicate information to other participants of the plurality of different participants. The method may further include determining, by the application, that one or more instances of data are associated with the conference based at least on one or more instances of data including content determined to be associated with at least one participant of the plurality of different participants. The method may further include biasing automatic speech recognition performed on audio data during the conference of the plurality of different participants according to content of the one or more instances of data. The audio data embodies speech from one or more participants communicating information to other participants. The method may further include generating, by the application, an entry for a conference document based on speech recognition results from the automatic speech recognition biased according to content of the one or more instances of data. The entry characterizes at least a portion of the information communicated from the one or more participants to the other participants.

[0073] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0074] In some embodiments, determining that one or more instances of the data are associated with the conference includes determining that one or more instances of the data include a document accessed and / or edited by at least one participant of the plurality of different participants prior to the conference. In some embodiments, determining that one or more instances of the data are associated with the conference additionally or alternatively includes determining that one or more instances of the data include a document accessed and / or edited by at least one participant of the plurality of different participants within a threshold duration prior to the conference. In some embodiments, determining that one or more instances of the data are associated with the conference additionally or alternatively includes determining that one or more instances of the data include a document that is accessed and / or edited by at least one participant of the plurality of different participants during the conference. In some embodiments, determining that one or more instances of the data are associated with the conference additionally or alternatively includes determining that one or more instances of the data include a document that embodies one or more terms identified in a conference invitation for the conference and is accessible to at least one participant of the plurality of different participants.

[0075] In some embodiments, determining that the one or more instances of the data are related to the conference includes determining that the one or more instances of the data include a document that embodies one or more terms identified in a title of a conference invitation for the conference.

[0076] In some embodiments, biasing the automatic speech recognition according to content of the one or more instances of data includes generating one or more candidate terms for inclusion in the entry for the conference document based on a portion of the audio data, and assigning a weight value to each term of the one or more candidate terms, each weight value being based at least in part on whether a particular term of the one or more candidate terms is included in content of the one or more instances of data.

[0077] In some embodiments, determining that the one or more instances of data are associated with the conference includes determining that one or more documents containing the content were accessed by at least one of the plurality of different participants within a threshold duration prior to the conference. In some of those embodiments, the threshold duration is based on when a conference invitation was received and / or accessed by at least one participant who accessed the document. In some embodiments, determining that the one or more instances of data are associated with the conference additionally or alternatively includes selecting one or more terms from the one or more documents as content that provides a basis for biasing the automatic speech recognition. The one or more terms may be selected based on an inverse document frequency of the one or more terms appearing in the one or more documents.

[0078] In some embodiments, a method implemented by one or more processors is described as including operations such as causing an application of a computing device to process audio data corresponding to spoken natural language content to facilitate generating a text entry for a conference document. The spoken natural language content is provided by a conference participant to one or more other participants of the conference. The method may further include determining a degree of relevance of the text entry to one or more instances of data related to the conference based on the text entry. The one or more instances of data include documents accessed by at least one participant of the conference prior to and / or during the conference. The method may further include determining whether to incorporate the text entry into the conference document based on the degree of relevance. The method may further include, when the application determines to incorporate the text entry into the conference document, causing the application to incorporate the text entry into the conference document, the conference document being rendered on a display interface of a computing device or additional computing devices accessed by one or more other participants of the conference during the conference.

[0079] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0080] In some embodiments, the method may further include, when the application determines to incorporate the text entry into the conference document, determining that during the conference, a particular participant of the conference selects to generate an action item based on the text entry in the conference document via a computing device, or an interface of another computing device. The action item is generated to provide a conditional reminder to at least one participant of the conference. In some embodiments, the conditional reminder is rendered for at least one participant of the conference if one or more conditions are met. The one or more conditions may be determined to be met using at least contextual data accessible to the application. For example, the contextual data may include a location of at least one participant of the conference, and the one or more conditions may be met when at least one participant of the conference is within a threshold distance of a particular location.

[0081] In some embodiments, determining a degree of relevance of the text entry to one or more instances of data related to the conference includes determining that a first participant of the conference provided a text input to a first document during the conference and that a second participant of the conference provided an additional text input to a second document during the conference. In some of those embodiments, the degree of relevance is based on whether the text input and the additional text input are correlated with a text entry generated from a spoken natural language input. In some embodiments, determining a degree of relevance of the text entry to one or more instances of data related to the conference additionally or alternatively includes determining that a first participant of the conference provided a spoken input during the conference and that a second participant of the conference provided an additional spoken input within a threshold duration after the participant provided the spoken natural language content. In some of those embodiments, the degree of relevance is based on whether the spoken input and the additional spoken input are correlated with the text entry. In some embodiments, determining a degree of relevance of the text entry to one or more instances of data related to the conference additionally or alternatively includes determining that at least one participant of the conference made a non-verbal gesture when the participant of the conference provided spoken natural language content, in some of these embodiments, the degree of relevance is based on an interpretation of the non-verbal gesture by the application or another application.

[0082] In some embodiments, a method implemented by one or more processors is described as including an operation such as determining, by an application, that a participant of the conference has provided natural language content to an accessible computing device during the conference. The application is accessible via the computing device, and the conference includes one or more other participants. The method may further include causing input data to be processed to facilitate generating a text entry for the conference document in response to determining that the participant has provided the natural language content. The input data is captured by an interface of the computing device and characterizes the natural language content provided by the participant. The method may further include determining, based on processing the input data, whether to incorporate the text entry into the conference document as an action item to be completed by at least one participant of the conference. Determining whether to incorporate the text entry as an action item is based at least in part on whether the natural language content embodies a request for at least one participant and / or application. The method may further include causing the application to incorporate the action item into the conference document if the application determines to incorporate the text entry into the conference document as an action item. The conference documents are accessible during the conference via a display interface of the computing device being accessed by one or more other participants in the conference or on another computing device.

[0083] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0084] In some embodiments, the method may further include, if the application determines not to incorporate the text entry into the meeting document action item, causing the text entry to be incorporated into the meeting document as a transcription of natural language content provided by the meeting participant. In some embodiments, the method may further include, if the application determines to incorporate the text entry into the meeting document as an action item, causing the application to render a conditional reminder for at least one participant when one or more conditions are met, the one or more conditions being determined to be met using at least context data accessible to the application.

Claims

1. 1. A method implemented by one or more processors, comprising: Determining, at a computing device and by an application, that a conference of a plurality of different participants is occurring or scheduled to occur; determining that the conference provides an opportunity for one or more participants of the plurality of distinct participants to communicate information to other participants of the plurality of distinct participants; determining, by the application, that one or more instances of data are associated with the conference based at least on the one or more instances of data including content determined to be associated with at least one participant of the plurality of different participants; biasing automatic speech recognition performed on audio data during the conference of the plurality of different participants according to the content of one or more instances of the data; biasing the audio data to embody speech from the one or more participants communicating the information to the other participants; and generating, by the application, a meeting document entry based on speech recognition results from the automatic speech recognition biased according to the content of the one or more instances of the data; generating an entry characterizing at least a portion of the information to be communicated from the one or more participants to the other participants; A method comprising:

2. Determining that one or more instances of the data are associated with the conference includes:

2. The method of claim 1, comprising determining that one or more instances of the data include a document that was accessed and / or edited by at least one of the plurality of different participants prior to the conference.

3. Determining that one or more instances of the data are associated with the conference includes:

2. The method of claim 1, comprising determining that one or more instances of the data include a document that was accessed and / or edited by at least one participant of the plurality of different participants within a threshold duration prior to the conference.

4. Determining that one or more instances of the data are associated with the conference includes:

10. A method according to any preceding claim, comprising determining that one or more instances of the data include a document that is being accessed and / or edited by at least one of the plurality of different participants during the conference.

5. Determining that one or more instances of the data are associated with the conference includes:

20. The method of any preceding claim, comprising determining that the one or more instances of the data include a document that embodies one or more terms identified in a conference invitation for the conference and is accessible to at least one participant of the plurality of different participants.

6. Determining that one or more instances of the data are associated with the conference includes:

5. A method according to any preceding claim, comprising determining that one or more instances of the data include documents that embody one or more terms identified in a title of a conference invitation for the conference.

7. Biasing the automatic speech recognition according to the content of one or more instances of the data includes: generating one or more candidate terms for inclusion in the entry of the meeting document based on a portion of the audio data; assigning a weight value to each term of the one or more candidate terms; assigning each weight value based at least in part on whether a particular term of the one or more candidate terms is included in the content of one or more instances of the data; 13. A method according to any preceding claim, comprising:

8. Determining that one or more instances of the data are associated with the conference includes: The method of claim 1 , further comprising determining that one or more documents containing the content were accessed by at least one participant of the plurality of different participants within a threshold duration prior to the conference.

9. The method of claim 8 , wherein the threshold duration is based on when a meeting invitation is received and / or when the document is accessed by at least one participant who accessed the document.

10. Determining that one or more instances of the data are associated with the conference includes: selecting one or more terms from the one or more documents as content that provides a basis for biasing the automatic speech recognition; selecting, wherein the one or more terms are selected based on an inverse document frequency of the one or more terms appearing in the one or more documents; The method of claim 8 , comprising:

11. 1. A method implemented by one or more processors, comprising: processing audio data corresponding to spoken natural language content to facilitate generating a text entry of a meeting document by an application on a computing device; the spoken natural language content being provided by a participant in the conference to one or more other participants in the conference for processing; determining, based on the text entry, a degree of relevance of the text entry to one or more instances of data related to the conference; determining that the one or more instances of data include documents accessed by at least one participant of the conference prior to and / or during the conference; determining whether to incorporate the text entry into the meeting document based on the degree of relevance; If the application determines to incorporate the text entry into the meeting document, causing the application to incorporate the text entry into the meeting document; the conference document is rendered on a display interface of the computing device or an additional computing device accessed by the one or more other participants of the conference during the conference; A method comprising:

12. If the application determines to incorporate the text entry into the meeting document, determining, during the conference, that a particular participant of the conference has selected, via an interface of the computing device or the other computing device, to generate an action item based on the text entry of the conference document; determining that the action item is generated to provide a conditional reminder to at least one participant of the conference; The method of claim 11 further comprising:

13. the conditional reminder is rendered for the at least one participant of the conference if one or more conditions are met; The method of claim 12 , wherein the one or more conditions are determined to be satisfied using at least contextual data accessible to the application.

14. the contextual data includes a location of the at least one participant of the conference; The method of claim 13 , wherein the one or more conditions are met when the at least one participant of the conference is within a threshold distance of a particular location.

15. Determining the degree of relevance of the text entry to one or more instances of the data related to the conference includes: determining that a first participant of the conference has provided text input to a first document during the conference and that a second participant of the conference has provided additional text input to a second document during the conference; determining whether the degree of relevance is based on whether the text input and the additional text input correlate with the text entry generated from the spoken natural language input; The method of any one of claims 11 to 14, further comprising:

16. Determining the degree of relevance of the text entry to one or more instances of the data related to the conference includes: determining that a first participant in the conference has provided spoken input during the conference and that a second participant in the conference has provided additional spoken input within a threshold duration after the participant provided the spoken natural language content; determining whether the degree of relevance is based on whether the spoken input and the additional spoken input are correlated to the text entry; The method according to any one of claims 11 to 14, comprising:

17. Determining the degree of relevance of the text entry to one or more instances of the data related to the conference includes: determining that the at least one participant in the conference has made a non-verbal gesture when the participant in the conference provides the spoken natural language content; determining, the degree of relevance being based on an interpretation of the non-verbal gesture by the application or another application; The method according to any one of claims 11 to 16, comprising:

18. 1. A method implemented by one or more processors, comprising: determining, by an application, that a participant of a conference has provided natural language content to an accessible computing device during said conference; determining that the application is accessible via the computing device and that the conference includes one or more other participants; responsive to determining that the participant has provided the natural language content, processing input data to facilitate generating a text entry for a conference document; the input data is captured by an interface of the computing device and characterised and processed to characterise and process the natural language content provided by the participant; determining whether to incorporate the text entry into the meeting document as an action item to be completed by at least one participant of the meeting based on processing the input data; determining whether to incorporate the text entry as the action item is based at least in part on whether the natural language content embodies a request for the at least one participant and / or the application; If the application determines to incorporate the text entry as an action item into the meeting document, causing the application to incorporate the action item into the meeting document; the conference document is accessible during the conference via a display interface of the computing device or another computing device being accessed by the one or more other participants of the conference; A method comprising:

19. If the application determines not to incorporate the text entry into the meeting document, the action item, incorporating the text entry into the meeting document as a transcription of the natural language content provided by the participants of the meeting; 20. The method of claim 18, further comprising:

20. If the application determines to incorporate the text entry as an action item into the meeting document, causing the application to render a conditional reminder to the at least one participant when one or more conditions are met; the one or more conditions are determined to be satisfied using at least context data accessible to the application; 20. The method of claim 18, further comprising:

21. one or more processors; A memory storing instructions that, when executed, cause the one or more processors to perform the operations of any one of claims 1 to 20; A system comprising:

22. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform the operations recited in any one of claims 1-20.

Images