Generate rich action items

Abstract

A computerized system is provided for automatically determining and providing a rich action item for an event such as a meeting. By automatically modifying or supplementing the action item based on contextual information (such as a related meeting discussion), the action item can be first determined, and then enhanced or clarified to be easier to understand. By determining the boundaries of the (one or more) occurrences of the action item, the action item can be further enhanced by clarifying so that irrelevant content that is irrelevant to the action item or does not contribute to the user's understanding is diluted or excluded from presentation to the user. Some embodiments include an intelligent graphical user interface with a function for receiving user feedback. The feedback can be used to reconfigure a logical unit, such as a machine learning model, for determining a rich action item to improve accuracy.

Description

Background Art

[0001] Computer-implemented technology can assist users in organizing and completing tasks, and particularly performs action items produced by events such as meetings. For example, some technologies (such as task lists or to-do list applications) are configured to receive manual input from users, such as minutes, the subject of a meeting or the date of a meeting, and store the input in a computer memory so that users can return to the information in reference to a static graphic display format. However, except for the need for manual input from the user, the stored information and user interface functions of these technologies are also limited. Usually, such solutions do not provide functions for intelligently presenting action items to users, etc. In addition, the advancement of computing technology and the increasing popularity of modern computing devices have led to a sharp increase in available data about meetings or events. But conventional solutions have not kept up with the pace of these advancements, and have also failed to utilize available data or provide robust functionality to address these new challenges. Summary of the invention

[0002] This summary of the invention is provided to introduce a selection of concepts in a simplified form, and these concepts will be further described in the specific embodiments below. The present disclosure is not intended to identify the key features or basic features of the claimed subject matter, nor is it intended to be used to assist in determining the scope of the claimed subject matter. Various aspects of the present disclosure relate to a computerized system for automatically determining and presenting rich action items for events such as meetings. Rich action items can include instructions for action items and related meeting content or contextual data useful for understanding action items, and they can be personalized for specific users such as meeting attendees. In particular, action items can be determined and then enhanced or clarified to be easier to understand. For example, some embodiments include automatically modifying or supplementing action items based on contextual information, such as meeting discussions related to action items, which can occur throughout the meeting or in previous meetings, and / or external data, which can, for example, be located on the user device of the event attendee or stored on a company server in the cloud.

[0003] In addition, action items can be automatically clarified by determining the boundaries at which the action items appear (once or multiple times) in the conference conversation, thereby de-emphasizing or excluding presentation to the user of irrelevant content that is irrelevant to the action item or that is not helpful for the user's understanding. For example, action items are often embedded or surrounded by non-action item words and character sequences in other irrelevant information or transcripts of events. Therefore, some aspects of the present disclosure automatically detect and highlight, extract, or otherwise mark event content to indicate specific words or character sequences that are part of an action item.

[0004] In some embodiments, an intelligent graphical user interface may be provided that includes functionality to improve the user experience, such as a feature that allows the user to provide feedback indicating whether a presented item is actually an action item. Based on this feedback, the logic or machine learning model used to determine rich action items can be retrained or updated so that future rich action items determined and presented to the user are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various aspects of the present disclosure are described in detail below with reference to the accompanying drawings, in which:

[0006] Figure 1 is a block diagram of an exemplary computing environment suitable for implementing embodiments of the present disclosure;

[0007] Figure 2 is a block diagram illustrating an exemplary system architecture for generating and presenting rich action items according to an embodiment of the present disclosure;

[0008] Figure 3 is a block diagram illustrating an exemplary system of a model or layers of a single model that can be used to generate action items and related rich content, and illustrating how various aspects of the model or layers can be tuned based on received user feedback, according to an embodiment;

[0009] Figure 4 is a schematic diagram of a presented action item embedded in an event context according to an embodiment;

[0010] Figure 5A illustrates exemplary screenshots of content indicating a conference event that may be used to determine an action item according to an embodiment;

[0011] Figure 5B The diagram shows a method based on the Figure 5A An exemplary screenshot of a meeting content indicating a list of action items or action item candidates;

[0012] Figure 5C illustrates exemplary screen shots indicating various other user interface features related to presenting action items to a user, according to an embodiment;

[0013] Figure 6 are exemplary screenshots of user interfaces according to an embodiment;

[0014] Figure 7 are exemplary screenshots of user interfaces according to an embodiment;

[0015] Fig. 8A illustrates exemplary action item candidates and associated user interface elements according to an embodiment;

[0016] Figure 8B It is a diagram according to an embodiment. Fig. 8A How user choices can be used to train or tune corresponding machine learning models;

[0017] Fig. 9 Describes an exemplary process for generating and providing rich action items according to an embodiment;

[0018] Fig.10 Depicted are exemplary processes for generating and presenting action items via a user interface according to an embodiment; and

[0019] Fig.11 is a block diagram of an exemplary computing device for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] The subject matter of various aspects of the present disclosure is specifically described herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. On the contrary, the inventor has contemplated that, in combination with other existing or future technologies, the claimed subject matter may also be embodied in other ways to include different steps or step combinations similar to those described in this document. In addition, although the terms "step" and / or "block" may be used herein to represent different elements of the method adopted, these terms should not be interpreted as implying any specific order among or between the various steps disclosed herein, unless and except when the order of the individual steps is clearly described. Each method described herein may include a computing process that can be performed using any combination of hardware, firmware and / or software. For example, various functions may be performed by a processor running instructions stored in a memory. The method may also be embodied as a computer-usable instruction stored on a computer storage medium. The method may be provided by a standalone application, a service or a managed service (standalone or in combination with another managed service) or a plug-in of another product, etc.

[0021] The specific aspects of the present disclosure automatically and intelligently determine and present the rich action items of events (e.g., meetings, interactive seminars, informal gatherings, etc.). "Action items" as described in this article can include tasks indicated in events, which are requested to be completed to promote the specific goals or purposes associated with the events. In various cases, action items can be issued via people's commands or other requests to make another person (or themselves) perform a certain action. In the illustrative example of action items, during a meeting about the development of a specific computer application, a person may say: "Bob, can you debug the application today?" This is an action item that Bob performs debugging operations today to prepare the application for deployment. "Rich" action items can include instructions for action items and related meeting content or context information useful for understanding the action items, and it can be personalized for specific users such as meeting attendees. In particular, action items can be determined, and then enhanced or clarified, so as to be easier to understand. For example, an action item may be "you do this next week, okay?". The action item may be enriched by clarifying that "you" refers to John, a participant in the meeting, and "this" refers to a specific task of "Project X Debug Project" based on past conversations stored on the user's device or server.

[0022] Manual note-taking or manual computer user input and CPU processing for meetings or other events are time-consuming, and in many cases, important action items are missing, misidentified, or unclear in some way. Action items may be missing for various reasons, such as lack of concentration, confusion about what an action item is, and poor communication. There are many ways for people to talk and interact with others, which may cause action items to be unrecognized or unclear. Action items from events are often incomplete because they lack important contextual data required to understand the action items. For example, action item candidates may lack information, such as information about the person responsible for completing the action item, the deadline for the action item, the spokesperson for the action item, or information about the specific tasks associated with the action item. Event content (such as conversations at meetings) is often fragmented by various missing parts, which makes it unclear what the action item really refers to. For example, a phrase spoken in a meeting may not include context, such as the date when the action item is due, who is speaking, who will complete the action item, and what the action item is clearly. In addition, action items can be embedded in various other texts, making it difficult to identify what the actual action item is. For example, a sentence may be "I will send the file tomorrow and you can handle it as you like." It is difficult for users and conventional techniques to identify and present portions of this or other sentences or character sequences as action items.

[0023] Various aspects of the present disclosure remedy these deficiencies by automatically modifying or supplementing action items based on contextual data and / or external data, such as meeting discussions related to the action item, which may occur during the entire meeting or in previous meetings. This contextual data may be useful for understanding the action item and may be personalized for specific users such as meeting attendees. In particular, the action item may be automatically determined and enhanced or clarified to be easier to understand. Various aspects of the present disclosure may determine the clarity of the action item candidates in order to display the action item.

[0024] In some embodiments, "context data" can represent the event context that is spoken or otherwise input (e.g., typed, via email / communication, etc.) before and / or after the action item candidate is spoken / input. "Event context" can include or contain one or more parts (e.g., sentences or time sequences) of the audio or written transcript of all (or some of) conversations or discourses (e.g., conversations, questions, laughter, action items) within the event representing the content of the event. Therefore, for example, for the action item "I will meet it", the sentence before the phrase in the transcript may be "John, can you meet the deadline for model design?" The context data of the event context provides a strong inference: that is, John is the spokesperson and person in charge of the "I will meet it" action item, and "it" refers to the deadline of the action item. Using this same description, the sentence directly after the action item candidate in the transcript may be "OK, I look forward to hearing it next Friday." This context may indicate that "next Friday" is the deadline for the model design action item candidate. Subsequently, in some embodiments, a user interface or other graphical display format may indicate the action item candidates and / or labels indicating who is speaking / receiving the action item candidate, when the due date is, and / or what the action item is. For example, using the above description, the final displayed action item may be "[John] will [complete the model design by next Friday]". In this way, the final presented action item can add, remove, replace, or otherwise alter the initially presented action item candidates and other character sequences.

[0025] Additionally or alternatively, context data can refer to any remote or "external context" obtained that is not a direct part of the transcript or content of the actual event (event context). For example, the external context can be data retrieved from a user device (e.g., via a calendar tool plug-in, text, social media, or other application), a sensor (e.g., a biometric sensor, a geolocation sensor), one or more servers, and / or other remote data stores (such as a database). Such external context can be used as one or more signals to determine information such as the due date of an action item, who is speaking, who will complete the action item, and / or what the action item clearly is. In the example, using the example above, for an event transcript including the action item "I will meet it", the event context may not indicate who said it, but a text message from a conference participant's phone may show: "John, we will meet in 1 week to discuss the model design you are responsible for, can you attend?" This external context would be a strong signal that the owner and speaker of the action item candidate is John.

[0026] By way of example, an initial action item candidate determined by an embodiment of the present disclosure may be "I will satisfy it." Using natural language processing and / or machine learning components as described herein, it may be determined that the sentence is unclear (e.g., it is below a threshold clarity score). For example, this may occur by determining that no names are present within the sentence but only pronouns and / or no date is identified. Thus, it may not be initially clear who is speaking or otherwise entering the sentence, what "it" refers to, when "it" will be completed, and by whom the action item will be completed. In various embodiments, in response to determining that the candidate is below a clarity score, the missing data can be determined based on the contextual data, as described above.

[0027] Various embodiments of the present disclosure automatically modify or supplement an action item based on contextual data by determining and presenting the amount of contextual data required for the clarity of the action item. This determination can be based on generating a clarity score. "Clarity score" is a score indicating the clarity of a specific action item based on one or more factors, such as "who" stated the office action, "when" the action item is due, etc. For example, if the candidate's action item clarity score is lower than a threshold, more context is presented relative to the candidate action item with a score higher than the threshold. In fact, in some embodiments, the clarity score is inversely proportional to the amount of contextual data presented. The higher the clarity score, the less contextual data is provided to, for example, a user interface. On the contrary, in some embodiments, the lower the clarity score, the greater the amount of contextual data provided. For example, the clarity score of the action item candidate "I will do this" may be lower than a threshold. Therefore, the text of the threshold amount in the transcript before and / or after the action item candidate can be presented to the user interface in addition. In this way, the user can scroll through or otherwise view more transcript text to help identify associated information (e.g., who stated the action item candidate, when the due date is, who is responsible for completing the action item).

[0028] In various embodiments of the present disclosure, the action items are automatically clarified by determining the boundaries where the action items appear (once or multiple times) in the conference conversation, so as not to emphasize or exclude the presentation of irrelevant content that is irrelevant to the action items or does not help the user understand. As described above, various technologies and events include action items embedded in other irrelevant or non-action item texts. This may make it difficult to find what the action item is. However, some embodiments highlight or otherwise determine what the action item is, even if it is embedded in other non-action item texts. For example, using the above description, a part of the transcript may show "I will send the file tomorrow, you can handle it at will." Various embodiments of the present disclosure can present the following exemplary display text, "I will send the file tomorrow, you can handle it at will", which indicates highlighting or otherwise presenting the action item "I will send the file tomorrow" so as to make it clear what the action item is and what the action item is not. For example, in this example, the text that is not an action item is not bold text, and the text that is an action item is bold text.

[0029] In certain embodiments, an intelligent graphical user interface can be provided, which includes the function of improving user experience, such as allowing a user to provide a feature of feedback indicating whether the presented project is actually an action project. For example, if a user does not understand what an action project is or other information (for example, a speaker, an owner) associated with an action project, then there may be a logical unit receiving feedback from a user. For example, a machine learning model can use such feedback as an additional training data input to improve prediction. For example, an action project can be presented on the user interface and the user can select a button or other GUI elements indicating that the action project is not an action project. Then, it can be provided to the machine learning system as training or tuning data, so that in future sessions, this same action project will not be presented as an action project. Alternatively or additionally, a user can view the list of an action project, rather than comprising the entire event transcript of the content that is not an action project. Alternatively or additionally, a user can select to present more or less event contexts.

[0030] Prior art has various functional defects.For example, various events and conference technology only manually receive static data (for example, the name of the meeting, other notes), and can show the conference transcript with unclear action items and unclear associated data at most.For example, in the chat of many computer implementations, the web application based on the meeting, the application based on the meeting or the transcript, it is often unclear that the person who states the action item, the person who is responsible for completing the action item, the due date of the action item, etc. are not clear. These technologies can not automatically determine the action item and the action item clarity based on specific rules.On the contrary, these technologies need the manual input of the user all the time, without considering providing any intelligence or rule of the specific functions described in this article to generate abundant action items.

[0031] Existing application or user interface technology is particularly defective in its function. Conventional event solutions (such as user interfaces of chat rooms or conference applications) only display all the texts of the entire event entered by the user. This forces the tedious user task to manually scroll threads or windows to see which action items exist or pass through various layers to obtain any relevant information. This also forces the user to determine the missing information, such as who is the person who completes the action item, the date of the action item, who is the speaker, etc. Due to human error, this may cause inaccuracy, and this determination is very time-consuming. Various embodiments of the present disclosure improve these display technologies, because they can, for example, only display action items (contrary to the entire event transcript) or reduce the depth exploration of the user interface via a single user interface. In addition, the embodiment allows to present or determine other context information, such as who the speaker is, who needs to complete the action item, due date, etc. Therefore, the embodiment improves the prior art by generating abundant action items, which is not available in today's technology.

[0032] In another example of a functional deficiency, some technologies, such as virtual assistants (e.g., smart speakers), fail to clarify information associated with an action item (e.g., who is the speaker, the due date of the action item, who needs to perform the action item). Although these technologies are generally able to process natural language requests and perform keyword recognition, these technologies are generally unable to determine the action item and use contextual data to clarify the action item. Although some speech-to-text technologies are able to parse text to simply output text, they are unable to clarify action items, such as the due date of the action item and the person responsible for completing the action item. Various embodiments improve these virtual assistants because they are able to use contextual data to clarify action items.

[0033] The prior art is also inefficient in terms of computer resource consumption (e.g., CPU, memory, I / O, network latency). For example, the prior art requires that a complete transcript of an event be either transmitted to a user device over a network and presented to a user interface, or stored in a memory device so that the user can manually parse the relevant information in the data. Such a transcript may include inputs such as the names of meeting participants, the time of the meeting, the name of the meeting, and all meeting contents. This may increase storage device I / O (e.g., excessive physical read / write head movement on a non-volatile disk) or network latency, because each time a user enters this data, the request must repeatedly reach the storage device to perform a read / write operation, which is time-consuming, error-prone, and will eventually wear out components such as read / write heads. In addition, in a session-based or network-based web application, each user input may require a packet generation cost (e.g., input header information) for a network protocol (e.g., TCP / IP), which may increase network latency each time the user input is transmitted over the network. In addition, presenting a user interface, storing, or transmitting the entire transcript of an event may consume unnecessary amounts of memory.

[0034] Various embodiments of the present disclosure improve these technologies by improving the computing resource consumption of computing devices. For example, some aspects only present, transmit or store the action items of events (and may also have a small amount of metadata and / or contextual data). For example, after determining the action items, an embodiment can only transmit the action items (rather than the entire meeting transcript) to the user equipment of the user who is the "owner" or the party responsible for executing the action items through a computer network. In this way, the packet generation cost is reduced, because only the action items are transmitted through the network, which allows more bits to be transmitted through the network in some cases. In another example, a user interface can be provided, which only includes the action items or only records the action items, rather than the entire transcript of the meeting. In this way, memory storage is optimized, and I / O operations are less.

[0035] Now turn to Figure 1, a block diagram showing an exemplary operating environment 100 in which some embodiments of the present disclosure can be adopted is provided.It should be understood that this and other arrangements described in this article are only set forth as examples.Other arrangements and elements (e.g., machines, interfaces, functions, commands, and functional groupings) can be used to supplement or replace those shown, and some elements can be completely omitted for clarity.In addition, many elements in the elements described in this article are functional entities, which can be implemented as discrete or distributed components or combined with other components, and are implemented in any suitable combination and position.The various functions described in this article as being performed by one or more entities can be performed by hardware, firmware, and / or software.For example, some functions can be performed by a processor running an instruction stored in a memory.

[0036] Among other components not shown, the exemplary operating environment 100 includes a plurality of user devices, such as user devices 102a and 102b to 102n; a plurality of data sources, such as data sources 104a and 104b to 104n; a server 106; sensors 103a and 107; and a network 110. It should be understood that in Figure 1 The environment 100 shown in FIG. 1 is an example of a suitable operating environment. Figure 1 Each of the components shown in the figure can be implemented via any type of computing device, such as a combination of Fig.11 The computing device 1100 is described. These components can communicate with each other via a network 110, which may include, but is not limited to, one or more local area networks (LANs) and / or wide area networks (WANs). In an exemplary implementation, the network 110 includes the Internet and / or a cellular network, as well as any of a variety of possible public and / or private networks.

[0037] It should be understood that within the scope of the present disclosure, any number of user devices, servers, and data sources may be employed within the operating environment 100. Each may include a single device or multiple devices cooperating in a distributed environment. For example, the server 106 may be provided via multiple devices arranged in a distributed environment that together provide the functionality described herein. In addition, other components not shown may also be included in the distributed environment.

[0038] The user devices 102a and 102b to 102n can be client devices on the client side of the operating environment 100, while the server 106 can be on the server side of the operating environment 100. The server 106 can include server-side software designed to work in conjunction with the client-side software on the user devices 102a and 102b to 102n to implement any combination of features and functions discussed in this disclosure. This division of the operating environment 100 is provided to illustrate one example of a suitable environment, and it is not required for every implementation that any combination of the server 106 and the user devices 102a and 102b to 102n remain as separate entities.

[0039] The user devices 102a and 102b to 102n may include any type of computing device that a user can use. For example, in one embodiment, the user devices 102a to 102n may be related to the present invention. Fig.11 Type of computing device described. By way of example and not limitation, a user device may be embodied as a personal computer (PC), a laptop computer, a mobile or mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), a music player or MP3 player, a global positioning system (GPS) or device, a video player, a handheld communication device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a camera, a remote control, a barcode scanner, a computerized measurement device, an appliance, a consumer electronic device, a workstation, or any combination of these delineated devices, or any other suitable computing device.

[0040] Data sources 104a and 104b to 104n may include data sources and / or data systems configured to combine data Figure 2 Any of the various components of the described operating environment 100 or system 200 may be used. (For example, in one embodiment, one or more data sources 104a to 104n may be used to Figure 2 The user data collection component 210 provides (or makes available for access to) user data. The data sources 104a and 104b to 104n may be separate from the user devices 102a and 102b to 102n and the server 106, or may be incorporated and / or integrated into at least one of those components. In one embodiment, one or more of the data sources 104a to 104n include one or more sensors 103a, 107, which may be integrated into or associated with one or more of the (one or more) user devices 102a, 102b or 102n or the server 106. Figure 2 The user data collection component 210 further describes examples of sensed user data available from the data sources 104a to 104n.

[0041] Operating environment 100 can be used to implement Figure 2 One or more of the components of the action item capture system 200 described in the specification include a component for automatically extracting action items. The operating environment 100 can also be used to implement Fig. 9 (or Fig.10 ) described in the process flow 900 (or 1000). Now refer to Figure 2 , combined with Figure 1 , the block diagram provided shows the various aspects of the exemplary computing system architecture that is suitable for realizing the embodiment of the present disclosure and is generally designated as action item presentation system 200.Action item presentation system 200 only represents an example of suitable computing system architecture.Other arrangements and elements can be used to supplement or replace those shown, and some elements can be omitted completely for the sake of clarity.In addition, like operating environment 100, many elements in the element described in this article are functional entities, which can be implemented as discrete or distributed components or combined with other components, and realized with any suitable combination and position.

[0042] The exemplary system 200 includes a combination of Figure 1 The network 110 described above is a network 110 that communicatively couples components of the system 200, including a user data collection component 210, a conference monitor 250, an action item determiner 260, a rich action item generator 280, a feedback processor 270, a user data collection component 210, a presentation component 220, and a storage device 225. In some embodiments, the system 200 may include or operate with one or more consumer applications 290. These components may be embodied as a set of compiled computer instructions or functions, program modules, computer software services, or arrangements of processes executed on one or more computer systems, such as in conjunction with Fig.11 A computing device 1100 is described.

[0043] In one embodiment, the functions performed by the components of system 200 are associated with one or more personal assistant applications, services, or routines. In particular, such applications, services, or routines may be operated on one or more user devices (such as user device 102a), servers (such as server 106), may be distributed on one or more user devices and servers, or implemented in the cloud. In addition, in some embodiments, these components of system 200 may be distributed in a network, including one or more servers (such as server 106) and client devices (such as user device 102a); in the cloud; or may reside on a user device, such as user device 102a. In addition, these components, the functions performed by these components, or the services performed by these components may be implemented at appropriate abstraction layers such as an operating system layer, an application layer, a hardware layer of (one or more) computing systems. Alternatively or additionally, these components and / or the functions of the embodiments described herein can be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), and complex programmable logic devices (CPLDs). Additionally, while functionality is described herein with respect to specific components shown in exemplary system 200, it is contemplated that in some embodiments the functionality of these components can be shared or distributed across other components.

[0044] continue Figure 2 , the user data collection component 210 is generally responsible for collecting data from one or more data sources (such as Figure 1The user data may be accessed or received (and in some cases also identified) by the data sources 104a and 104b to 104n of the system 200. In some embodiments, the user data collection component 210 may be used to facilitate the accumulation of user data for a particular user (or in some cases, multiple users of crowdsourced data) of the conference monitor 250 and / or action item determiner 260. The data may be received (or accessed) by the user data collection component 210 and optionally accumulated, reformatted, and / or combined and stored in one or more data stores (e.g., storage device 225), where it may be made available to other components of the system 200. For example, the user data may be stored in or associated with the user profile 240, as described herein. In some embodiments, any personally identifying data (i.e., user data that specifically identifies a particular user) is either not uploaded or is otherwise provided with the user data from one or more data sources without being permanently stored and / or is not available to components or subcomponents of the system 200. In some embodiments, users may opt in or out of services provided by the techniques described herein and / or select which user data and / or which sources of user data are to be used by these techniques.

[0045] User data may be received from a variety of sources, where the data may be available in a variety of formats. For example, in some embodiments, user data received via the user data collection component 210 may be determined via one or more sensors, which may be on or associated with one or more user devices (such as user device 102a), servers (such as server 106), and / or other computing devices. As used herein, a sensor may include functionality, routines, components, or combinations thereof for sensing, detecting, or otherwise obtaining information such as user data from a data source 104a, and may be embodied as hardware, software, or both. By way of example and not limitation, user data may include data sensed or determined from one or more sensors (referred to herein as sensor data), such as location information of mobile device(s), attributes or characteristics of user device(s) (such as device status, charging data, date / time, or other information obtained from a user device such as a mobile device), user activity information (e.g., app usage; online activity; searches; voice data, such as automatic voice recognition; activity logs; communication data, including calls, texts, instant messages, and emails; website posts; other user data associated with communication events), and in some embodiments, user activity occurring on more than one user device, user history, session logs, application data, contact data, calendar and schedule data, notification data, social network data, news (including popular or trending items on search engines or social networks), online gaming data, e-commerce activity (including from websites such as Facebook, Facebook Live, and other social networking sites), and other user data associated with communication events. Video streaming services, gaming services, or Xbox (e.g., data from an online account such as a personal assistant application or service), user account data (which may include data from user preferences or settings associated with a personal assistant application or service), home sensor data, device data, global positioning system (GPS) data, vehicle signal data, traffic data, weather data (including forecasts), wearable device data, other user device data (which may include device settings, profiles, network related information (e.g., network name or ID, domain information, workgroup information, connection data, Wi-Fi network data or configuration data, data about the model, firmware or device, device pairing, such as a user pairing a mobile phone with a Bluetooth headset, or other network related information)), gyroscope data, accelerometer data, payment or credit card usage data (which may include information from a user's PayPal account), purchase history data (e.g., from a user's Xbox Live, Amazon.com, or eBay account), other sensor data that may be sensed or otherwise detected by (one or more) sensor (or other detector) components (including data originating from sensor components associated with the user (including location, motion, orientation, position, user access, user activity, network access, user device charging, or other data that can be provided by one or more sensor components)), data derived from other data (for example, location data that can be derived from Wi-Fi, cellular network, or IP address data), and virtually any other data source that can be sensed or determined as described herein.

[0046] User data can be received by user data collection component 210 from one or more sensors and / or computing devices associated with a user. Although it is contemplated that user data can be processed, for example, by sensors or other components not shown, so that user data collection component 210 can be interpreted, the embodiments described herein do not limit user data to processed data and can include raw data. In some embodiments, user data collection component 210 or other components of system 200 can determine explanatory data based on received user data. Explanatory data corresponds to data used by components of system 200 to interpret user data. For example, explanatory data can be used to provide context to user data, which can support determinations or inferences made by components or subcomponents of system 200, such as location information from a location, a text corpus from a user's voice (i.e., speech to text), or various aspects of spoken language understanding. In addition, it is contemplated that, for some embodiments, components or subcomponents of system 200 can use user data and / or user data combined with explanatory data to perform the goals of subcomponents described herein.

[0047] In some aspects, user data may be provided in a user data stream or signal. A "user signal" may be a feed or stream of user data from a corresponding data source. For example, a user signal may come from a smartphone, a home sensor device, a smart speaker, a GPS device (e.g., for location coordinates), a vehicle sensor device, a wearable device, a user device, a gyroscope sensor, an accelerometer sensor, a calendar service, an email account, a credit card account, or other data source. In some embodiments, the user data collection component 210 receives or accesses user-related data continuously, periodically, when it becomes available, or as needed.

[0048] continue Figure 2 , the exemplary system 200 includes a conference monitor 250. The conference monitor 250 is generally responsible for determining and / or detecting conference features from online conferences and / or face-to-face conferences, and making the conference features available to other components of the system 200. For example, such monitored activities can be the location of the meeting (e.g., determined by the geographic location of the user device), the subject of the meeting, the invitees of the meeting, whether the meeting is repeated, the relevant deadlines, projects, etc. In some aspects, the conference monitor 250 determines and provides a set of conference features (such as described below) for a specific meeting and for each user associated with the meeting. In some aspects, the meeting can be a past (or historical) meeting or a current meeting. In addition, it should be appreciated that the conference monitor 250 can be responsible for monitoring any number of meetings, for example, each online meeting associated with the system 200. Therefore, the features corresponding to the online meetings determined by the conference monitor 250 can be used to analyze multiple meetings and determine corresponding patterns.

[0049] In some embodiments, the input to conference monitor 250 is sensor data and / or user device data of one or more users at the event and / or contextual information from conference invitations and / or emails or other device activities of users at the conference. In some embodiments, this includes user data collected by user data collection component 210 (which can be accessed via user profile 240).

[0050] The meeting activity monitor 252 monitors user activity via one or more sensors (e.g., microphone, video), devices, chats, presented content, etc. In some embodiments, the meeting activity monitor 252 outputs a transcript or activity that occurs during the meeting. For example, the activity or content can be timestamped or otherwise associated with the meeting transcript. In an illustrative example, the meeting activity monitor 252 can indicate the clock time when the meeting starts and ends. In some embodiments, the meeting activity monitor 252 monitors user activity information from multiple user devices associated with the user and / or from a cloud-based service associated with the user (such as an email, calendar, social media, or similar information source), and it may include context data associated with the transcript or content of the event. For example, an email can detail a conversation between two participants, providing context for the meeting transcript by describing the details of the meeting (such as the purpose of the meeting). The meeting activity monitor 252 can determine current or near real-time user activity information, and can also determine historical user activity information, which in some embodiments can be determined based on observations of user activity collected over time, access to user logs of past activities (such as browsing history). Additionally, in some embodiments, the meeting activity monitor can determine user activity (which can include historical activity) from other similar users (ie, crowdsourcing).

[0051] In some embodiments, using contextual information associated with the user device, the user device is identified by the conference activity monitor 252 by detecting and analyzing characteristics of the user device, such as device hardware, software such as an OS, network-related characteristics, user accounts accessed via the device, and similar characteristics. For example, as previously described, information about the user device can be determined using functions of many operating systems to provide information about hardware, OS versions, network connection information, installed applications, etc. In some embodiments, a device name or identification (device ID) can be determined for each device associated with the user. This information about the identified user device associated with the user can be stored in a user profile associated with the user, such as stored in (one or more) user accounts and (one or more) devices 244 of the user profile 240. In an embodiment, the user device can be polled, queried, or otherwise analyzed to determine contextual information about the device. This information can be used to determine a label or identification (e.g., device ID) of the device, so that user activity on one user device can be identified and distinguished from user activity on another user device. Additionally, as previously described, in some embodiments, a user can declare or register a user device, such as by logging into an account via the device, installing an application on the device, connecting to an online service that queries the device, or otherwise providing information about the device to an application or service. In some embodiments, a device (such as a user account) logged into an account associated with the user can be registered. An account or NetPassport, email account, social network, etc.) is identified and determined to be associated with the user.

[0052] User profile 240 may include information learned about meeting participants or attendees, personal preferences of meeting attendees, etc. User profile 240 may include user event activity information 242. This may include an indication of when attendees or speakers tend to mention action items (e.g., via pattern recognition in previous meetings). For example, a particular attendee may always state an action item within the last 5 minutes of a meeting. This information can be used to make predictions throughout the meeting to determine whether a statement is an action item. User profile 240 may also include how attendees or speakers mention action items. For example, historical meeting events may indicate that a particular user always states "let's mark it" after the end of each action item to ensure that the action item is marked as important in the notes, otherwise completed.

[0053] In some embodiments, user event activity information 242 corresponds to an identified pattern of historical behavior of one or more participants of an event. Such patterns may be or include: patterns of participant speech (e.g., participant A always jokes even when issuing commands, making the candidate unlikely to be an action item), a tendency to mention action items at a specific time or time window of an event, a pattern of never being the owner of any action item or publishing any action item, a pattern of always being the owner of an action item or sending an action item, a pattern of people expressing action items or interacting with others (e.g., person A is a manager and always says "job assignment number [x]..." before an action item, which is a signal that the sentence that follows is an action item), the user's language and style, a conversation history of previous communications between participants (e.g., chats, emails, and documents), and the like. Historical patterns can be used to predict whether a candidate action item is indeed an action item. For example, if the conversation history between two participants reveals that the next upcoming meeting Y will be where they decide on the location of the upcoming event Z, the phrase "OK, John, let's hold it in Glendale" at meeting Y may be determined to be an action item based at least in part on the conversation history about the event and the event location. In some embodiments, the patterns identified as participant behavior history 346 are determined by one or more machine learning models (e.g., Bayesian graphs, random forests, linear regression, etc.).

[0054] User profile 240 can include user preferences 244, which generally include user settings or preferences associated with conference monitor 250. By way of example and not limitation, such settings may include user preferences regarding specific events (and related information) that the user wishes to explicitly monitor or not monitor, or categories of events that are monitored or not monitored, crowdsourcing preferences (such as whether to use crowdsourced information), or whether the user's event information may be shared as crowdsourced data; preferences regarding which event consumers may consume the user's event pattern information; and thresholds, and / or notification preferences, as described herein.

[0055] In some embodiments, user profile 240 includes the participant role of each participant in the event. The participant role can be used as a signal to help determine whether an action item candidate is indeed an action item. This is because a particular statement is more likely or less likely to be an action item, depending on the role of the participant who made the statement. For example, if a senior or management engineer mentions "We should create a different design for this customer," the sentence may be an action item, while if a new employee makes the same statement, it is not necessarily an action item because typical employment structures require action items to be either stated by an authority figure or approved by an authority figure.

[0056] In some embodiments, user profile 240 includes alternative or additional information, such as the participant's age, the participant's family members and associated information, the participant's address, etc., to help determine whether a candidate action item is indeed an action item. For example, if it is known that speaker A has a child, Jane Doe, at school, and speaker A says "I have to pick up Jane Doe today," this may be selectively not selected as an action item based on the family member data.

[0057] The context information extractor / determinator 254 is generally responsible for determining context information (e.g., external context) about the event. This information may be metadata or other data that is not the actual meeting content or payload itself, but describes relevant information. For example, the context information may include who is attending or invited to the meeting, the subject of the meeting, whether the meeting is repeated, the location of the meeting, the date of the meeting, other items or relationships between other items, information about the invited or actual meeting attendees (e.g., company role, whether the participants are from the same company, etc.).

[0058] The conference content assembler 256 is generally responsible for receiving the event content and related context information, and generating a rich conference activity timeline. In some embodiments, the timeline is a transcript including tags and / or other associated content. For example, the timeline can include structured data (e.g., a database) including records, wherein each record includes a timeline for each conversation and a timestamp indicating when the conversation started / stopped. The record can alternatively or additionally include context information or metadata, such as information about the participants of the event or the event itself (e.g., any information in the theme, file, slide, or user profile 240 of the event). The rich conference activity timeline can be the output of the conference monitor 250.

[0059] Action item determiner 260 is generally responsible for identifying possible action items from event content. In some embodiments, the input includes the output of conference monitor 250 (e.g., user data and conference-related data from sensors (microphone, video, user activity, etc.), user data collection component 210, and user profile 240 of the user). In some embodiments, the output of action item determiner 260 is a list of possible action items and related corresponding information, such as context data, related files, who the action item belongs to or who must complete the action item, date, etc. In some embodiments, the output of action item determiner 260 includes a structured data record (e.g., a database record) that includes various attributes, such as action item name, attribute (who must complete the action item), action item category / type, related files, and / or content to be provided to remind the user to complete the action item.

[0060] The conference content clarifier 261 is generally responsible for structuring or cleaning up the conference content (e.g., transcripts and contextual data). For example, sentences can be marked using part-of-speech (POS) identifiers, and words can be removed, replaced, and added. In some embodiments, the conference content clarifier 261 parses or tokenizes event content and / or other external information (e.g., information received by the user data collection component 210) and reconstructs the information. In some embodiments, the event content is or includes a document or transcript of the order and content of everything said in the event written in a natural language. For example, the event content can be a written transcript of everything said or spoken during the entire duration of the meeting. In some embodiments, the event content can alternatively or additionally include the audio content of everything said during the event. In some embodiments, the conference content clarifier 261 processes the event content in near real time (e.g., when each statement is stated during the meeting or shortly after each statement is stated). Alternatively or additionally, in some embodiments, the conference content clarifier processes the event content after each event or meeting has ended.

[0061] In some embodiments, the conference content clarifier 261 tokenizes and splits character sequences (e.g., sentences, words) from the content, and adds or connects other character sequences (e.g., semantically similar words) and / or tags (e.g., parts of speech) to the tokenized character sequences. In some embodiments, the connected character sequences are part of a natural language processing (NLP) component or library, in which each content is parsed into its words and some or each of the words are tagged with a part of speech identifier.

[0062] "Tokenization" or parsing in various embodiments corresponds to a computer-implemented process that segments content into words, sentences, symbols, character sequences, and / or other elements of content. "Syntax" or syntactic attributes refer to the structure of the character sequences of the content (as opposed to semantics or meaning), such as the structure of a sentence. This can include a set of rules for analyzing messages, such as words and / or POS order. For example, for the sentence "The girl jumped for joy", the syntax may correspond to a word order, where the structure is subject-verb-object (or subject, verb, object, etc.). In various embodiments, the POS of the message elements are labeled.

[0063] In some embodiments, NLP derives the semantic and syntactic content of semi-structured or unstructured data (e.g., data in image files, blogs, posts, websites, data in text descriptions of classifier models). This is in contrast to analyzing "structured" data (such as data in a database). NLP can be configured to parse content to determine semantic context (e.g., by analyzing each word in the content relative to each other and relative to training data to determine the meaning of the word) and syntactic context (e.g., a set of rules for managing sentence structure in a given language). NLP is configured to identify keywords, contextual information, and metadata tags associated with one or more parts of a data set. In a specific embodiment, NLP analyzes summary information, keywords, text descriptions contained in a data set, and uses the syntactic and semantic elements present in the information to identify contexts of interest. Syntactic and semantic elements can include information such as word frequency, word meaning, text font, italics, hyperlinks, proper names, noun phrases, parts of speech (e.g., nouns, adverbs, adjectives, etc.) and / or the context of surrounding words. Other syntactic and semantic elements are also possible.

[0064] In some embodiments, the meeting content clarifier 261 removes redundant or repeated character sequences from the content. This takes into account that a meeting participant stutters, stammers, mumbles, accidentally says the same thing, or otherwise provides redundant or repeated data. For example, a meeting participant may say "I will... [pause as someone interrupts]... OK, I will send the mmm file." In this example, the meeting content clarifier 261 can remove the "I will" phrase from the character sequence because it is redundant (e.g., it has already been stated).

[0065] In some embodiments, the meeting content clarifier 261 replaces, removes and / or adds a word or a set of words to a sequence of characters in the content to make the content easier to process. In some embodiments, the component uses the semantic and syntactic analysis described above with respect to NLP. Alternatively or additionally, in some embodiments, the component uses a word embedding model (e.g., word-2-Vec, GloVe, etc.) to determine semantically similar words. For example, the meeting content clarifier can modify the phrase "I will make sure you are not blocked" to "[person][will] make sure [person][is not] blocked". Therefore, the sequence structure component 218 can determine that "I'm going" is semantically similar to "I will" and "not" is semantically similar to "not" and make corresponding replacements. In addition, the sequence structure component 218 can determine that "person" is speaking or is the subject and / or object of the sentence.

[0066] In some embodiments, action item candidate identifier 262 can determine candidate based on grammar or other content structure (for example, whether sentence is command).In some embodiments, action item candidate identifier 262 excludes content or action item candidate based on candidate that may end during meeting or private command or request that is irrelevant or not a part of event.For example, statement "I will end discussion of X in the next slide" may be excluded from action item because it is likely to end during meeting.In some embodiments, action item candidate identifier 262 determines which parts of content are candidates for becoming action item.In some embodiments, this occurs after the processing of conference content clarifier 261 or occurs in response to the processing.Action item candidate identifier 262 identifies patterns or syntactic relations in character sequence in content to define boundaries or cutoffs for action item.For example, using the above illustration, after natural language sequence normalizer determines that character sequence is "people will ensure that people are not blocked", it can be identified as candidate for further processing by candidate identifier. In some embodiments, action item candidate identifier 262 identifies language or natural breaks in the content, such as periods, semicolons, when commands are issued, etc., to determine that the content between these characters or commands is a candidate for further processing. For example, if the event content has a total of 4 sentences, and 3 of the 4 sentences are commands, candidate identifier 223 identifies the 3 sentence commands as individual action item candidates for further processing.

[0067] In some embodiments, action item candidate identifier 262 uses action item identification logic unit 230 to identify action item candidate.Action item identification logic unit 230 can include pattern recognition logic unit.Pattern recognition logic unit can include rule, association, condition, prediction and / or classification model, or pattern inference algorithm.Action item identification logic unit 230 can take many different forms depending on specific response activity pattern or the mechanism for identifying response activity pattern, or identify the feature similarity between observed response activity events to determine the pattern corresponding to the task associated with user command.For example, some embodiments of action item identification logic unit 230 can adopt machine learning mechanism to determine feature similarity, or other statistical measurements to determine the response activity event belonging to the exemplary response action set supporting determined response activity pattern, as further described below.Event activity can be received from conference monitor 250, and information about identified similar features can be received from the feature of action item candidate identifier 262.

[0068] Action item candidate classifier 264 is usually responsible for classifying action item candidates as action items or non-action items. In certain embodiments, this component can use any suitable model or a group of models or technology to predict, such as random forest model, deep neural network, Bayesian network or any other suitable machine learning model. In some embodiments where the model is supervised, for example, label or classification target is whether the candidate is an action item. For example, in a random forest embodiment, each leaf node can indicate whether the candidate is an action item, which is based on the decision at branch and root node (for example, based on the extracted value and context data in the user profile 340).

[0069] In certain embodiments, action item candidate classifier 264 detects or classifies written or audio transcripts according to the type of meeting.For example, whether this component can classify an event is a broadcast conference, a tutorial, a board meeting, etc. (for example, based on contextual data information).Alternatively or additionally, action item candidate classifier 264 additionally or alternatively calculates the probability that an action item will be present in the content based at least in part on the type of event and / or other information in the contextual data.In certain embodiments, this probability is used for extra probability to indicate whether a specific candidate is indeed an action item.For example, if it is determined that an event is a tutorial or a speech, wherein, between a speaker and an attendee, there is seldom or no feedback or dialogue, then it is possible to predict that an event will contain a very low probability (for example, a low confidence level) of an action item.Therefore, anything stated or said during an event may be automatically weighted or otherwise scored to a low probability.

[0070] In some embodiments, the action item candidate classifier 264 uses Naive Bayes or other Bayesian network models to make its predictions. The Naive Bayes algorithm is a set of probabilistic algorithms that use probability theory and Bayes' theorem to predict text tags. For example, if the event name does not automatically provide the name or subject of the meeting, the event type detector 332 can use the Naive Bayes algorithm to infer or predict the subject or name of the event. In an illustrative example, the sentence "Let's do more rounds of tracking and debugging" can be labeled with "Software Performance Engineering" and the like. Bayesian network graphs map the relationship between nodes (e.g., events) according to probabilistic mapping. These graphs show how the occurrence of a particular event affects the probability of other events occurring. Each node is also conditionally independent of its non-descendants. These graphs follow the basic principles of Bayes' theorem and are represented as:

[0071]

[0072] Where A and B are events and P(B)≠0. That is, the probability of A given B (P) = the probability of B given A multiplied by the probability of all (A) on the probability of B. The directed acyclic graph includes various nodes, directed edges, and conditional probability tables. For example, each node can represent a feature in the context data associated with the event, so that the probabilities are interdependent, such as because the probability of the event location at X is high, the probability that the subject of the event or event name 372 is Z is Y, and because of the probability Y, the probability that participants 1, 2, and 3 will attend the meeting is T.

[0073] In some embodiments, the action item candidate classifier 264 classifies parts of the content (e.g., action item candidates, paragraphs, sentences, pages, etc.) as specific (the probability that the part is an action item is high) or descriptive (e.g., the probability that the part is an action item is low). Therefore, the component is able to parse the content and map these parts in the memory as possible or impossible action items (e.g., based on syntactic structure). For example, sentences or phrases without any commands or requests (e.g., sentences that are not imperative sentences), such as "Project X is important to our customers" or "We were very happy with the last meeting" or "We have made great progress since the last meeting" are unlikely to become action items. On the contrary, sentences or phrases that are imperative sentences or otherwise include commands or requests are more likely to be action items. Imperative sentences are used to issue commands, make requests, or provide suggestions. Imperative sentences usually end with a period or exclamation point or have other syntactic properties, such as a subject followed by a predicate, and then an object (e.g., "John [subject], please join [predicate] project [object]."). In some embodiments, the component can distinguish between action items and those that appear to be action items but are not based on, for example, syntactic properties and / or history (e.g., participant behavior history). For example, the statement "You can press it" may be labeled as not an action item based on a dictionary or other data structure that lists colloquial terms or spoken phrases, as opposed to an action item.

[0074] In some embodiments, the action item candidate classifier 264 uses a predictive model, such as a hidden Markov model (HMM). HMM is a computational tool for representing probability distributions over observation sequences. These tools model time series data. HMM enhances Markov chains. Markov chains are models that provide insights into the probabilities of sequences of random variables, states, each of which takes values ​​from a data set. These data sets can be words, tags, symbols, etc. The assumption of the Markov chain is that any prediction is based only on the current state, not the state before the current state. The state before the current state has no effect on the future state. In various cases, events of interest are hidden because they cannot be directly observed. For example, POS tags are not usually observed in text. Instead, the words themselves are directly observed and the POS tags (hidden states) are inferred.

[0075] HMM allows the model to use both observed events and hidden events, which are essentially causal factors in probabilistic algorithms. HMM is represented by the following components: a set of N states Q = q1q2...q N , the transition probability matrix AA = a 11 …a ij …a NN , each a ij represents the probability of transitioning from state i to state j, subject to TObservation sequence O=o1o2…o T , each observation is from the vocabulary V = v1, v2, ...v T Extract, observe the likelihood sequence B = b i (o t ), also called the emission probability, each represents an observation o t The probability generated from state i and the initial probability distribution on the state π=π1π2...π N π i is the probability that the Markov chain starts in state i. Some states j may have π j = 0, which means they cannot be the initial state.

[0076] The probability of a particular state depends only on the previous state, so the Markov assumption is introduced: P(q i |q1...q i-1 )=P(q i |q i-1 ). Output observation o i The probability of generating an observation q depends only on i state, and does not depend on any other state or any other observation, resulting in the output of the independent term O(o i |q1...q...,q r, o1, ..., o i ,... T )=P(o i |q i ). This allows the component to state that given an observation o (e.g., whether a sentence is in the form of a command, a type of meeting, or any other value in the context data), the algorithm is able to find a hidden sequence of Q states (e.g., whether a portion of the content is likely to be an action item). In some embodiments, the action item classifier takes as input all of the computed features and / or other character sequences or context data and classifies or otherwise predicts whether each action item candidate is an action item.

[0077] In the illustrative example of how the action item candidate classifier 264 predicts, first, the conference content clarifier 261 can use the entire transcript as input to reconstruct the transcript as described above. In response, the action item candidate identifier 262 can identify the action item candidate based on sentence syntax, exclusion rules, and / or can extract various features from the user profile 240 or context data as described above. In response, the action item candidate classifier 264 can classify events (e.g., via Naive Bayes), determine whether the part or action item candidate in the transcript is the probability of the action item (e.g., via HMM), and then finally classify whether the candidate action item is indeed an action item. For example, the action item candidate classifier 264 can determine whether the candidate is an action item with a specific confidence level based at least in part on the participant preference of each participant in the participant's role and a specific meeting and / or any other data and other context data in the user profile 240.

[0078] In some embodiments, the action item candidate classifier 264 uses a machine learning model, such as a deep learning classification neural network (e.g., a convolutional neural network (CNN), a recurrent neural network (RNN), and a transformer). In a particular embodiment, it is possible to first identify a label, category, or target classification, such as "action item" or "not an action item". These labels, categories, or target classifications can be hard (e.g., class membership is a binary "yes" or "no") or soft (e.g., the label is attached with a probability or possibility). Alternatively or additionally, transfer learning can occur. Transfer learning is the concept of reusing a pre-trained model to solve a new related problem. For example, the confidence level obtained for detecting an action item can be used to detect a non-action item. The new data set is typically similar to the original data set used for pre-training. Therefore, the same weights can be used to extract features from the new data set. In an illustrative example, the original data set may include statements made by conference participants that are classified as action items. It may also be that participants make the statement 95% of the time when posting an action item. Therefore, via transfer learning and for a new incoming data set, the user may not make the statement. Using the same weights, it can be inferred that the statement is not an action item.

[0079] The neural network can include convolutional layers, pooling layers, and fully connected layers. In a particular embodiment, the machine learning model neural network can be fed or receive the output provided by the conference content clarifier 261 and / or the action item candidate identifier 262 at the convolutional layer as input. Each input can be converted into a 2D input vector array of values, such as an integer number of ones and zeros. Each value represents or describes the specific value of the input and the strength of the value. For example, a word or character sequence can be represented by one, while other words or character sequences can be represented by zero. The convolutional layer utilizes one or more filter maps, each of which represents the features of the input (e.g., participant role 344). There may be various features of the input, and therefore there may be various linearly stacked filter maps for a given input. The filter map is also an array of values, which represents a set of values ​​and weights, wherein when a value matches a corresponding value (e.g., training data indicating a participant's behavioral history) or a value set in a corresponding portion of the input, its weight is higher. The convolutional layer includes an algorithm that uses each filter map to scan or analyze each portion of the input. Thus, each value of each filter map is compared and matched to a corresponding value in each portion of the input and weighted according to similarity. In some embodiments, the convolutional layer performs a linear function or operation to obtain the filter map by multiplying each value by its own value and then performing a sum function of each product, which is then divided by the total number of values ​​in the input features.

[0080] In a particular embodiment, the pooling layer reduces the dimension or compresses each feature map by selecting a window size (i.e., the amount of dimensional values ​​to be analyzed in the feature map) and selecting the maximum value of all values ​​in the feature map as the only output for the modified feature map. In some embodiments, the fully connected layer maps or votes for each value of each modified feature to each classification (i.e., the candidate is an "action item" or "not an action item"). The voting strength of each value is based on its weight or value score. The output is a score (e.g., a floating point value, where 1 is a 100% match) indicating the probability that a given input or a set of modified features fits in a specific defined category (either an action item or not an action item). For example, the input may include a set of text stating "Let's start this meeting after I tell a joke... Jake told John to finish the product by Monday." The classification or label type may be "action item" or "not an action item." After the first character sequence "Let's start this meeting after I tell a joke..." is fed through each of the layers, the output may include a floating point value score for each classification type, indicating "Action item: .21", and "Not an action item: .90", which indicates that the character sequence has a 90% probability of not being an action item. Training or tuning can include minimizing a loss function between a target variable or output (e.g., .90) and an expected output (e.g., 100%). Therefore, it may be desirable to get as close to 100% confidence in a particular classification as possible to reduce prediction error. This may occur over time as more training inputs and baseline data sets are fed into the learning model, allowing classification to occur with higher predicted probabilities.

[0081] In some embodiments, action item candidate classifier 264 classifies action item candidates based at least in part on information retrieved by meeting monitor 250 and / or information in user profile 240. For example, action item candidate classifier 264 can receive contextual information from contextual information determiner 254 indicating that action item Y discussed prior to the current meeting has a due date X. Then, during the current meeting, administrative participant John Doe may say “Okay, let’s start doing Y tomorrow.” Action item candidate classifier 264 can then predict that Y is an action item based on the administrator being the speaker and prior information indicating that Y is an action item and has a due date.

[0082] In some embodiments, the action item classifier uses logic unit 235. Logic unit 235 may include rules, associations, conditions, predictions and / or classification models, or pattern inference algorithms. Logic unit 235 can take many different forms depending on a specific response activity pattern or a mechanism used to identify a response activity pattern, or identify feature similarities between observed response activity events to determine a pattern corresponding to a task associated with a user command. For example, some embodiments of logic unit 235 may use a machine learning mechanism to determine feature similarity, or other statistical measurements to determine whether an action item candidate is an action item.

[0083] The enriched action item generator 280 is usually responsible for generating the data that supplements and / or modifies the action item, so as to clarify and provide the additional event context for the action item. As illustrated in the exemplary embodiment of system 200, the enriched action item generator 280 includes action item enrichment 282, action item context generator 286 and enriched action item assembler 288. Action item enrichment 282 is usually responsible for identifying the content of specific action items and enriching the action items. For example, some embodiments of action item enrichment 282 identify the specific text indicating the action item, and determine and / or clarify the data associated with the action item, such as the owner, speaker, expiration date or similar relevant information.

[0084] In some embodiments, action item enrichment 282 includes action item boundary maker 283 and action item clarifier 285.Action item boundary maker 283 is usually responsible for highlighting or otherwise defining the action item boundary (for example, the beginning and ending words of action item) in a larger text body for presentation.In some embodiments, action item boundary maker 283 calculates the beginning and / or ending of the action item for presentation, which indicates the character sequence boundary.For example, an embodiment can indicate that in the text "I will send a file tomorrow, you can handle it at will", the action item is "I will send". Word "I" can be defined as the beginning boundary for the action item, and word "tomorrow" can be defined as the ending boundary, so that the displayed action item text is "I will send a file tomorrow".

[0085] In some embodiments, identification of the start and / or end boundaries of an action item includes using one or more models (e.g., deep neural networks (DNNs)), such as CNNs, recurrent neural networks (RNNs), recurrent neural networks) and / or natural language processing components (e.g., meeting content clarifier 261). For example, identifying a correct start may include identifying a verb or other part of speech or word (e.g., a person's name) within a character sequence, indicating an initiation of a command or imperative sentence (e.g., work, please, Johndoe). These indicate the start of an action item, such as "John, please complete task X by Friday," where John is the start. In some embodiments, identifying a correct end may include identifying natural language interrupt identifiers, such as periods, question marks, semicolons, etc. These indicate the end of an action item. For example, using the above illustration, "John, please complete task X by Friday" may include a period directly after the word Friday, which indicates the end of a period-based action item.

[0086] In certain embodiments, the first model is used to identify the correct start of the text presented, and the second model is used to identify the correct end of the text presented. These or other model techniques can be carried out together with several tests, so that a network understands the meaning of sentences or other character sequences (e.g., phrases, paragraphs) based on tests. Therefore, these combinations of models can predict, for example, whether the next sentence is the answer to the first sentence. In these embodiments, the relationship between the sentences or other character sequences is learned, and the label or classification can be the character sequence candidate "is the next sentence" or "is not the next sentence". In this way, for example, it is possible to predict whether the first sentence is the sentence that starts or follows the second sentence, or whether the sentence is a random sentence. This can be useful in some cases, for example, conference participants begin to participate in action items, are interrupted or otherwise participate in the discussion of other topics, and then the action items continue. In this way, the embodiment can predict that the first two character sequences are not compatible with each other. For example, a portion of a transcript may be: "Bob, we have a meeting on Friday, so [first character sequence].... Wait Jack, let me say before you go on.... [second character sequence] I mean Bob, let's get Project X done [third character sequence]." Thus, an embodiment is able to predict that the first character sequence and the third character sequence appear together, which can help indicate that "Complete Project X" may be the end of the action item, while "Bob" is the beginning of the action item.

[0087] In an exemplary illustration of a model that can be used to define the start and / or end of an action item, a BERT model or other similar models can be used. For example, BERT generates a language model by using an encoder to read all content at once or in parallel (i.e., it is bidirectional), rather than reading text from left to right. This allows the model to learn the context of words or other character sequences based on their event contexts. The input is a sequence of tokens (e.g., a natural language sentence or other character sequence), which are first embedded in a vector (e.g., via GloVe or other word embeddings), and then processed by a neural network. The output is a sequence of vectors, wherein each vector corresponds to an input token with the same index. These models overcome the typical language model that predicts the next word or other character sequence using a directional scheme. However, the directional scheme inherently limits context learning because it only uses the context before or after the word or character sequence.

[0088] In some embodiments, the end of the definition action project includes an algorithm of two or more steps. The first existing punctuation mark or the end character sequence end identifier (e.g., period, question mark, semicolon, etc.) can be identified to determine the character sequence length of the text on the left or before the sequence end identifier. For the second step, the component can extract or copy the next or subsequent five (or other predetermined number) words in the start word or other character sequence, and feed them through the DNN to determine whether the third (or other predetermined number) word or other character sequence is the end position of the action project. In various embodiments, the second step is completed at least in part based on the part of speech of the predetermined word, its corresponding updated display text length and / or the specific word itself, and the specific word can be a word usually associated with a command or action project (e.g., a request to complete an action project before "tomorrow", where tomorrow is the end of the action project). In various cases, the longer the action project candidate is, the more likely the word is to be selected as an earlier end point in the character sequence, so as to make the presented action project shorter. On the contrary, the longer the action project candidate is, the more likely the word is to be selected as a later end point in the character sequence, so as to make the presented action project longer. For example, for the phrase "Don, please visit Jake today", it can be determined based on all ending candidate parts of speech, words and lengths whether "visit", "Jake" or "today" is the best word to end the action item. In view of the short length of the character sequence, "today" is a keyword indicating the end of the action item, and the part of speech of "today", the word "today" can be selected as the end boundary of the action item, so that the action item presented is "Don, please visit Jake today."

[0089] In some embodiments, the action item boundary generator 283 also performs deduplication. Deduplication of action items can include removing redundant or repeated statements of action items. For example, the action item "Complete Y by Friday" may be stated twice in a meeting, and therefore one of the statements can be removed. Verifying consistency and merging can include combining different statement versions of the same office action. For example, the first action item may be "Rex, let's do project Y". The second action item may say "So yes... let's keep working on completing project Y." These two action items can be merged into one action item, which is converted in a manner consistent with these two statements, such as "Rex will perform project Y". In some embodiments, the action item clarifier uses a deduplication logic unit 237. In some embodiments, a word embedding model (e.g., Word2Vec or other structure) can be used to indicate whether the statements are semantically similar (e.g., within the co-signing distance). Therefore, when the statements are semantically similar, they can be combined by linearly combining or generating a single sentence (e.g., via NLP) to represent all the statements.

[0090] Action item clarifier 285 is usually responsible for generating clarity score and clarifying action item candidate, such as about the content referred to by action item, who has action item (that is, who is responsible for completing action item), who states this action item and / or when action item expires.In some embodiments, action item clarifier 285 receives action item candidate with appropriate boundary or start and end words as input, as described above about action item boundary generator 283.In some embodiments, this can be based on the context data obtained by conference monitor 250, the information in user profile 240 and / or the classification made by action item candidate classifier 264 to obtain.Some statements made during conference or other events are incomplete or unclear about the rest of context in other ways.Therefore, action item enrichment 282 can use context data as signal to clarify action item.For example, the statement in the meeting may say "OK, do X from tomorrow." Only from this statement, it may be unclear who is talking and who should perform X from tomorrow. However, external contextual data obtained by meeting monitor 250 from a chat prior to the meeting may indicate that Jane Doe should do X and her boss Alexis will tell her during the meeting when X is completed. Therefore, Alexis may be the owner of the action item and the action item may therefore be attributed to her.

[0091] In some embodiments, action item clarifier 285 generates a clarity score that indicates how clear the action item is (e.g., based on whether the action item candidate indicates who owns the action item, the due date of the action item, who stated the action item, etc.). In some embodiments, the score is part of a regression model in which a continuous score or range of scores is provided, e.g., the higher the score, the more likely the candidate is clear, and the lower the score, the more likely the candidate is unclear. Alternatively, in some embodiments, the score is part of a classification model in which the label is a category and a confidence level score can be associated with the label. In these embodiments, the label can be, for example, "clear" or "unclear."

[0092] In some embodiments, there are three separate models, each of which is used to determine who has / stated an action item, what the action item is, and when the action item expires. In some embodiments, these three models are part of ensemble learning in an ensemble model. Ensemble learning helps improve machine learning results by combining multiple models. That is, various meta-algorithms combine multiple machine learning techniques into one prediction model to reduce variance or bagging, bias or boosting, and / or improve prediction or stacking. In this way, for example, each of the three scores of "who", "what" and "when" can be combined to make a prediction. Using these three scores, it is possible to determine whether a character sequence has a clear object, such as a noun, a name (indicating "who"), a time indicator (e.g., "tomorrow" or other dates indicating "when"), etc. Using ensemble learning, better prediction performance can be obtained using a relatively small training data set compared to the prior art.

[0093] In an illustrative example of a model for determining "who", "what" and "where", DNN, Naive Bayes, HMM and / or Random Forest models can be used. For example, using Random Forest for the "who" clarity score, the classification label corresponding to the leaf node can be the name of each meeting attendee indicated in the context data. At least a portion of the root node and branch node tests within each decision tree may be different but reach the same leaf node. For example, two leaf nodes may be Bob and Jack indicating meeting participants. A branch node of the first decision tree may be whether the action candidate has a specific utterance or phrase, such as "Let's light a fire...", which may indicate that Bob said this based on similar utterances of Bob, as indicated in the historical external context data of past meetings. Therefore, traversing down the tree may be down the path leading to Bob instead of Jack. In addition, different decision trees may have branch node tests for specific speech recognition fingerprints, and if the fingerprint of the real-time or voice recording matches Bob, the Bob leaf node may be selected. Then, a vote can be performed to see which leaf nodes are selected the most in each decision tree, and Bob has been unanimously selected using this example. In some embodiments, a similar process can be performed for the owner of the action item, when the action item expires, and what the action item is.

[0094] In some embodiments, the action item clarifier 285 can map the content character sequence to the identity of the speaker or the person who is responsible for completing the action item in any suitable manner. For example, in some embodiments, the speech recognition component can be used for audio content input to map the input phoneme to the known or predetermined phoneme library of a specific user (for example, as found in the user behavior history of context data). Therefore, the speech recognition component can record the voice of each user in the user profile 240 (for example, each user who can potentially attend the meeting). In this way, it is possible to predict that a specific parsed character sequence is spoken by a specific user. Alternatively or additionally, mapping or association can include a written transcript with a name or other identifier next to each character sequence. In an illustrative example, the action item clarifier 285 can use the written transcript of the meeting as input. The sentence "We need to complete the draft by Friday" can be tokenized into individual words, wherein the subject of the sentence is marked (for example, we), the predicate of the sentence is marked (for example, complete), and the object of the sentence is identified (for example, draft), and semantic context, such as "the action item draft will be completed by Friday", and the instruction manager John said this sentence.

[0095] Action item context generator 286 is usually responsible for determining and / or causing the presentation of the event context (and / or external context) of a specific amount associated with the action item. For convenience, event context can be shown to the user, and may be useful when the user is still unclear about the information associated with the action item displayed. For example, even after the action item enriches 282 and has generated all necessary information and subsequently displayed the action item, the user may still not understand, for example, what the action item refers to or who owns the action item. Therefore, the user can scroll down, open a window or otherwise view the other parts of the event transcript to view the event context so as to determine this unclear information. In certain embodiments, the event context of a specific amount to be displayed is determined based at least in part on the clarity score and / or other functions (for example, the beginning and end boundaries of the action item candidate) about the action item enriches 282. For example, the amount of the event context displayed can have an inverse relationship with the clarity score. That is, the clarity score indicating the action item candidate is clear is higher, and the context provided for display is less. On the contrary, the clarity score indicating the action item candidate is unclear is lower, and the context provided is more. In some embodiments, if the clarity score exceeds a threshold score, a certain amount of event context is shown. Conversely, if the clarity score is below the threshold score, another certain amount of event context is shown.

[0096] Abundant action items assembler 288 is usually responsible for generating the list of action items and can also provide relevant information, such as event context.For example, abundant action items assembler 288 can generate the list of action items, and it is merged with other information that action items enrich 282 determine, such as who has action items, who has stated action items, when action items expire and what action items are.In certain embodiments, abundant action items assembler 288 assembles other contextual data additionally or alternatively, such as file, project name and / or other associated information.For example, the meeting may have included POWERPOINT presentation, and this presentation may have action items list or other information for display.

[0097] In some embodiments, the enriched action item generator 280 uses an enrichment logic unit 239. The enrichment logic unit 239 may include rules, associations, conditions, predictions, and / or models or pattern inference algorithms for determining rich content related to action items. For example, depending on the specific tasks associated with the action items, event contexts, and / or external contextual information, the enrichment logic unit 239 can take many different forms. Some embodiments of the enrichment logic unit 239 may include one or more models, machine learning mechanisms, and / or other statistical measurements to determine the boundaries of action item candidates, clarify action items, and / or generate action item contexts, such as described above in conjunction with the enriched action item generator 280 or its subcomponents (e.g., the machine learning models described with respect to the action item boundary generator 283 and / or the action item clarifier 285). In some embodiments, the enrichment logic unit 239 can be included in Figure 3 (one or more) definition models / layers 308, Figure 3 The start / end model / layer 303 or otherwise used by the system 300, such as in conjunction with Figure 3 Further described.

[0098] continue Figure 2 In the system 200 in FIG. 2 , the feedback processor 270 is generally responsible for capturing or receiving user feedback and retraining the model to update the model or otherwise recording the feedback in a memory. For example, according to one embodiment, a user interface is provided in which a user is able to select a button (or otherwise provide input) that indicates that the action item is not understandable (e.g., in terms of who stated the office action or who owns the office action) or is not an action item at all. The feedback processor 270 can record the feedback input and use the feedback input to reconfigure specific logic units or retrain or otherwise tune specific one or more models used to generate rich action items. For example, but not limited to, in some embodiments, the (one or more) models that can be tuned or retrained via the feedback processor 270 can include a combination of Figure 2 The action item identification logic unit 230 or enrichment logic unit 239 described below, or Figure 3 The action item prediction model / layer 302 and / or clarity model / layer 308 described further herein. In the illustrative example, if the user indicates that the displayed action item is not an action item, the action item prediction model / layer 302 can use this information to determine a lower score (or other confidence level) so that at a future time, if the same or similar action item candidate is submitted as an action item, it will be marked as lower in confidence as an actual action item. In some embodiments, the feedback processor 270 is combined with Fig. 8A and Figure 8B Process the feedback as described.

[0099] The exemplary system 200 also includes a presentation component 220, which is generally responsible for presenting content and related information to the user, such as content related to user feedback. The presentation component 220 may include one or more applications or services on a user device, across multiple user devices, or in the cloud. For example, in one embodiment, the presentation component 220 manages presenting content to the user across multiple user devices associated with the user. Based on the content logic unit, device characteristics, and / or other user data, the presentation component 220 can determine on which (which) user devices the content is presented, as well as the context of the presentation, such as how (or in what format and how much content, which may depend on the user device or context) to present, when to present. In particular, in some embodiments, the presentation component 220 applies the content logic unit to device characteristics or sensed user data to determine various aspects of content presentation.

[0100] In some embodiments, presentation component 220 generates user interface features associated with information generated by action item enrichment 282, action item context generator 286 and / or enriched action item assembler 288. Such features can include interface elements (such as graphic buttons, sliders, menus, audio prompts, alarms, warnings, vibrations, pop-up windows, notification bars or status bar items, in-application notifications or other similar features interacting with users), queries and prompts. For example, presentation component 220 can present action items including the highlighted borders of action items determined by action item enrichment 282. Presentation component 220 can additionally or alternatively present other missing information, such as who has the action item, who stated the action item, the expiration date of the action item and what the action item refers to, as determined by action item enrichment 282. Presentation component 220 can additionally or alternatively present the context of the required specific amount as determined by action item context generator 286. Presentation component 220 can use the information assembled by enriched action item assembler 288 as input, and make the information presented on the user interface.

[0101] In some embodiments, a personal assistant service or application operating in conjunction with presentation component 220 determines when and how to present content (e.g., presenting only action items or action items with metadata, rather than the entire meeting transcript). In such embodiments, content comprising a content logic unit can be understood as a recommendation to presentation component 220 (and / or personal assistant service or application) when and how to present a notification, which can be overridden by the personal assistant app or presentation component 220.

[0102] continue Figure 2As previously described, the exemplary system 200 may include or operate in conjunction with one or more consumer applications 290. Consumer application 290 generally refers to one or more computer applications or services, such as online / cloud applications or locally stored applications, that consume or utilize rich action item information determined by system 200. Specifically, consumer application 290 may receive rich action item information for a particular user and present, manage, or schedule aspects of the action items to the user. In some embodiments, consumer application 290 may utilize presentation component 220 to provide rich action items to users (and / or other users who are meeting participants) responsible for completing and / or stating action items. Examples of consumer application 290 may include, but are not limited to, computer applications or services for facilitating meetings or communications (e.g., MICROSOFT ); email, messaging, chat, or phone; project management; calendar or scheduling; and task list or to-do apps (e.g., ). In some embodiments, consumer application 290 can access a rich action item (or a portion of its content).

[0103] Steering Figure 3 , provides a block diagram of an exemplary system 300 according to some embodiments, which illustrates a model (or layers of a single model) that can be used to generate rich action items and associated data, and how the model or layer can be tuned based on receiving feedback. In some embodiments, Figure 2 The rich action item generator 280 utilizes Figure 3 , to predict or infer action items and / or rich content associated with action items. Likewise, in some embodiments, Figure 2 The action item determiner 260 uses the action item prediction model / layer 302 to predict or infer. The action item prediction model / layer 302 (for example, by the action item candidate classifier 264) is used to classify or otherwise predict whether a group of characters are action items. The input to the action item prediction model / layer 302 is a group of character sequences, such as a transcript of the entire meeting in which both action items and non-action items exist, which may include event context. In some embodiments, external context may also be provided. In some embodiments, the action item prediction model / layer 302 is a CNN model or other suitable machine learning models. The output is a collection or list of action item candidates 304, and context data, both of which are inputs for starting / ending the model / layer 306.

[0104] The start / end model / layer 306 (e.g., enriched by action items 282) is used to determine the best start and / or end of an action item candidate, such as described above about the boundary of the action item candidate generated by the action item boundary generator 283. For example, the start / end model / layer 306 can adopt all event contexts and utilize more event contexts (e.g., add words) to supplement the action item candidate so that the supplemented information is now the action item candidate. Alternatively or additionally, the model can be used to remove character sequences from the action item candidate so that the new action item candidate reflects the removed text. In an illustrative example, the start / end model / layer 306 can use the rest of the action item candidate "he will send" and the event context to add event context so that the new action item candidate is "tomorrow he will send a file", wherein "tomorrow file" is the event context that is originally directly located after the action item candidate "he will send". In this way, "he" and "tomorrow" mark the boundary for the new action item candidate, wherein "he" is the start word of the action item candidate, and "tomorrow" is the end word of the action item candidate. The output of the start / end model / layer 306 is a new action item candidate (e.g., "tomorrow he will send the file") and updated context data 320-1, because in some embodiments some context data 320 may now have become an action item candidate. However, in some embodiments, context data 320-1 is the same context data as context data 320. In some embodiments, the start / end model / layer is an additional machine learning model or set of models, such as a deep learning machine learning model to determine the boundaries of the action item candidates.

[0105] The component (e.g., action item enrichment 282) then uses this output information from the start / end model / layer 306 and the clarity model(s) 308 to modify or supplement new action item candidates with any missing or uncompleted information that at least partially completes or at least partially clarifies any missing or uncompleted information within or associated with one or more action item candidates. For example, using the "tomorrow he will send the file" described above, a clarity score can be generated indicating that it is unclear who made the statement and who owns the statement. When the context data and the action item candidate(s) are run through the clarity model(s) / layer(s) 308, the output 304-2 can include the name of the person(s) who made the statement and who owns the statement, as well as the action item candidate. For example, new output 304-2 may include replacement words within action item candidate 304-1, such as “[John]: “Tomorrow [Jake] will send the file,” indicating that John made the statement and Jake is the person responsible for sending the file tomorrow. Alternatively, replacement words embedded in action item candidate 304-1 need not be provided at final output 304-2, but rather action item 304-1 can be output with supplemental information. For example, using the above description, the output “Tomorrow he will send the file” can be supplemented with information such as “John made the statement and Jake is responsible,” representing output 304-2. In some embodiments, (one or more) clarity models / layers can be one or more machine learning models, such as an ensemble model, or any other suitable model described with respect to action item enrichment 282. In some embodiments, output 304-2 is content presented or displayed to a computing device, such as within a user interface.

[0106] Feedback 310 represents any user feedback indicating whether the action item presented is indeed an action item or whether there is still missing or unclear information. The input made by the user indicating that the action item is not actually (or is) an action item is fed to the action item prediction model / layer 302. The input made by the user indicating that the information (e.g., the spokesperson of the action item, the owner of the action item, the expiration date of the action item) that is still unclear or missing (or the action item is correct) is fed to (one or more) clarity models / layers 308. These inputs can be used for additional tuning or training of the model / layer, so as to learn from this user feedback. For example, in response to receiving the selection of the thumbs-down icon (indicating that the action item presented is not an action) on the user interface, the action item prediction model / layer 302 can be tuned so that the same or similar submitted action item candidate is actually towards being an action item and scoring lower (e.g., confidence level is lower).

[0107] Figure 44 is a schematic diagram of a presented action item 404 embedded in an event context 400 according to some embodiments. In some embodiments, the event context 400 is presented within a user interface or display text, such as in a chat room or other online event venue. Figure 4 As illustrated in , action item 404 "We should do it" may appear unclear, such as "who" is to do it, and what "it" refers to. Various embodiments of the present disclosure can use other event contexts, such as pre-event context 402 (e.g., a sequence of characters entered before action item candidate 404 is entered) and post-event context 406 (e.g., a sequence of characters entered after action item candidate 404 is entered). Alternatively or in addition, various embodiments can use external context data, such as text, email, calendar data, etc. For example, using Figure 4 , an embodiment (e.g., via action item enrichment 282 of a BERT model) is able to predict that "it" refers to ending the delay chart based on statements 402-2 and 402-1 made by Mark and Emily and an external email describing the delay chart as a topic of the meeting. Additionally, an embodiment is able to predict that the due date is "this week" based on statement 406-1 made by Emily.

[0108] Figure 5A Illustrated is an exemplary screenshot 500 indicating the content of a conference event in accordance with some embodiments. Figure 5A Represents a virtual or online meeting or chat between meeting participants or attendees. Screenshot 500 may represent all event context for the event, such as everything said or stated during the event. This may include statements or threads that do not indicate action items (such as 503 and 501) as well as statements or threads that include action items (such as 505). In some embodiments, all content within screenshot 500 (and / or 500-1, 500-2) or similar screenshots is provided for display and is included in the screenshot. Figure 2 Output 304-2, or indicates Figure 2 The action items enrich the output of 282. In some embodiments, the screenshot 500 (and / or 500-1, 500-2) is composed of Figure 2 The presentation component 220 is used to generate.

[0109] Figure 5B An exemplary screenshot 500-1 indicating a list of action items / action item candidates is illustrated in accordance with some embodiments. In some embodiments, in response to receiving a selection of button 502 (e.g., in Figure 5A500), each action item and event context are listed.For example, for statement or character sequence 504, both context 504-1 and action item candidate 504-2 before the event are displayed.As illustrated in statement 504, action item candidate 504-2 is highlighted, and it shows the position where the boundary of action item or action item starts and ends.That is, word " I " starts action item 504-2, and word " morning " ends action item 504-2.In certain embodiments, as illustrated above, action item enriches 282 and finds the best boundary, and causes the presentation to boundary, as illustrated in action item 504-2.For example, action item enriches 282 and can use start / end model / layer 306 to define and present the boundary of highlighting.Highlighting can include superimposed data or otherwise change the font or format of character sequence, such as the bold character sequence illustrated in action item 504-2. In some embodiments, the symbol (1) within button 502 indicates that only 1 action item is determined within a particular event (or a particular UI view of a particular event), but it will be appreciated that any number of action items may exist within a particular view.

[0110] Figure 5C An exemplary screenshot 500-2 indicating various other user interface features according to some embodiments is illustrated. In some cases, a statement or other character sequence does not include any action items, such as statement 520. Therefore, an embodiment can receive an indication of a selection of user interface element 514 (trash can), which indicates that the user input indicates that the associated statement or character sequence is not an action item. For example, as soon as the user sees the character sequence 520, the user can hover the graphical pointer over element 514, and a prompt "This is not an action item" is displayed. The user can then select element 514, at which point an embodiment (e.g., feedback processor 270) receives the selection and removes the character sequence 520 and / or otherwise marks the character sequence 520 as not an action item. Alternatively, an embodiment can also receive an indication of a selection of user interface element 512. In response to receiving an indication of a selection of element 512, which corresponds to a user input indicating that the associated statement or character sequence is an action item, an embodiment (e.g., feedback processor 270) can mark the associated character sequence as an action item. As described in more detail below, selection of user interface elements 514 and 512 or similar elements can be used to train a machine learning system.

[0111] In response to receiving an indication of a selection of “hide context” feature 516, an embodiment (e.g., action item context generator 286) can remove additional event context for display. Similarly, in response to receiving an indication of a selection of “show context” feature 518, an embodiment (e.g., action item context generator 286) can add event context for display. This may occur, for example, in a situation where the user is still unclear about information associated with the action item (such as the action item's due date, etc.). Alternatively, the user may want to view only action item candidates or only character sequences that include action item candidates (e.g., Figure 5B 504), and thus the user may wish to collapse or remove other event contexts. As described above, in some embodiments, the amount of event context is initially presented or displayed based at least in part on the clarity or clarity score of the action item candidate. Thereafter, the user may still select features 516 and / or 518 to manually adjust the presented event context.

[0112] Figure 6 is an exemplary screenshot 600 of a user interface according to some embodiments. In some embodiments, the screenshot 600 includes Figure 3 In some embodiments, the screenshot 600 includes the final output of the system 200, if any. Figure 2 In some embodiments, the screenshot includes a Figure 2 The action item enriches 282 the information determined or provided. Figure 6 Various conference event notes and specific user interface elements are illustrated. User interface element 602 represents a field or data object configured to receive an action item that a user has manually added. For simplicity, the text of the element is not shown. User interface element 604 represents an action item that an embodiment (e.g., system 200) has generated or otherwise implicitly extracted from an event context. User interface element 606 is an optional feature that is configured to receive a user selection, and the user selection indicates confirmation that the action item is accurate and / or clear. Alternatively, in some embodiments, user interface element 606 is used to expand or show more context. In some embodiments, the context as illustrated is less, for example, by indicating a strikethrough mark of the less context shown.

[0113] User interface element 608 is an optional feature that is configured to receive a user selection indicating that the user requests to edit text within an action item or other event context associated with the action item. Therefore, in some embodiments, in response to receiving an indication that element 608 has been selected, a particular embodiment may make the text within action item candidate 604 editable. For example, after selecting element 608, the user may enter the owner, deadline, spokesperson, etc. of the action item.

[0114] User interface element 610 is an optional feature that is configured to receive a user selection indicating user feedback, i.e., an action item is not actually an action item or other information (e.g., a speaker, an owner, etc.) is unclear. User interface element 612 is an optional feature that is configured to receive a user selection indicating user feedback about a specific action item (e.g., action item 604). For example, in response to receiving a selection of feature 612, some embodiments (e.g., presentation component 220) may cause the presentation of a window, field, or other element, so that a user can manually enter a note about a corresponding action item, such as an owner, a speaker, a deadline, etc. User interface element 614 is a field or other object configured to receive manual user input, and the manual user input indicates general feedback about the entire conference event or a list of all action items. For example, user interface element 614 may receive natural language input from a user, such as "Overall, the action item is accurate" or "Note: The next meeting is Wednesday, the 25th". In some embodiments, selection of elements 606, 608, 610, 612, and / or 614 can be used by an embodiment (e.g., feedback processor 270 as feedback 310) to train or tune one or more machine learning models, as described in detail below.

[0115] Figure 7 is an exemplary screenshot 700 of a user interface according to some embodiments. In some embodiments, the screenshot 700 includes Figure 3 In some embodiments, the screenshot 700 includes the final output of the system 200, such as the output 304-2 of the system 200. Figure 2 In some embodiments, the screenshot 700 includes a Figure 2 The action item enriches 282 the information determined or provided. Figure 7 Various event action items, contextual data, and specific user interface elements are illustrated.

[0116] User interface element 702 is selectable and is configured to cause display more content or event content in response to its selection. For example, in response to receiving the indication of the selection of user interface element 702, the embodiment can cause to display more event contexts in addition to action item 706, similar to event context 708 (for example, the statement carried out before and / or after action item 706 is stated). User interface element 704 is optional and is configured to cause display less content in response to its selection. For example, in response to receiving the indication of the selection of user interface element 704, the embodiment can cause the removal of context data 708 in screenshot 700, so that the only display item associated with this event context is action item 710.

[0117] The labeling of the action item 710 presented indicates that its boundaries are highlighted (e.g., underlined and bolded as generated by action item enrichment 282 as described above). In this way, the user can quickly identify which text corresponds to an action item and which text does not correspond because the non-action item data is not highlighted. In some embodiments, the default number or amount of event context presented with each action item candidate (e.g., 710, 706) is determined by the clarity score of the corresponding action item, as described above with respect to Figure 2 As described by the action item context generator 286 of . For example, a high clarity score indicating that the action item 706 is not missing any information or is otherwise clear may be generated. In response, a high clarity score may be generated as described in Figure 7 Only action item 706 is shown. In another example, a low clarity score may be generated indicating that action item 710 lacks information or is unclear. In response, event context 708 may be additionally displayed with action item 710, such as in Figure 7 As shown in the figure.

[0118] User interface element 712 is selectable and is configured to cause an expansion of event context or other content in response to its selection. For example, in response to receiving an indication of a user selection of element 712, an embodiment (e.g., presentation component 220) may cause more event context and / or action items to be displayed, such as an entire transcript of the event. For example, screenshot 700 may represent a list of action items identified in a meeting and their selectively generated event context, as described above. In response to selection of user interface element 712, an embodiment may cause display of the entire transcript of the meeting event or additional event context relative to the event context, as shown as a result of receiving an indication of a selection of elements 702 and / or 704.

[0119] User interface elements 714, 716, and 718 represent different user feedback features for a particular action item 706. The same user interface elements are also displayed under other corresponding action items, such as action item 710. In some embodiments, user interface elements 714, 716, and 718 perform operations respectively related to Figure 6 5 . In some embodiments, user interface element 714 additionally or alternatively includes the same or similar functionality as user interface elements 514 and 512 of FIG. 5 , respectively. Thus, an embodiment (e.g., feedback processor 270) can receive one or more selected indications indicating that the action item is clear / unclear or otherwise missing information and / or is / is not an actual action item. In this way, user feedback (e.g., feedback 310) can be used by one or more machine learning models to make future predictions based on the information about the action item and / or the amount of event context to be displayed, as described above.

[0120] Fig. 8A illustratively shows exemplary action item candidates and associated user interface elements according to some embodiments. In some embodiments, user interface elements 803, 805, and 807 represent or execute actions respectively associated with Figure 7 User interface elements 714, 716 and 718 and / or respectively Figure 6 5 . In some embodiments, user interface element 803 additionally or alternatively includes functionality that is the same as or similar to user interface elements 514 and 512 of FIG. 5 , respectively. Thus, an embodiment (e.g., feedback processor 270) can receive indications of one or more selections indicating that action item 801 is / is not clear or otherwise missing information and / or is / is not an actual action item. In this manner, user feedback (e.g., feedback 310) can be used by one or more machine learning models to make future predictions about action items and / or the amount of event context to be displayed based on the information, as described above. Such user feedback is useful in Figure 8B is illustrated in FIG, which is an illustration of how to use Fig. 8A A schematic diagram of a corresponding machine learning model selected by a user to train or tune.

[0121] In response to receiving an indication of a user selection of user interface element 805, an embodiment (e.g., presentation component 220) can cause display of window 809. Subsequently, an embodiment can receive a user selection of a label within window 809 that reads "This is not an action item" or "This is hard to understand." In response to receiving an indication of a selection of the "This is not an action item" text (indicating that the text is selected as in Figure 8B), which is then run through a neural network 806 (which can be any machine learning model) so that the embodiment can be tuned or otherwise learned for future predictions 804. For example, in response to the selection, an embodiment (e.g., presentation component 220) can cause action item 801 to be removed from the display or otherwise not highlighted to indicate that it is an action item. Such removal can occur during the same session that initially indicated action item 801 or for future sessions. Alternatively or additionally, any future similar action item candidates can be scored toward a lower probability of being an action item based on the selection of element 805.

[0122] In a similar manner, in response to receiving an indication of a user selection of user interface element 803 (represented as in Figure 8B 814), an embodiment (e.g., presentation component 220) can run the selection through neural network 804 so that the embodiment can be tuned or otherwise learn this for future predictions 804. Positive feedback 810 indicates that the user indicated that a particular context is indeed an action item. For example, in response to the selection, an embodiment (e.g., presentation component 220) can cause action item 801 to be retained for display or otherwise highlighted to indicate that it is an action item (for both the current session and future sessions). Alternatively or additionally, any future similar action item candidates can be scored toward a higher probability of being an action item based on the selection of element 803. In some embodiments, neural network 806 represents Figure 3 Action item prediction model / layer 302.

[0123] In response to receiving an indication of a user selection of user interface element 805, an embodiment (e.g., presentation component 220) can cause display of window 809. In response to receiving an indication that the "THIS IS HARD TO UNDERSTAND" text is selected (indicating that the text is displayed as in Figure 8B ), the selection is then run through a neural network 802 (which can be any machine learning model) so that the embodiment can be tuned or otherwise learn this for future predictions 804. For example, in response to the selection, an embodiment (e.g., presentation component 220) can cause action item 801 to be removed from display or modified in some way, such as adding or replacing information (e.g., the name of the action item owner, the name of the speaker, the deadline for the action item, what the action item really is, etc.). Such modifications can occur during the same session in which action item 801 was initially displayed or for future sessions. Alternatively or additionally, any future similar action item candidates can be scored with a lower clarity score based on the selection of element 805.

[0124] In a similar manner, in response to receiving an indication of a user selection of user interface element 803 (represented as in Figure 8B 803), an embodiment (e.g., presentation component 220) can run the selection through neural network 802 so that the embodiment can be tuned or otherwise learn this for future predictions 804. Positive feedback 810 indicates that the user indicated that the action item was not difficult to understand. For example, in response to the selection, an embodiment (e.g., presentation component 220) can cause action item 801 to be retained for display or otherwise highlighted to indicate that it is a clearly stated action item (e.g., in the current session or a future session). Alternatively or additionally, any future similar action item candidates can be scored toward a higher clarity score based on the selection of element 803. In some embodiments, neural network 802 indicates that Figure 3 (One or more) clarity models / layers 308.

[0125] Now go to Fig. 9 , according to some embodiments, an exemplary process 900 for providing rich action items. Process 900 (and / or any function described herein) can be performed by a processing logic unit including hardware (e.g., circuits, dedicated logic units, programmable logic units, microcode), software (e.g., instructions running on a processor to perform hardware simulation), firmware, or a combination thereof. Although the specific blocks described in this disclosure are referenced in a specific order and in a specific amount, it should be understood that any block can be substantially parallel to any other block or appear before or after any other block. In addition, there may be more (or fewer) blocks than illustrated. For example, in some embodiments, process 900 does not include block 965. Any added block may include a block embodying any function described herein. A computer-implemented method, system (including at least one computing device having at least one processor and at least one computer-readable storage medium) and / or a computer storage medium as described herein can perform or be caused to perform process 900 or any other function described herein. In some embodiments, some or each of blocks 905 to 965 may be automated or otherwise occur in a non-manual manner (eg, without an affirmative user device request), which improves upon the prior art, as described above.

[0126] According to box 905, determine the content of the meeting (e.g., by the meeting monitor 250). For example, the meeting activity monitor 252 can use one or more sensors or other components to monitor a chat, a portion of the context presented or a transcript. The context information extractor / determinator 254 can determine the context information of the event, such as who is present or invited to attend the meeting, the theme of the meeting, the location of the meeting, or other contexts or character sequences within the transcript or meeting content itself. Then, the meeting content assembler 256 can generate a rich meeting activity timeline, such as a timeline including each conversation and a tag or structured data indicating when the conversation starts / stops. In a specific embodiment of box 905, determine the content from the meeting, which can be determined by monitoring the meeting information about the meeting, such as transcript information, or other information about the meeting, such as attendees, meeting topics and / or related information context information. In some embodiments, box 905 includes monitoring the meeting to determine a set of meeting contents. In particular, in one embodiment, the meeting monitor can receive an indication of each user input in a chat or other online forum, which indicates a live meeting.

[0127] According to block 915, a set of action item candidates is determined (e.g., by action item determiner 260). Embodiments of block 915 determine a set of candidate action item candidates based on the meeting content information determined in block 905. For example, based on the conversation and context data of the meeting, a set of possible action items may be determined. Some embodiments of step 915 may be combined with action item candidate classifier 264 ( Figure 2 In some embodiments of block 530, the content of the meeting can be clarified first, such as in conjunction with Figure 2 In some embodiments of block 530, it is then possible to Figure 2 The action item candidate is identified by the action item candidate identifier 262 and then determined by the action item candidate classifier 264. In some embodiments, additional details of step 520 are combined with Figure 2 915. In some embodiments, the action items determined at block 915 can include action item candidates (e.g., determined by action item determiner 260 and / or enriched action item generator 280). In some embodiments, the determination of action item candidates can be accomplished using logic unit 230 within storage device 225 (e.g., via CNN or other machine learning models).

[0128] According to block 925, rich content for the action item is generated. For example, information such as who is responsible for completing the action item, the due date of the action item, who is the spokesperson for the action item, and what the action item specifically refers to is generated. In some embodiments, block 925 is composed of Figure 2The action item enrichment 282 is performed so that the boundaries of the action items can be determined and / or the clarity scores and associated information can be generated, such as about Figure 2 In some embodiments, generation of enriched content for action items can be accomplished using enrichment logic 239 within storage 225 (eg, via one or more machine learning models).

[0129] According to block 935, an action item context is generated. In some embodiments, the context generated in block 935 is any context data, such as an event context in which the action item is embedded or otherwise associated. For example, referring back to Figure 4 , a certain amount of pre-event context 402 and post-event context 406 can be generated in preparation for display. In some embodiments, the generated action item context occurs via action item context generator 286, as described above with respect to Figure 2 as described.

[0130] According to block 945, enriched action items are assembled (e.g., by enriched action item assembler 288). For example, not only can action items be assembled, but also a certain amount of context (e.g., determined by action item context generator 286), certain boundaries of the action item (e.g., determined by action item boundary generator 283), and / or various missing or unclear information (e.g., determined by action item clarifier 285), such as an owner / spokesperson for the action item, a due date for the action item, etc. For example, the enriched action items can be assembled in a structured or tagged format that includes or is replaced by the data described above (e.g., action item owner / spokesperson).

[0131] According to block 955, rich action items can be provided. In some embodiments, providing the action items includes generating the assembled rich action items according to block 945 on a user interface or other displayed format. For example, block 955 can include information about Figure 4 , Figure 5A , Figure 5B , Figure 5C , Figure 6 , Figure 7 and / or Fig. 8A Generate a similar user interface (or data within a user interface). For example, action items may be provided in a list of action items with appropriately highlighted borders, as well as optional features that allow the user to expand or minimize the amount of visible event context or provide feedback. Figure 2 The presentation component 220 performs the provision of rich action items according to block 955 .

[0132] According to box 965, a particular embodiment updates the rich action item related logic unit. In some embodiments, box 965 occurs based on receiving user feedback. For example, after providing the rich action item according to box 955, the user is able to view the displayed rich action item and determine that it is not actually an action item and / or various aspects of the displayed action item are still unclear (e.g., owner / speaker, due date, etc.). Therefore, the embodiment can receive a user selection indicating this feedback and use the data to update the rich action item. In some embodiments, the update occurs in near real time, so that the information is updated in the background during the same session in which the rich action item was initially provided. In some embodiments, the update occurs in subsequent sessions alternatively or additionally. In some embodiments, the feedback posted by the user is used as training or tuning data by one or more machine learning models (e.g., enrichment logic unit 239 and / or action item identification logic unit 230) to make subsequent predictions more accurate. For example, in some embodiments, the feedback can represent feedback 310 that is provided back Figure 3 The action item prediction model / layer 302 and (one or more) clarity model / layer 308. In some embodiments, block 965 includes Figure 2 The functionality of the feedback processor 270 is described.

[0133] Now go to Fig.10 , according to some embodiments, an exemplary process 1000 for generating a user interface. Process 1000 (and / or any of the functions described herein) can be performed by a processing logic unit including hardware (e.g., circuits, dedicated logic units, programmable logic units, microcode), software (e.g., instructions running on a processor to perform hardware simulation), firmware, or a combination thereof. Although specific blocks described in the present disclosure are referenced in a specific order and in a specific amount, it should be understood that any block can be substantially parallel to any other block or appear before or after any other block. In addition, there may be more (or fewer) blocks than illustrated. For example, in some embodiments, process 1000 does not include blocks 1006 and / or 1008. Any added blocks may include blocks embodying any of the functions described herein. A computer-implemented method, system (including at least one computing device having at least one processor and at least one computer-readable storage medium) and / or a computer storage medium as described herein may perform or be caused to perform process 1000 or any other functions described herein. In some embodiments, some or each of blocks 1004 to 1012 may be automated or otherwise occur in a non-manual manner (eg, without an affirmative user device request), which improves upon the prior art, as described above.

[0134] According to box 1002, (e.g., by conference monitor 250) detects an indication of an event or one or more parts of an event. The event includes natural language content provided (e.g., narrated or typed) by one or more event participants or attendees in a conference event. For example, the event can be a meeting of 3 participants, and a part of the conference event that can be detected is the opening remarks of the first participant. In some embodiments, one or more user devices, sensors, or other devices are queried (e.g., by conference monitor 250) to determine whether a meeting has started or is about to start, which can be an indication of an event. Therefore, when a conference participant begins to participate in a discussion or when the time when the meeting will start is identified, an indication of the event can be detected in near real time. For example, using contextual data from a calendar, text, or other data, it can be determined that the meeting will start at 4 pm. Therefore, the conference monitor 250 can, for example, cause a control signal to be sent to a microphone device near or online at the meeting location to be activated, and a conference event will be detected when the first speech of a conference participant receiving the meeting is detected. In some embodiments, the "indication" of detecting an event includes detecting when to start. Alternatively or additionally, detecting an indication of an event can include detecting at least a portion of the event or a transcript of the event (eg, detecting when a person speaks to start a meeting).

[0135] In some embodiments, when the event is completed, a full transcript or copy of the event can be detected. For example, a device can record the entire meeting event and an administrator can upload the meeting event to a computing device, which causes natural language text to be output (e.g., via voice to text), at which point the transcript can be detected. For illustrative purposes and the examples described below, a full example transcript of a meeting event may be as follows: "...how was your weekend. Great...we went fishing. Looks like Bob is coming. Okay, let's get started, you remember our last meeting we discussed sales figures. We did well this quarter, but we need to do more. John, how is your business unit doing? We're fair, with a profit of X. We definitely need to improve. Set a goal [phone rings, pause] Set a goal of a profit of X for the next quarter. Okay, Jack, did we call the event coordinator? Yes, but I didn't get him. Okay, text him this time. Sometimes he doesn't answer the phone. In fact, I'll text him now. Jake, before I forget, my daughter needs to be picked up from school too. Can you do that? Yes."

[0136] According to box 1004, (for example, by action item candidate classifier 264) a group of action item candidates can be determined. Some embodiments can calculate a group of action item candidates according to multiple character sequences associated with an event. An action item can be a task indicated in a conference event, and the task is requested to be completed to promote a specific goal or purpose associated with a conference event. In some embodiments, box 1004 includes tokenizing the transcript of the conference event via natural language processing to clean up or otherwise provide insights for predicting action items (for example, by conference content clarifier 261). Tokenization can include generating part-of-speech tags for multiple character sequences. For example, using the example transcript above, each word can be parsed and marked using a POS identifier (for example, "how [verb] did you [pronoun] [name] spend the weekend [interrogative pronoun]"). In this way, syntax or other phrase structures can be used to identify action item candidates. For example, the typical syntax of English imperative sentences uses basic verbs without subjects at the beginning of a sentence. Thus, in the example transcript, only sentences or phrases with similar structures might be “Okay, text him this time” and “Set a goal [phone rings, pause] Set a goal to make X amount of profit next quarter”, “Actually, I’ll text him right now. Jake, before I forget, my daughter needs to be picked up from school too. Can you do that…”, which might all be identified as candidates.

[0137] In some embodiments, block 1004 additionally or alternatively includes (e.g., by action item candidate identifier 262) excluding action item candidates that may be completed during the conference event and / or excluding action item candidates that are commands or requests that are not related to the conference event. For example, in the exemplary transcript, "Actually, I'll text him right now" can be excluded because it is likely to be completed during the conference. Similarly, "Can you [pick up my daughter from school]" can also be excluded because it is not related to the conference.

[0138] In some embodiments, block 1004 additionally or alternatively includes excluding language from repeated or redundant action item candidates. For example, using the example transcript above, "Set a goal [phone rings, pause] Set a goal of X profit amount next quarter" can be modified to "The person will try to sell X profit amount next quarter" to remove "Set a goal" because it is said twice. In some embodiments, block 1004 includes processing the character sequence of the transcript through a word embedding vector model or a semantic model. The word embedding vector model changes natural language characters into digital vectors (e.g., about Figure 4). In this way, the vector can be used as an input to a machine learning model. A semantic model (such as Word2Vec) can include a set of action item candidates that are reconstructed by deleting, adding or replacing specific words. The reconstruction indicates clarifying what the action item candidates are. In this way, language is cleaned up or otherwise added, removed or replaced based on semantic context. For example, using the exemplary transcript, "Set target [phone rings, pause] Set target to profit amount of X for next quarter" can be modified to "Personnel will try to sell X profit amount next quarter". This reconstruction can be done in various embodiments for all text of the exemplary transcript described above (or only identified action item candidates) to clean up the text. As described above, in some embodiments, the reconstruction is based on using a natural language processing component to identify POS tags and semantic context, and reconstructs the action item set based at least in part on POS tags and semantic context.

[0139] According to frame 1006, (for example, by action item clarifier 285) generate clarity score for each action item candidate.In certain embodiments, described clarity score at least partially indicates the amount of missing or incomplete information in each action item candidate.In certain embodiments, frame 1006 can alternatively determine the missing or incomplete information associated with each action item candidate.In certain embodiments, frame 1006 includes or before / after is to calculate the boundary of one or more action item candidates, it includes determining the start words of one or more action item candidates and determining the end words of one or more action item candidates.For example, using the exemplary action item candidate from exemplary transcript above, for "Okay, send him a text this time (Okay, give him a text this time)" candidate, the start boundary can be "give (give)", and the end boundary can be "time (time)".This can be used to highlight or otherwise generate the user interface including all characters between the boundary and the boundary, so that the action item candidate can be displayed as "send him a text this time". In this manner, for example, the user interface can include non-highlighted text that does not represent an action item candidate and highlighted text that represents the boundaries (and words between the boundaries) of the action item candidate. Figure 5B , the action item 504 - 2 is highlighted, while the other event context 504 - 1 is not highlighted.

[0140] In some embodiments, the computation of the action item boundaries is based at least in part on running one or more action item candidates through one or more machine learning models. Figure 3 The start / end model / layer 306 operation described, or about Figure 2A similar model of action as described for item clarifier 285 operates.

[0141] In some embodiments, determining missing or incomplete information includes determining that the first event participant responsible for completing the action item candidate is missing from the action item candidate or determining that the first meeting participant responsible for explaining the action item candidate is missing from the action item candidate, so that the modification or supplementation at box 1012 includes providing the name or identifier of the first event participant. For example, in the action item candidate "Send him a text message this time", it may not be clear who said it and who owns it. Therefore, the action item can be supplemented with the name of the person who made the statement and the person who needs to complete the statement. Similar examples and embodiments are about Figure 2 and Figure 4 The action items are enriched as described in 282.

[0142] In some embodiments, determining missing or incomplete information includes determining that the first action item candidate is not directly stated, so that the modification or supplementation at block 1012 includes clarifying what the action item candidate is. For example, using the action item candidate "Text him this time", it may not be clear what information the text should contain. Therefore, the action item candidate can, for example, include the following information "Text him [event coordinator] [ask if the building is available tomorrow]". With respect to clarity model(s) 308, Figure 3 and Action Item Enrichment 282 describe other examples.

[0143] In some embodiments, the clarity score or other clarity determination associated with block 1006 is based at least in part on external context and event context. For example, using the illustration above, the character sequence "event coordinator" in the action item candidate "text him [event coordinator] [ask if the building will be available tomorrow]" may be derived from pre-event context of the exemplary transcript showing that the present discussion involves an event coordinator. In another example, "ask if the building will be available tomorrow" may come from external context of text on a user device, email, social media, or any other channel that is not the exemplary transcript. For example, other examples of block 906 with respect to Figure 4 Give a description.

[0144] According to block 1008, context data for each action item candidate is determined (e.g., by the action item context generator 286). In some embodiments, the context data includes at least a portion of the plurality of character sequences that are not each action item candidate. For example, referring back to Figure 4 , character sequences that are not action item candidates 404 are pre-event context 402 (including character sequence 402 - 1 ) and post-event context 406 (including character sequence 406 - 1 ).

[0145] In some embodiments, the context data includes at least one of an event context and an external context. The external context is not part of a plurality of character sequences (e.g., a meeting transcript, a meeting chat) and is not spoken or written in the meeting event. In other words, the external context includes data derived from outside of the transcript. For example, the external context data can include text from a user device, data from a social media account, data within an email, data in a calendar, and / or data from a file, all of which may not be the direct meeting transcript or actual content stated / entered by the user at the event.

[0146] In some embodiments, the event context can include a portion of the transcript that is not a specific action item candidate. In some embodiments, the event context can include a first set of character sequences before each action item candidate in the transcript and other character sequences after each action item candidate. For example, return to reference Figure 4 , which may include pre-event context 402 and post-event context 406. In this manner, the event context can also include portions of the transcript that are not candidates for a particular action item.

[0147] In some embodiments, the first set of character sequences and the other character sequences are processed in parallel (e.g., substantially simultaneously) to generate a clarity score. For example, a BERT model can be used to perform this operation.

[0148] According to block 1010 , action item candidates are modified or supplemented with data to be displayed based at least in part on the contextual data.

[0149] In some embodiments, this includes clarifying one or more action item candidates by at least supplementary information, removal information or replacement information associated with each action item candidate. For example, "supplementary" information can include using "action item owner: Jack....Action item: text asking whether the building is available" to supplement the action item candidate "send him a text message", so as to indicate who owns the action item and clearly state what the action item is. In some embodiments, removal information can include removing words or other character sequences from the action item candidate, such as removing the word "actually" in the character sequence "actually, I'll send him a text message now", to better indicate what the action item boundary is. In some embodiments, replacement information includes replacing words or character sequences in the action item candidate with other words or character sequences. For example, for the action item candidate "send him a text message", the new replacement can be "send [event coordinator] a text message [asking whether the building is available tomorrow]", wherein "he" is replaced by the event coordinator. Such supplementation, removal or replacement can occur for any feature that requires clarity, such as who stated the action item, the expiration date of the action item, what the action item is really and / or who owns the action item.

[0150] In various embodiments, block 1010 includes modifying or supplementing one or more action item candidates with data that at least partially completes or at least partially clarifies missing or incomplete information within (or outside of or associated with) each action item candidate. Such an example is described above with respect to the "text him" example of supplementation, deletion, or replacement. This will be described with respect to Figure 2 and Figure 4 The action item enrichment 282 of what the phrase "we should do this" corresponds to is further described according to the owner, spokesperson, action item, and data description. In some embodiments, modifying or supplementing one or more action item candidates with data includes determining which conference event participant is responsible for completing the action item candidate or determining which conference event participant is responsible for stating the action item candidate and causing a mark indicating the conference event participant to be displayed (at box 1012). This is also referred to as the owner and spokesperson of the action item, respectively, in this article. In some embodiments, modifying or supplementing one or more action item candidates with data includes determining that there is a lack of an expiration date for the action item candidate, determining the expiration date, and causing a mark indicating the expiration date to be displayed (at box 1012) in the user interface.

[0151] According to box 1012, a user interface is generated (e.g., by presentation component 220) at least in part based on the modification or supplementation, which includes the modified or supplemented data. In some embodiments, box 1012 includes causing the display of supplemented information, information that has not been removed, and / or replaced information within the user interface of the computing device based at least in part on the supplemental information, removal information, or replacement information associated with each action item candidate as described above. For example, a new replacement candidate "Send a text message to [event coordinator] [ask if the building is available tomorrow]" can be generated within the user interface, thereby replacing "he" with "event coordinator" and "he" is therefore not generated in the user interface. In this example, supplemental information of "ask if the building is available tomorrow" is generated in the user interface. For example, the information that has not been removed is "send a text message" and the replacement information is "event coordinator".

[0152] In some embodiments, the amount of event context to be displayed within the user interface can be determined (e.g., by action item context generator 286) based at least in part on the clarity score prior to block 1012. Thus, block 912 can include generating a particular amount of context within the user interface, such as reference to Figure 7 and Figure 6 as described.

[0153] In some embodiments, an indication of a selection on a user device can be received (e.g., by feedback processor 270). The selection indicates that a user of the user device does not understand at least a portion of the action item candidate. Based on the selection, a machine learning model can be trained. In some embodiments, the training includes modifying the clarity score. These embodiments are related to Fig. 8A , 8B , Figure 7 , Figure 5C , Figure 6 The examples shown and Figure 3 The feedback 310 is described.

[0154] In some embodiments, the user interface includes a certain amount of event context to be displayed within the user interface. The event context can include all content within the transcript. These embodiments are related to Fig. 8A , 8B , Figure 7 , Figure 5C , Figure 6 The examples shown and Figure 3 In some embodiments, a component (e.g., context generator 27) can cause an adjustment in the amount of context to be displayed in response to a user selection of an element within the user interface. Figure 7 Elements 702 and 704 of FIG. 5 or elements 516 and 518 of FIG. 5 describe examples of such situations.

[0155] In some embodiments, a component (e.g., presentation component 220) may receive an indication of a selection of an element within a user interface that indicates that only action items are to be displayed. In response, only action items may be displayed. For example, returning to reference Figure 5B In response to selecting user interface element 502, a list of action items is displayed. In some embodiments, a particular portion of the event context is caused to be displayed based at least in part on the clarity score of one or more action items (block 906). For example, if the clarity score is above a threshold, a smaller amount of event context is displayed relative to when the clarity score is below the threshold. This is, for example, with respect to a request for a specific action item. Figure 7 In some embodiments, a component (e.g., feedback processor 270) receives user feedback indicating annotations or other information to provide more clarity to one or more action items. An example of this may be in response to a request to Figure 7 User interface element 718 or Figure 6 The selection of 614 occurs.

[0156] Having described various implementations, an exemplary computing environment suitable for implementing embodiments of the present disclosure is now described. Fig.11 , an exemplary computing device is provided and generally referred to as computing device 1100. Computing device 1100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the present disclosure. Neither should computing device 1100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.

[0157] Embodiments of the present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-usable or computer-executable instructions (e.g., program modules) executed by a computer or other machine (e.g., a personal data assistant, a smart phone, a tablet PC, or other handheld device). Typically, a program module including routines, programs, objects, components, data structures, etc. refers to code that performs a specific task or implements a specific abstract data type. Embodiments of the present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, and the like. Embodiments of the present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including memory storage devices.

[0158] refer to Fig.11, computing device 1100 includes bus 10 that directly or indirectly couples: memory 12, one or more processors 14, one or more presentation components 16, one or more input / output (I / O) ports 18, one or more I / O components 20, and an illustrative power supply 22. Bus 10 represents what may be one or more buses (e.g., an address bus, a data bus, or a combination thereof). Although Fig.11 The various blocks of FIG. 111 are shown with lines for clarity, but in reality, these blocks represent logical, not necessarily actual components. For example, a presentation component such as a display device may be considered an I / O component. In addition, a processor has memory. The inventors hereby recognize that such is the nature of the art, and reiterate that the diagram of FIG. 111 is merely illustrative of an exemplary computing device that may be used in conjunction with one or more embodiments of the present disclosure. No distinction is made between categories such as "workstation," "server," "laptop," "handheld device," etc., as all are contemplated herein. Fig.11 and reference to “computing device”.

[0159] The computing device 1100 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1100, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (e.g., computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the computing device 1100. Computer storage media do not include the signal itself. Communication media typically embody computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information delivery medium. The term "modulated data signal" refers to a signal that sets or changes one or more of its characteristics in a manner that encodes the information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0160] Memory 12 includes computer storage media in the form of volatile and / or non-volatile memory. Memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. Computing device 1100 includes one or more processors 14 that read data from various entities such as memory 12 or I / O components 20. Presentation component 16 presents data indications to a user or other device. In some implementations, presentation component 220 of system 200 may be embodied as presentation component 16. Other examples of presentation components may include a display device, a speaker, a printing component, a vibration component, and the like.

[0161] I / O port 18 allows computing device 1100 to be logically coupled to other devices, including I / O components 620, some of which may be built-in. Illustrative components include microphones, joysticks, game controllers, satellite antennas, scanners, printers, wireless devices, etc. I / O components 20 may provide a natural user interface (NUI) for processing air gestures, voice, or other physiological inputs generated by a user. In some cases, the input may be transmitted to an appropriate network element for further processing. NUI may implement any combination of voice recognition, touch and stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition associated with a display on computing device 1100. Computing device 1100 may be equipped with a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, and a combination of these, for gesture detection and recognition. In addition, computing device 1100 may be equipped with an accelerometer or gyroscope capable of detecting motion. The output of the accelerometer or gyroscope may be provided to a display of computing device 1100 to present an immersive augmented reality or virtual reality.

[0162] Some embodiments of the computing device 1100 may include one or more radios 24 (or similar wireless communication components). The radio 24 sends and receives radio or wireless communications. The computing device 1100 may be a wireless terminal suitable for receiving communications and media over various wireless networks. The computing device 1100 may communicate via wireless protocols such as code division multiple access (“CDMA”), global system for mobile (“GSM”), or time division multiple access (“TDMA”) to communicate with other devices. Radio communications may be short-range connections, long-range connections, or a combination of both short-range and long-range wireless telecommunication connections. When we refer to “short” and “long” types of connections, we are not referring to the spatial relationship between two devices. Instead, we generally refer to short range and long range as different categories or types of connections (i.e., primary and secondary connections). By way of example and not limitation, a short-range connection may include a connection to a device that provides access to a wireless communication network (e.g., a mobile hotspot). A connection such as a WLAN connection using the 802.11 protocol; a Bluetooth connection to another computing device is a second example of a short-range connection or a near field communication connection. By way of example and not limitation, a long-range connection may include a connection using one or more of CDMA, GPRS, GSM, TDMA, and 802.16 protocols.

[0163] Many different arrangements of the various components depicted, as well as components not shown, are possible without departing from the scope of the following claims. The embodiments of the present disclosure have been described with illustrative, rather than restrictive, intent. Alternative embodiments will become apparent to the reader of the present disclosure after and because of reading the present disclosure. Alternative ways of implementing the foregoing may be accomplished without departing from the scope of the following claims. Certain features and sub-combinations are of practical use and may be employed without reference to other features and sub-combinations and are contemplated to be within the scope of the claims.

[0164] The following embodiments represent exemplary aspects of the concepts contemplated herein. Any of the following embodiments may be combined in a multi-dependency manner to be dependent on one or more other clauses. Furthermore, any combination of dependent embodiments (e.g., clauses that explicitly depend on previous clauses) may be combined while remaining within the scope of the various aspects contemplated herein. The following clauses are exemplary and non-limiting in nature:

[0165] Clause 1. A computerized system comprising: one or more processors; and a computer storage memory having computer executable instructions stored thereon, wherein the computer executable instructions, when executed by the one or more processors, implement a method comprising: detecting an indication of a conference event, the conference event comprising natural language content provided in the conference event by one or more conference participants; calculating or determining one or more action items based on a plurality of character sequences associated with the natural language content, wherein an action item is a task indicated in the conference event and requested to be completed to facilitate a specific goal or purpose associated with the conference event; generating a checklist for each of the one or more action items; A clarity score at least in part indicating an amount of missing or incomplete information within each of the action items; determining contextual data for each of the action items, the contextual data comprising at least a portion of the plurality of character sequences that is not each of the action items; based at least in part on the determination of the contextual data, modifying or supplementing the one or more action items with data that at least partially completes or at least partially clarifies the missing or incomplete information within each of the action items; and based at least in part on the modification or supplementation of the one or more action items with the data, generating a user interface that displays the modified or supplemented data associated with the one or more action items.

[0166] Clause 2. The system of clause 1, wherein the context data comprises external context data that is not part of the plurality of character sequences and is not spoken or written within the conference event.

[0167] Clause 3. According to the system of clause 1, the method also includes calculating boundaries of the one or more action items, wherein the calculation includes determining a start word of the one or more action items and determining an end word of the one or more action items.

[0168] Clause 4. A system according to Clause 1, wherein using the data to modify or supplement the one or more action items includes: determining which conference event participant is responsible for completing the action item or determining which conference event participant is responsible for stating the action item, and generating a mark indicating the conference event participant in the user interface.

[0169] Clause 5. A system according to clause 1, wherein using the data to modify or supplement the one or more action items includes: determining that a due date for an action item is missing, determining the due date, and causing a mark indicating the due date to be displayed within the user interface.

[0170] Clause 6. The system of clause 1, wherein the method further comprises determining an amount of event context to be displayed within the user interface based at least in part on the clarity score, wherein the event context comprises a transcript of the conference event.

[0171] Clause 7. According to the system of Clause 1, the method also includes: receiving an indication of a selection on a user device, wherein the selection indicates that a user of the user device does not understand at least a portion of the action item; and based on the selection, training a machine learning model, wherein the training includes modifying the clarity score for the action item or a subsequent action item.

[0172] Clause 8. A computer-implemented method comprising: detecting an indication of an event, wherein the event includes natural language content provided in the event by one or more event participants; calculating one or more action item candidates in a transcript of a character sequence associated with the event, wherein an action item is a task indicated in the event and requested to be completed to facilitate a specific goal or purpose associated with the event; determining missing or incomplete information associated with each action item candidate; determining context data for each action item candidate, the context data comprising at least one of the following: an event context and an external context, the event context comprising at least a portion of the transcript that is not each action item candidate, the external context comprising data derived from outside the transcript; based at least in part on the determination of the context data, modifying or supplementing the one or more action item candidates using data that at least partially completes or at least partially clarifies the missing or incomplete information within each action item candidate; and causing presentation of the modified or supplemented data based on modifying or supplementing the one or more action item candidates using the data.

[0173] Clause 9. A method according to clause 8, wherein the event context includes a first set of character sequences before each action item candidate within the transcript and other character sequences after each action item candidate, wherein the first set of character sequences and the other character sequences are processed in parallel to generate a clarity score.

[0174] Clause 10. The method according to clause 8 further includes: calculating the boundaries of the one or more action item candidates based at least in part on running the one or more action item candidates through one or more machine learning models.

[0175] Clause 11. A method according to Clause 8, wherein determining missing or incomplete information includes: determining that the first event participant responsible for completing the action item candidate is missing from the action item candidate, or determining that the first event participant responsible for stating the action item candidate is missing from the action item candidate, and wherein the modification or supplementation includes providing the name or identifier of the first event participant.

[0176] Clause 12. The method of clause 8, wherein determining missing or incomplete information comprises determining that a first action item candidate is not directly stated, and wherein the modifying or supplementing comprises clarifying what the action item candidate is.

[0177] Clause 13. The method of clause 8, further comprising causing display within a user interface of a certain amount of event context, the event context including all of the content within the transcript.

[0178] Clause 14. The method of claim 13, further comprising causing adjustment of an amount of event context to be displayed in response to a user selection of an element within the user interface.

[0179] Item 15. One or more computer storage media having computer executable instructions embodied thereon, the computer executable instructions causing the one or more processors to perform a method, the method comprising: detecting an indication of an event, wherein the event comprises natural language content provided in the event by one or more event participants; determining one or more action item candidates from a plurality of character sequences of the natural language content, wherein an action item is a task indicated in the event and requested to be completed to facilitate a specific goal or purpose associated with the event; clarifying the one or more action item candidates at least by supplemental information, removal information, or replacement information associated with each action item candidate; and causing display within a user interface of a computing device, based at least in part on the supplemental information, removal information, or replacement information associated with each action item candidate: supplemental information, information that has not been removed, or replacement information.

[0180] Clause 16. The computer storage medium of clause 15, wherein the clarification of the one or more action item candidates is based at least in part on an external context and an event context.

[0181] Clause 17. A computer storage medium according to clause 15, wherein the clarification of the one or more action item candidates includes highlighting the boundaries of the action item candidates so that the user interface includes non-highlighted text that does not represent the action item candidates and highlighted text that represents the boundaries of the action item candidates.

[0182] Clause 18. The computer storage medium of clause 15, the method further comprising receiving an indication of a selection of an element within the user interface to display only the action item.

[0183] Clause 19. The computer storage medium of Clause 15, the method further comprising causing display of a particular portion of the event context based at least in part on the clarity scores of the one or more action item candidates.

[0184] Clause 20. The computer storage medium of clause 15, the method further comprising receiving user feedback indicating annotations or other information to provide more clarity to the one or more action items.

Claims

1. A computerized system comprising: one or more processors; as well as A computer storage memory having stored thereon computer executable instructions, the computer executable instructions when executed by the one or more processors implementing a method comprising: detecting an indication of a conference event, wherein the conference event includes natural language content provided in the conference event by one or more conference participants; Determining one or more action items based on a plurality of character sequences associated with the natural language content, wherein an action item is a task indicated in the conference event and requested to be completed by one of the one or more conference participants to facilitate a specific goal or purpose associated with the conference event; generating a clarity score for each of the one or more action items, wherein the clarity score indicates, at least in part, an amount of missing or incomplete information within each action item; determining context data for each of the action items, the context data comprising at least a portion of the plurality of character sequences that is not each of the action items; Based at least in part on generating the clarity score and determining the contextual data, modifying or supplementing the one or more action items with data that at least partially completes or at least partially clarifies the missing or incomplete information within each action item; and Based at least in part on the modification or supplementation of the one or more action items using the data, a user interface is generated that displays the modified or supplemented data associated with the one or more action items.

2. The system according to claim 1, wherein: The context data includes external context data that is not part of the plurality of character sequences and is not spoken or written during the conference event.

3. The system of claim 1, the method further comprising calculating boundaries of the one or more action items, wherein: The calculating includes determining a start word for each of the one or more action items and determining an end word for each of the one or more action items.

4. The system according to claim 1, wherein: Modifying or supplementing the one or more action items using data includes determining which conference event participant is responsible for completing the action item or determining which conference event participant is responsible for stating the action item, and generating a mark indicating the conference event participant in the user interface.

5. The system according to claim 1, wherein: Modifying or supplementing the one or more action items with data includes determining that a due date for an action item is missing, determining the due date, and causing display of an indicia indicative of the due date within the user interface.

6. The system of claim 1, wherein the method further comprises: An amount of event context to display within the user interface is determined based at least in part on the clarity score, wherein the event context includes a transcript of the conference event.

7. The system of claim 1, wherein the method further comprises: receiving, at a user device, an indication of a selection, wherein the selection indicates that a user of the user device does not understand at least a portion of the action item; and Based on the selection, a machine learning model is trained, wherein the training includes modifying the clarity score for the action item or a subsequent action item.

8. A computer-implemented method comprising: detecting an indication of an event, wherein the event includes natural language content provided in the event by one or more event participants; computing one or more action item candidates in a transcript of a sequence of characters associated with the event, wherein an action item is a task indicated in the event that is requested by one of the one or more event participants to be completed to further a particular goal or objective associated with the event; Identify missing or incomplete information within each action item candidate; determining context data for each of the action item candidates, the context data comprising at least one of: an event context and an external context, wherein the event context comprises at least a portion of the transcript that is not each of the action item candidates, and wherein the external context comprises data derived from outside of the transcript; Based at least in part on determining the missing or incomplete information and determining the contextual data, modifying or supplementing the one or more action item candidates with data that at least partially completes or at least partially clarifies the missing or incomplete information within each action item candidate; and Based on utilizing the data to modify or supplement the one or more action item candidates, causing presentation of the modified or supplemented data.

9. The method according to claim 8, wherein: The event context includes a first set of character sequences preceding each action item candidate and other character sequences following each action item candidate within the transcript, wherein the first set of character sequences and the other character sequences are processed in parallel to generate an intelligibility score.

10. The method according to claim 8, further comprising: Based at least in part on running the one or more action item candidates through one or more machine learning models, boundaries of the one or more action item candidates are calculated.

11. The method according to claim 8, wherein: Determining the missing or incomplete information includes: determining that the first event participant responsible for completing the action item candidate is missing in the action item candidate, or determining that the first event participant responsible for stating the action item candidate is missing in the action item candidate, and wherein the modification or supplement includes providing the name or identifier of the first event participant.

12. The method according to claim 8, wherein: Determining the missing or incomplete information includes determining that a first action item candidate is not directly stated, and wherein the modifying or supplementing includes clarifying what the action item candidate is.

13. The method according to claim 8, further comprising: Display of a specified amount of event context is caused to be displayed within a user interface, wherein the event context includes all of the content within the transcript.

14. The method according to claim 8, further comprising: The amount of the event context to be displayed is caused to adjust in response to a user selection of an element within the user interface.

15. One or more computer storage media having computer executable instructions embodied thereon, the computer executable instructions, when executed by one or more processors, causing the one or more processors to perform a method comprising: detecting an indication of an event, wherein the event includes natural language content provided in the event by one or more event participants; determining one or more action item candidates from a plurality of character sequences in the natural language content, wherein an action item is a task indicated in the event and requested to be completed by one of the one or more event participants to facilitate a specific goal or purpose associated with the event; clarifying the one or more action item candidates at least by supplementing information, removing information, or replacing information within the one or more item candidates; and Based at least in part on the supplemental information, the removed information, or the replacement information within the one or more action item candidates, causing display within a user interface of a computing device: the supplemental information, the information that has not been removed, or the replacement information.

Images