Automatically determine and present personalized action items from events

Abstract

A computerized system is provided for automatically determining action items for events such as meetings. The determined action items can be personalized for specific users, such as meeting attendees, and can include contextual information that enables the users to understand the action items. Specifically, personalized action items can be determined based in part on: determining and utilizing specific factors in conjunction with event conversations, such as the language style of event speakers; user roles in an organization; historical patterns of communication; event purpose, name, or location; event participants, or other contextual information. Specific statements are evaluated to determine whether the statement is likely to be an action item or not. Contextual information can be determined for action items, which can then be provided to specific users during or after the event.

Description

Background Art

[0001] Various computer-implemented event or meeting technologies are available to assist users in organizing events. For example, some technologies are configured to receive manual input from a user, such as a meeting topic, meeting date, or other notes related to the meeting, and store the input in a computer memory so that the user can review the information in a graphical display format. However, the available stored information and the functionality of these technologies are limited. Typically, such solutions do not automatically determine and present action items, etc. for meetings or other events. Advances in computing technology and the increasing popularity of modern computing devices have caused a dramatic increase in the available data about events. Traditional solutions have not kept pace with these advances, failed to take advantage of the available data, or provide powerful functionality to address these new challenges. Summary of the invention

[0002] This summary is provided to introduce a selection of design concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0003] Aspects of the present disclosure relate to a computerized system for automatically determining whether a statement or speech spoken during an event (e.g., a meeting) is an action item. Manual note-taking or manual computer user input and CPU processing for meetings or other events are very time-consuming, and in many cases, important action items are omitted or not recorded. Various aspects of the present disclosure are determined to mitigate these defects by at least partially based on specific factors, such as the language style of the speaker (or user), the user role in the organization, the historical pattern in the communication, the event purpose, the event name, the event location, the name of the participant in the event, the specific words in the sentence, etc. Action items are determined. In some cases, these factors may include or be used to identify, extract, clarify and / or attribute one or more models of action items or be used in combination with them. In addition, some aspects of the present disclosure can calculate the probability that a specific statement or statement set in an event is an action item or is not an action item.

[0004] Existing technologies (e.g., virtual assistants and conferencing applications) have various functional deficiencies, such as requiring cumbersome and often inaccurate manual user input. Existing technologies are also expensive in terms of computing resource consumption (e.g., memory, network latency, I / O, etc.). Various embodiments of the present disclosure improve upon these technologies via new functionality that is not currently employed by these existing technologies or computing devices, such as automating the determination of action items based on certain factors or rules (e.g., the user's role, meeting type, past meeting history, etc.). Various embodiments of the present disclosure also improve upon existing technologies by reducing computing resource consumption such as memory, network latency, I / O, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Aspects of the present disclosure are described in detail with reference to the accompanying drawings, in which:

[0006] Figure 1 is a block diagram of an exemplary computing environment suitable for implementing some embodiments of the present disclosure;

[0007] Figure 2 is a block diagram illustrating an example action item capture system in which some embodiments of the present disclosure may be used;

[0008] Figure 3 is a block diagram illustrating an example action item capture system in which some embodiments of the present disclosure may be used;

[0009] Figure 4 A schematic diagram depicting an example model flow for determining one or more action items according to an embodiment of the present disclosure;

[0010] Figure 5 Depicted is a process for determining and providing personalized action items from a meeting in accordance with an embodiment of the present disclosure;

[0011] Figure 6 is a flowchart illustrating a method for generating enriched events according to an embodiment of the present disclosure;

[0012] Figure 7 is a block diagram of an exemplary computing environment suitable for implementing embodiments of the present disclosure; and

[0013] Figure 8 An example conference event is illustratively depicted in which personalized action items are automatically determined for conference participants in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] The subject matter of aspects of the present disclosure is described herein with specific details to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the present invention contemplates that the claimed subject matter may also be embodied in other ways in combination with other current or future technologies to include a combination of different steps or steps similar to the steps described in this document. In addition, although the terms "step" and / or "frame" may be used herein to refer to different elements of the method used, these terms should not be interpreted as implying any specific order among or between the various steps disclosed herein, unless and except that the order of the various steps is explicitly described. Each method described herein may include a computing process that can be performed using any combination of hardware, firmware and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. The method may also be embodied as a computer-usable instruction stored on a computer storage medium. The method may be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service) or a plug-in of another product (to name a few).

[0015] Certain aspects of the present disclosure automatically determine personalized action items for events (e.g., meetings, interactive seminars, informal gatherings, etc.). The term "action item" as described herein is used broadly and may, for example, but not limited to, include a task indicated in an event that is required or desired to be completed to facilitate a certain goal or purpose that may be associated with the event. In various cases, an action item may be issued via a command, prompt, or other request issued by a person to cause another person (or themselves) to perform some action. In an illustrative example of an action item, during a meeting about developing a certain computer application, a person may say, "Bob, can you perform a round of debugging on the app today?" This is an action item for Bob to perform debugging actions today to get the application ready for deployment.

[0016] At a high level, Figure 8 An illustrative example use case of an embodiment of the technology described herein is shown in FIG. This example embodiment is used to determine personalized action items for conference participants. Specifically, Figure 8Two scenes 810 and 850 of a meeting are depicted. The meeting includes meeting attendees who are physically present at the meeting (e.g., attendees 807 and 809) and meeting attendees who are present online (e.g., attendee 830), which are displayed on a computer display 820. In the first scene 810, manager 805 is shown saying in statement 815 that the meeting has concluded and that many important details were covered. (Also, it can be inferred from statement 815 that the meeting was long because it lasted all afternoon.) Manager 805 further states in statement 815, "now everyone should know what they need to do, right?" In response to this statement by manager 805, the other meeting attendees in scene 810 appear worried because they do not know what they need to do. For example, attendee 807 displays the thought “I have no idea what I'm supposed to do” in thought bubble 817, and attendee 809 displays the thought “I hope someone else got all of that because I sure didn't” in thought bubble 819.

[0017] Typically, these meeting attendees may be in trouble, or even at risk of losing their jobs, if they miss important action items in a lengthy discussion with manager 805. The attendees may have taken some notes, but it is often difficult or impossible to actively participate in the discussion while trying to capture important information (e.g., action items). Using traditional meeting recording technology, such as recording or dictation devices, attendees 807 and 809 may be able to go back and listen to the meeting in order to try to figure out the action items. But as further described herein, this task is cumbersome and may still result in missing or erroneous action items because it does not take into account other important information, such as information about the speaker of the action item, historical information from previous meetings, or other relevant contextual information.

[0018] Now turn to Figure 8 850, depicting some aspects of an example embodiment. Specifically, the example embodiment is implemented using a virtual assistant, such as Company Assistant, which is integrated with conferencing or communication applications (such as Microsoft ) combined operation. As shown in scene 850, computer display 820 depicts a representation of virtual assistant 860, such as the Cortana virtual assistant. As described herein, based on the meeting discussion and relevant contextual information, virtual assistant 860 has automatically determined personalized action items for each meeting attendee. Virtual assistant 860 then says in sentence 865 "I have sent each meeting attendee their personal action items." As described herein, each meeting participant (or any individual responsible for the action items) can be provided with the action items from the meeting that they are responsible for or that are relevant to them.

[0019] In response to statement 865, the meeting attendees (e.g., attendees 807 and 809) appear relieved and less worried. In addition, attendee 809 holds user device 859, which is displaying a list of her action items received from virtual assistant 860. In this way and as further described below, embodiments of the technology described herein improve upon traditional meeting-related technology. (Although this example depicts providing action items to meeting attendees, in some cases, as further described herein, instructions for action items may be provided to users who are not present at the meeting but are responsible for the action items.)

[0020] like Figure 8 As described in the example of , manual note-taking (or manual computer user input and CPU processing for meetings or other events) is tedious, time consuming, and in many cases, important action items and their context are missed. Action items may be missed for a variety of reasons, such as lack of focus, confusion about what the action item is, interruptions, illogicality, or changing topics during a meeting discussion (e.g., while discussing details about an action item, a meeting may be interrupted or unrelated discussions may occur for a moment or extended interval before the meeting discussion returns to continue discussing the details), and miscommunication. There are many different ways that people talk and interact with others, which may result in action items not being identified.

[0021] Some aspects of the present disclosure mitigate these problems by determining action items based at least in part on certain factors, such as the speaker's or user's language style, the user's role in the organization, historical patterns of communication (e.g., instant chat, email, documents), event purpose, event name, event location, event participants, etc. In some embodiments, one or more of these factors may be utilized in addition to the speaker's utterances in order to determine possible action items. In this way, embodiments of the technology described herein improve upon conventional techniques by being able to determine with a higher degree of certainty or probability whether a statement or set of statements is or is not an action item.

[0022] For example, and as further described herein, some embodiments of the present disclosure may detect (e.g., via a speech recognition function) who is speaking at an event, identify a user profile for that person (which includes the person's role and other history), identify meeting attributes, and generate a set of one or more possible action items based on that information. For example, assume that a particular user regularly starts meetings with the statement "Ok let's get this started." Embodiments of the technology described herein may identify the user who said the statement from the user's speech, and then, for example, use the user's profile to determine that the particular user says the statement at the beginning of almost every meeting, indicating the start of a meeting, rather than an action item to be taken. (It is also conceivable that, regardless of who the speaker is, some embodiments may identify the statement as an indication to start a meeting, rather than an action item to be taken.) Therefore, the embodiment may be selectively marked or otherwise indicated that the statement is not an action item. (Or, as further described herein, it may be determined that the statement is less likely to be an action item.)

[0023] In another example, suppose that during a meeting, a particular participant says, "we need to remove all outliers before conducting the analysis." In this case, an embodiment may identify the speaker, determine that the speaker is a management software engineer, that the meeting involved clustering machine learning model techniques, and that similar statements in similar contexts were previously considered action items in past meetings. Thus, the embodiment may determine that the user's statement is an action item based on this information about the speaker, the meeting, and other context. Alternatively, for example, if it is determined that for this same statement, the speaker and those participants in the meeting are sales representatives, and the context of their discussion is humorous, then the same statement may be considered not to be an action item (or may receive a lower probability of being an action item).

[0024] As mentioned above, the conventional techniques that can be used to determine the action items in a meeting have various technical and functional defects. For example, conventional techniques can be used to record meetings or capture manual user input, such as a user taking notes on a word processor application. However, these techniques lack the function of automatically determining action items based on meeting discussions, and more specifically, specific personalized action items for specific users. In addition, although speech recognition technology (or speech to text technology) can be used to capture the transcription of the meeting, and then analyze to identify keywords (such as commands) to try to determine the action items, this method can still be prone to error. Specifically, as shown in the examples described herein, users often say sentences that sound like action items but are not. For example, sentences like "let's get this started (let's get started)" or "I need to leave now topick up my kids (I need to leave now to pick up my children)" can be mistakenly classified as action items under this conventional method.

[0025] Various embodiments of the present disclosure address these problems and therefore improve upon these prior arts via new functionality that is not currently employed by these prior arts or computing devices. For example, some embodiments improve upon existing software technology by providing functionality that automatically determines action items via certain logic or rules (e.g., the user's role, meeting type, past meeting history, user behavior history, user preferences, event location, etc.). As described above, in various prior arts, tasks must be performed manually and historically have only been performed by humans or manual input by users. In specific embodiments, prior art processes are improved by providing new functionality that effectively performs these tasks automatically and more accurately in conjunction with these specific logic and rules, which will be described in more detail below.

[0026] In another example of a functional deficiency, some technologies, such as virtual assistants (e.g., smart speakers), are unable to automatically determine action items. While these technologies can generally process natural language requests and perform keyword spotting, these technologies are generally unable to parse meeting transcriptions or other data sets to determine which portions of the data set are action items for an event and which portions of the same data set are not action items. Various embodiments improve these virtual assistants because they can parse meeting transcriptions or audio input (e.g., in near real time) to determine what input is an action item.

[0027] The prior art is also inefficient in terms of computer resource consumption (e.g., CPU, memory, I / O, network latency). For example, the prior art requires that the entire transcription of the event be sent to a user device over a network or stored in a memory device so that the user can manually parse out the relevant information within the data. Such a transcription may include inputs such as the names of meeting participants, meeting times, meeting names, and all meeting content. This may increase storage device I / O (e.g., excessive physical read / write head movement on a non-volatile disk) or network latency because each time a user enters this data, the component must repeatedly contact the storage device to perform the read / write operation, which is time-consuming and error-prone, and will eventually wear out components such as read / write heads. In addition, in session-based or network-based Web applications, each user input requires a packet generation cost (e.g., input header information) of a network protocol (e.g., TCP / IP), which increases network latency each time the user input is sent over the network. In addition, storing or sending the entire transcription of an event may consume an unnecessary amount of memory.

[0028] Various embodiments of the present disclosure further improve these techniques by improving the consumption of computing resources of computing devices. For example, some aspects only send or store action items of an event (and possibly a small amount of metadata, such as a meeting name). For example, after determining the action items, specific embodiments may send the action items (rather than the entire meeting transcription) to a user device of a user who is the "owner" or the party responsible for executing the action items via a computer network. In this way, packet generation costs are reduced because only action items are sent over the network, which in some cases allows more bits to be sent over the network. In another example, a user interface may be provided that includes only action items or only records action items, rather than the entire transcription of the meeting. In this way, memory storage is optimized and there are fewer I / O operations.

[0029] Now go to Figure 1 , a block diagram of an example operating environment 100 is provided that illustrates some embodiments of the present disclosure in which it can be used. It should be understood that this arrangement and other arrangements described herein are set forth as examples only. Other arrangements and units (e.g., machines, interfaces, functions, sequences, and functional groups) may be used in addition to or in place of these illustrated arrangements and units, and some units may be omitted entirely for clarity. In addition, many of the units described herein are functional entities that may be implemented as discrete or distributed components, or in combination with other components, and in any suitable combination and position. The various functions described herein as being performed by one or more entities may be performed by hardware, firmware, and / or software. For example, some functions may be performed by a processor that runs instructions stored in a memory.

[0030] Among other components not shown, the example operating environment 100 includes a plurality of user devices (e.g., user devices 102a and 102b to 102n), a plurality of data sources (e.g., data sources 104a and 104b to 104n), a server 106, sensors 103a and 107, and a network 110. It should be understood that: Figure 1 The operating environment 100 shown in FIG. 1 is an example of a suitable operating environment. Figure 1 Each component shown in the figure may be implemented via any type of computing device, such as a combination of Figure 7 The computing device 700 described herein can communicate with each other via a network 110, and the networking device can include, but is not limited to, one or more local area networks (LANs) and / or wide area networks (WANs). In an exemplary implementation, the network 110 includes the Internet and / or a cellular network, as well as any of a variety of possible public and / or private networks.

[0031] It should be understood that within the scope of the present disclosure, any number of user devices, servers, and data sources may be used within the operating environment 100. Each may include a single device or multiple devices cooperating in a distributed environment. For example, the server 106 may be provided via multiple devices arranged in a distributed environment that together provide the functionality described herein. In addition, other components not shown may also be included within the distributed environment.

[0032] The user devices 102a and 102b to 102n may be client devices on the client side of the operating environment 100, while the server 106 may be on the server side of the operating environment 100. The server 106 may include server-side software designed to work together with the client-side software on the user devices 102a and 102b to 102n to implement any combination of features and functions discussed in the present disclosure. This division of the operating environment 100 is provided to illustrate one example of a suitable environment, and for each implementation, it is not required that the server 106 and any combination of the user devices 102a and 102b to 102n remain as separate entities.

[0033] User devices 102a and 102b to 102n may include any type of computing device that can be used by a user. For example, in one embodiment, user devices 102a to 102n may be Figure 7Type of computing device described. By way of example and not limitation, a user device may be embodied as a personal computer (PC), a laptop computer, a mobile device or mobile device, a smart phone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), a music player or MP3 player, a global positioning system (GPS) or device, a video player, a handheld communication device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a camera, a remote control, a barcode scanner, a computerized measurement device, a household appliance, a consumer electronic device, a workstation, or any combination of these devices, or any other suitable computing device.

[0034] Data sources 104a and 104b to 104n may include data sources and / or data systems configured to make data available to any of the various components of operating environment 100, or in combination with Figure 2 (For example, in one embodiment, one or more data sources 104a to 104n provide Figure 2 The user data collection component 210 provides user data (or makes user data accessible). The data sources 104a and 104b to 104n may be separate from the user devices 102a and 102b to 102n and the server 106, or may be combined and / or integrated into at least one of those components. In one embodiment, one or more of the data sources 104a to 104n include one or more sensors 103a, 107, which may be integrated into or associated with one or more of the user devices 102a, 102b or 102n or the server 106. Figure 2 The user data collection component 210 also depicts examples of sensed user data that may be obtained by the data sources 104a - 104n .

[0035] The operating environment 100 may be used to implement Figure 2 One or more components of the action item capture system 200 described in the specification include components for automatic action item extraction. The operating environment 100 may also be used to implement Figure 6 The various aspects of the process flow 600 described in Figure 2 , combined with Figure 1, a block diagram is provided showing aspects of an example computing system architecture suitable for implementing an embodiment of the present disclosure and generally designated as an action item capture system 200. The action item capture system 200 represents only one example of a suitable computing system architecture. Other arrangements and elements may be used in addition to or in place of those shown (e.g., those shown in system 300), and some elements may be omitted entirely for clarity. In addition, with respect to operating environment 100, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or implemented in conjunction with other components, and implemented in any suitable combination and position.

[0036] The example system 200 includes a combination of Figure 1 The network 110 described above and communicatively couples the components of the system 200, including the user data collection component 210, the meeting monitor 250, the action item generator 260, the action item assembler 269, the user profile 240, the user data collection component 210, and the presentation component 220. These components may be embodied, for example, in one or more computer systems (e.g., such as in conjunction with Figure 7 A collection of compiled computer instructions or functions, program modules, computer software services or processes executed on the computing device 700).

[0037] In one implementation, the functions performed by the components of system 200 are associated with one or more personal assistant applications, services, or routines. Specifically, such applications, services, or routines may be operated on one or more user devices (e.g., user device 102a), servers (e.g., server 106), may be distributed across one or more user devices and servers, or may be implemented in the cloud. In addition, in some embodiments, these components of system 200 may be distributed throughout a network, including one or more servers (e.g., server 106) and client devices (e.g., user device 102a) in the cloud, or may reside on a user device (e.g., user device 102a). In addition, these components, the functions performed by these components, or the services performed by these components may be implemented at a suitable abstraction layer (e.g., an operating system layer, an application layer, a hardware layer of a computing system). Alternatively or additionally, the functions and / or embodiments of these components described herein may be performed at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs). Additionally, although functionality is described herein for specific components shown in the example system 200, it is contemplated that, in some embodiments, functionality of these components may be shared or distributed across other components.

[0038] continue Figure 2 , the user data collection component 210 is generally responsible for collecting data from one or more data sources (e.g. Figure 1The user data may be accessed or received (and in some cases identified) from data sources 104a and 104b to 104n of the system 200. In some embodiments, the user data collection component 210 may be used to facilitate the accumulation of user data for a particular user (or, in some cases, multiple users including crowdsourced data) for use with the meeting monitor 250 or the action item generator 260. The data may be received (or accessed) by the user data collection component 210 and optionally accumulated, reformatted, and / or combined and stored in one or more data stores (e.g., storage 225) where it may be available to other components of the system 200. For example, as described herein, the user data may be stored in or associated with the user profile 240. In some embodiments, any personally identifiable data (i.e., user data that specifically identifies a particular user) is either not uploaded or otherwise provided from one or more data sources with the user data, is not permanently stored, and / or is not available to components or subcomponents of the system 200. In some embodiments, users may opt in or out of services provided by the techniques described herein and / or select which user data and / or which resources of user data are to be utilized by these techniques.

[0039] User data may be received from a variety of sources, where the data may be available in a variety of formats. For example, in some embodiments, user data received via the user data collection component 210 may be determined via one or more sensors, which may be on or associated with one or more user devices (e.g., user device 102a), servers (e.g., server 106), and / or other computing devices. As used herein, a sensor may include functionality, routines, components, or combinations thereof for sensing, detecting, or otherwise obtaining information (e.g., user data) from a data source 104a, and may be embodied as hardware, software, or both. By way of example and not limitation, user data may include: data sensed or determined from one or more sensors (referred to herein as sensor data), such as location information of a user device, attributes or characteristics of a user device (such as device status, billing data, date / time, or other information derived from a user device (such as a mobile device)), user activity information (such as: application usage; online activity; searches; voice data, such as automatic voice recognition; activity logs; communication data, including calls, texts, instant messages, and emails; website posts; other user data associated with communication events), including in some embodiments user activity occurring on more than one user device, user history, session logs, application data, contact data, calendar and schedule data, notification data, social network data, news (including popular or trending items on search engines or social networks), online gaming data, e-commerce activity (including from websites such as Amazon. Video streaming services, gaming services, or Xbox data from an online account associated with a personal assistant application or service), user account data (which may include data from user preferences or settings associated with a personal assistant application or service), home sensor data, appliance data, GPS data, vehicle signal data, traffic data, weather data (including forecasts), wearable device data, other user device data (which may include device settings, profiles, network-related information (e.g., network name or ID, domain information, workgroup information, connection data, Wi-Fi network data, or configuration data, data about the model, firmware, or device, device pairing, such as where a user pairs a mobile phone with a Bluetooth headset, or other network-related information)), gyroscope data, accelerometer data, payment or credit card usage data (which may include device settings, profiles, network-related information (e.g., network name or ID, domain information, workgroup information, connection data, Wi-Fi network data, or configuration data, data about the model, firmware, or device, device pairing, such as where a user pairs a mobile phone with a Bluetooth headset, or other network-related information)), This may include information from a user's PayPal account), purchase history data (e.g., information from a user's XboxLive, Amazon.com, or eBay account), other sensor data that may be sensed or otherwise detected by a sensor (or other detector) component, including data derived from a sensor component associated with the user (including location, motion, orientation, positioning, user access, user activity, network access, user device billing, or other data that may be provided by one or more sensor components), data derived based on other data (e.g., location data that may be derived from Wi-Fi, cellular network, or IP address data), and virtually any other data source that may be sensed or determined as described herein.

[0040] The user data may be received by the user data collection component 210 from one or more sensors and / or computing devices associated with the user. Although it is contemplated that the user data may be processed, for example, by a sensor or other component not shown, so as to be interpreted by the user data collection component 210, the embodiments described herein do not limit the user data to processed data, but may include raw data. In some embodiments, the user data collection component 210 or other components of the system 200 may determine explanatory data from the received user data. The explanatory data corresponds to data used by the components of the system 200 to interpret the user data. For example, the explanatory data may be used to provide context for the user data, which may support determinations or inferences made by components or subcomponents of the system 200, such as venue information from a location, a text corpus from a user's speech (i.e., speech to text), or various aspects of spoken language understanding. In addition, it is contemplated that, for some embodiments, components or subcomponents of the system 200 may use user data and / or a combination of user data and explanatory data to achieve the goals of the subcomponents described herein.

[0041] In some aspects, user data may be provided in a user data stream or signal. A "user signal" may be a feed or stream of user data from a corresponding data source. For example, a user signal may come from a smartphone, a home sensor device, a smart speaker, a GPS device (e.g., for location coordinates), a vehicle sensor device, a wearable device, a user device, a gyroscope sensor, an accelerometer sensor, a calendar service, an email account, a credit card account, or other data source. In some embodiments, the user data collection component 210 receives or accesses data related to the user continuously, periodically, when the data becomes available, or when needed.

[0042] continue Figure 2 , the example system 200 includes a meeting monitor 250. The meeting monitor 250 is generally responsible for determining and / or detecting meeting features from online meetings and / or in-person meetings, and making the meeting features available to other components of the system 200. For example, such monitored activities can be meeting locations (e.g., determined by the geographic location of the user device), the subject of the meeting, the invitees of the meeting, whether the meeting is repeated, relevant deadlines, projects, etc. In some aspects, the meeting monitor 250 determines and provides a set of meeting features (as described below) for a specific meeting and for each user associated with the meeting. In some aspects, the meeting can be a past (or historical) meeting or a current meeting. In addition, it should be understood that the meeting monitor 250 can be responsible for monitoring any number of meetings, for example, each online meeting associated with the system 200. Therefore, features corresponding to online meetings determined by the meeting monitor 250 can be used to analyze multiple meetings and determine corresponding patterns (e.g., by action item recognition logic 230).

[0043] In some embodiments, the input to conference monitor 250 is sensor data and / or user device data of one or more users at the event and / or contextual information from the conference invitation and / or email or other device activity of the users at the conference. In some embodiments, this includes user data collected by user data collection component 210 (which is accessible via user profile 240).

[0044] The meeting activity monitor 252 monitors user activity via one or more sensors (e.g., microphone, video), devices, chats, presented content, etc. In some embodiments, the meeting activity monitor 252 outputs a transcription or activity that occurs during the meeting. For example, the activity or content can be timestamped or otherwise associated with the meeting transcription. In an illustrative example, the meeting activity monitor 252 can indicate the clock time when the meeting starts and ends. In some embodiments, the meeting activity monitor 252 monitors user activity information from multiple user devices associated with the user and / or from a cloud-based service associated with the user (e.g., email, calendar, social media, or similar information source), and it may include contextual information associated with the transcription or content of the event. For example, an email may detail a conversation between two participants that provides context for the meeting transcription by describing the details of the meeting (e.g., the purpose of the meeting). The meeting activity monitor 252 can determine current or near real-time user activity information, and in some embodiments can also determine historical user activity information, which can be determined based on collecting observations of user activity over time and / or accessing user logs of past activity (e.g., browsing history). Additionally, in some embodiments, the meeting activity monitor can determine user activity (which can include historical activity) from other similar users (ie, crowdsourcing).

[0045] In an embodiment using contextual information related to a user device, the user device may be identified by the conference activity monitor 252 by detecting and analyzing characteristics of the user device, such as device hardware, software (e.g., OS), network-related characteristics, user accounts accessed through the device, and similar characteristics. For example, as previously described, information about the user device may be determined using functions of many operating systems to provide information about hardware, OS version, network connection information, installed applications, and the like. In some embodiments, a device name or identification (device ID) may be determined for each device associated with the user. This information about the identified user device associated with the user may be stored in a user profile associated with the user, such as in a user account and device 244 of a user profile 240. In an embodiment, the user device may be polled, queried, or otherwise analyzed to determine contextual information about the device. This information may be used to determine a label or identification (e.g., device ID) of the device so that user activity on one user device may be identified and distinguished from user activity on another user device. Additionally, as previously described, in some embodiments, a user may declare or register a user device, such as by logging into an account via the device, installing an application on the device, connecting to an online service that queries the device, or otherwise providing information about the device to an application or service. In some embodiments, identifying and determining to log into an account associated with the user (e.g., Account or Net Passport, email account, social network, etc.) is associated with the user.

[0046] User profile 240 may include information learned about conference participants or attendees, personal preferences of conference attendees, etc. In some embodiments, user profile 240 includes Figure 3 All or part of the information contained in the user profile 340 of the user profile 240, as described in more detail below. In an illustrative example, the user profile 240 may include a user pattern / model 242. This may include an indication of when an attendee or speaker tends to mention action items (e.g., via pattern recognition in previous meetings). For example, a particular attendee may always say an action item within the last 5 minutes of a meeting. This information can be used to make predictions throughout the meeting to determine whether a statement is an action item. The user profile 240 may also include how an attendee or speaker mentions action items. For example, historical meeting events may indicate that a particular user always states "let's mark it" after each action item to ensure that the action item is marked as important in the notes, but otherwise completed.

[0047] User profile 240 may include user preferences 244, which generally include user settings or preferences associated with meeting monitor 250. By way of example and not limitation, such settings may include user preferences regarding specific events (and related information) that the user desires to be explicitly monitored or not monitored, or categories of events to be monitored or not monitored, crowdsourcing preferences, such as whether crowdsourced information is to be used, or whether the user's event information can be shared as crowdsourced data; preferences regarding which event consumers may use the user's event pattern information; and thresholds, and / or notification preferences, as described herein.

[0048] In some embodiments, the conference activity monitor 252 monitors user data associated with a user device and other relevant information on the user device across multiple computing devices (e.g., associated with all participants in a conference) or in the cloud. Information about the user's device can be determined from the user data available via the user data collection component 210, and can be provided to the action item generator 260, as well as other components of the system 200, to predict whether a character sequence or other content is an action item. In some embodiments of the conference activity monitor 252, the user device can be identified by detecting and analyzing the characteristics of the user device, such as device hardware, software (e.g., OS), network-related characteristics, user accounts accessed through the device, and similar characteristics, as described above. For example, many operating system functions can be used to determine information about the user device to provide information about hardware, OS version, network connection information, installed applications, etc. Similarly, some embodiments of the conference activity monitor 252 or its subcomponents can determine the device name or identification (device ID) of each device associated with the user.

[0049] The context information extractor / determinator 254 determines context information about the event. This information can be metadata or other data that is not the actual meeting content or payload itself, but describes relevant information. For example, the context information may include who is attending or invited to the meeting, the subject of the meeting, whether the meeting is repeated, the location of the meeting, the data of the meeting, the relationship between other projects or other meetings, information about the invited or actual attendees of the meeting (e.g., company role, whether the participants are from the same company, etc.). In some embodiments, the context information extractor / determinator 254 determines Figure 3 Some or all of the information within a user profile 340 or event profile 370, as described in more detail below.

[0050] The conference content assembler 256 receives event content and related contextual information and generates an enriched conference activity timeline. In some embodiments, the timeline is a transcription including tags and / or other associated content. For example, the timeline may include structured data (e.g., a database) including records, each of which includes a timeline for each conversation and a timestamp indicating when the conversation started / stopped. The records may alternatively or additionally include contextual information, such as information about event participants or the event itself (e.g., the event's theme, files, slides, or any information in user profile 240). The enriched conference activity timeline may be an output of the conference monitor 250.

[0051] Action item generator 260 identifies possible action items from event content. In some embodiments, input includes output from conference monitor 250 (e.g., user data and conference-related data from sensors (microphone, video, user activity, etc.), user data collection component 210, and user profile 240 from the user. In some embodiments, the output is a list of possible action items and related corresponding information, such as related files, who the action item belongs to or who must complete the action item, date, etc. In some embodiments, the output of action item generator 260 is a structured data record (e.g., a database record) that includes various attributes, such as action item name, attributes (who must complete the action item), action item category / type, related files, and / or content to be provided to remind the user to complete the action item.

[0052] Meeting content clarifier 261 constructs or cleans up meeting content (e.g., transcription and contextual data). For example, sentences can be tagged using part-of-speech (POS) identifiers, and words can be deleted, replaced, and added. In some embodiments, meeting content clarifier 261 and Figure 3 The natural language sequence normalizer 312 of the embodiment of the present invention is the same as or contains similar functions, which will be described in more detail below. In an illustrative example, natural language processing (NLP) techniques can be performed to replace certain words or phrases. For example, the phrase "I am going to expect you to finish project X" can be reconstructed as "person will finish project X". The NLP techniques will be described in more detail below.

[0053] In some embodiments, the action item candidate identifier 262 can determine candidates based on grammatical or other content structure (e.g., whether a sentence is a command). In some embodiments, the action item candidate identifier 262 excludes content or action item candidates based on candidates that may be completed during a meeting or are unrelated private commands or requests or are not part of an event. For example, the sentence "I will finish talking about X in the next slide" can be excluded from action items because it may be completed during a meeting. In some embodiments, the action item candidate identifier 262 and Figure 3 The action item candidate determiner 321 is the same as or contains similar functionality, as described in more detail below.

[0054] In some embodiments, action item candidate identifier 262 uses action item identification logic 230 to identify action item candidate.Action item identification logic 230 may include pattern recognition logic.Pattern recognition logic may include rule, association, condition, prediction and / or classification model, or pattern inference algorithm.Action item identification logic 230 may depend on specific response activity pattern or the mechanism for identifying response activity pattern to take many different forms, or identify the feature similarity between observed response activity events to determine pattern, which corresponds to the task associated with execution and user command.For example, some embodiments of action item identification logic 230 may use machine learning mechanism to determine feature similarity, or use other statistical measurement results to determine the response activity event belonging to the example response action set supporting the determined response activity pattern, as further described below.Event activity may be received from conference monitor 250, and information about the similar features identified may be received from action item candidate identifier 262.

[0055] The action item candidate classifier 264 classifies the action item candidate as an action item or not an action item. Figure 3 The classification component 330 is the same as or includes similar functionality, as described in more detail below. In some embodiments, the action item classifier 264 classifies action item candidates based at least in part on information retrieved by the meeting monitor 250 and / or information in the user profile 240. For example, the action item classifier 264 may receive contextual information from the contextual information extractor / determiner 254 that indicates an expiration date X for action item Y discussed prior to the current meeting. Then during the current meeting, an administrative participant, John Doe, may say, "okay, let's start doing Y beginning tomorrow." The action item classifier 264 may then predict that Y is an action item based on the administrator as the speaker and prior information indicating that Y is an action item and has an expiration date.

[0056] In some embodiments, the action item candidate classifier 264 uses a machine learning model, such as a deep learning classification neural network (e.g., a convolutional neural network (CNN), a recurrent neural network (RNN), or a transformer). In some embodiments, a label, category, or target classification, such as "action item" or "not an action item", may be first identified. These labels, categories, or target classifications may be hard (e.g., membership of a class is a binary "yes" or "no") or soft (e.g., with a probability or possibility attached to the label). Alternatively or additionally, transfer learning may occur. Transfer learning is the concept of reusing a pre-trained model for a new, related problem. For example, the confidence level obtained for detecting an action item may be used to detect a non-action item. The new data set is typically similar to the original data set used for pre-training. Therefore, the same weights can be used to extract features from the new data set. In an illustrative example, the original data set may include statements spoken by conference participants that are classified as action items. It may also be the case that participants say the statement 95% of the time when posting an action item. Therefore, via transfer learning and for a new incoming data set, the user may not say the statement. Using the same weights, it can be inferred that the statement is not an action item.

[0057] In some embodiments, the action item candidate classifier uses logic 235. Logic 235 may include rules, associations, conditions, predictions, and / or classification models, or pattern inference algorithms. Logic 235 may take many different forms depending on a particular response activity pattern or a mechanism for identifying a response activity pattern, or identify feature similarities between observed response activity events to determine a pattern that corresponds to a task associated with the execution and user command. For example, some embodiments of logic 235 may use a machine learning mechanism to determine feature similarities, or use other statistical measurements to determine whether an action item candidate is an action item.

[0058] Action item attributor 266 determines to whom the attribution or action item belongs. For example, action item attributor 266 can determine the person who is responsible for completing the action item and / or the person who speaks the action item. In some embodiments, attribution can be based on the information obtained by conference monitor 250, the information in user profile 240 and / or the classification performed by action item classifier 264. For example, some sentences spoken during a meeting or other event are incomplete or otherwise unclear for the remaining context. Therefore, action item attributor 266 can use this external context information as a signal to attribute the action item. For example, the sentence in the meeting may be "okay go do X starting tomorrow". From this sentence alone, it may be unclear who is talking, and who should perform X from tomorrow. However, the context information obtained by conference monitor 250 from the chat before the meeting can indicate that Jane Doe should perform X and her boss Alexis will tell her when to complete X during the meeting. Therefore, Alexis may be the owner of the action item, and therefore the action item can be attributed to her.

[0059] Action item attributor 266 can map content character sequence to the identity of the speaker or the person who is responsible for completing the action item in any suitable manner.For example, in some embodiments, speech recognition component can be used for audio content input to map the phoneme of input to the known or predetermined phoneme library (for example, found in participant behavior history 346) of specific user.Therefore, speech recognition component can record the voice of each user (for example, each user who attends the meeting potentially) in user profile 240.In this way, the prediction that the character sequence about specific resolution is spoken by a specific user can be made.Alternatively or additionally, mapping or association can be included in the written transcription with the name of the person or other identifier beside each character sequence.In the illustrative example, action item attributor 266 can be with the written transcription of the meeting as input. The sentence "we need to finish the draft by Friday" can be tokenized into individual words, where the subject of the sentence is tagged (e.g., we), the verb of the sentence is tagged (e.g., finish), the object of the sentence is identified (e.g., draft), and the semantic context, such as "action item draft to be finished by Friday" is identified and indicates that manager John said this sentence.

[0060] Action item clarifier 268 performs deduplication of action items, merges action items, and / or verifies consistency between action items. Deduplication of action items can include removing redundant or repeatedly spoken action items. For example, the action item "Get Y done by Friday" may be said twice in a meeting, so one of the statements can be removed. Verifying consistency and merging can include combining different spoken versions of the same office action. For example, the first action item may be "Rex, let's perform project Y". The second action item may say "so yes...let's keeping plugging away to get project Y done." The two action items can be merged into one action item, which is converted in a manner consistent with the two statements, such as "Rex will perform project Y". In some embodiments, the action item clarifier uses deduplication logic 237. In some embodiments, a word embedding model (e.g., WORD2VEC) or other structure can be used to indicate whether the statements are semantically similar (e.g., within the co-signing distance). Therefore, when sentences are semantically similar, they can be combined by linear combination or generating a single sentence (e.g., via NLP) to represent all sentences.

[0061] Action item assembler 269 generates a list of action items and may also provide related information. For example, such related information may be the person to whom it is attributed, the expiration date of the action item, the file, the project name, the timestamps of different conversations, the person who is speaking, the owner of the action item, etc. In some embodiments, the related information may be obtained from the context information obtained by conference monitor 250.

[0062] The example system 200 also includes a presentation component 220, which is generally responsible for presenting content and related information to the user, such as content related to user feedback. The presentation component 220 may be included on a user device, across multiple user devices, or one or more applications or services in the cloud. For example, in one embodiment, the presentation component 220 manages to present content to the user across multiple user devices associated with the user. Based on content logic, device features, associated logical hubs, the user's inferred logical position and / or other user data, the presentation component 220 can determine on which user devices to present content, and the context of presentation, such as how to present (or in what format and how much content to present, which may depend on the user device or context) and / or when to present the context. Specifically, in some embodiments, the presentation component 220 applies content logic to device features, associated logical hubs, inferred logical positions, or sensed user data to determine aspects of content presentation. For example, clarification and / or feedback requests can be presented to the user via the presentation component 220.

[0063] In some embodiments, presentation component 220 generates user interface features associated with clarification and / or feedback requests. These features may include interface elements (e.g., graphical buttons, sliders, menus, audio cues, reminders, alerts, vibrations, pop-ups, notification or status bar items, in-application notifications, or other similar features for interfacing with a user), queries, and prompts.

[0064] In some embodiments, a personal assistant service or application operating in conjunction with presentation component 220 determines when and how to present content (e.g., presenting only action items or presenting action items with metadata, rather than presenting the entire meeting transcript). In such embodiments, the content including the content logic may be understood as a recommendation to presentation component 220 (and / or personal assistant service or application) for when and how to present notifications, which may be overridden by the personal assistant application or presentation component 220.

[0065] Now turn to Figure 3 , the system 300 includes a network 110, combined with Figure 1A network 110 is described that communicatively couples components of the system 300, including a natural language sequence normalizer 312, an action item candidate determiner 321, a classification component 330, a user data collection component 310, a presentation component 320, a user profile 340 (via storage 345), and an event profile 370 (via storage 345). As described herein, user features in the user profile 340 are generated based on different user signals, such as user preferences, user roles (e.g., position within an organization), etc. In addition, signals from the event profile 370 and additional user signals can be combined together to generate combined insights or features. As an example, user historical behavior historical patterns can be considered in conjunction with meetings with specific attendees. The components of the system 300 can be embodied as an arrangement of a set of compiled / interpreted computer instructions or functions, program modules, computer software services, or processes executed on one or more computer systems. In some embodiments, the system 300 is relative to Figure 2 Alternative or additional system embodiments of system 200 .

[0066] It should be understood that Figure 3 The motion item capture system 300 shown in FIG. 1 is an example of a system in which embodiments of the present disclosure may be used. Some or each of the components shown may include the same components as those in the reference photo. Figure 7 The action item capture system 300 may be configured to be similar to one or more computing devices of the type described herein. The action item capture system 300 should not be interpreted as having any dependency or requirement related to any single module / component or combination of modules / components shown therein. Each may include a single device or multiple devices cooperating in a distributed environment. For example, the system 300 may include multiple devices arranged in a distributed environment that jointly provide the functionality described herein. In addition, other components not shown may also be included in the network environment. It should be understood that, according to various embodiments, the system 300 and / or its various components may be located anywhere.

[0067] The action item capture system 300 generally operates to determine specific content in an event that is an action item and to determine which content character sequences are not action items. The system 300 collects, organizes and analyzes data including user preferences, user roles, event locations, event names, etc. to determine whether content is associated with an action item.

[0068] The user data collection component 310 is generally responsible for collecting data from one or more data sources (e.g. Figure 1The data sources 104a and 104b to 104n of the calendar item enrichment system 300 access or receive (and in some cases identify) event and user data. In some embodiments, the user data collection component 310 can be used to facilitate the accumulation of event / meeting data (including crowdsourced data) for all users of the system 300. The data can be received (or accessed) by the user data collection component 310 and optionally accumulated, reformatted and / or combined and stored in one or more data stores (e.g., storage 345), where the data is available to other components of the calendar item enrichment system 300. In addition, the user data collection component 310 can be configured to associate each event data unit with one or more user profiles and store the associated calendar event data unit in the corresponding user profile 340.

[0069] The example system 300 also includes storage 345. Storage 345 generally stores information including data, computer instructions (e.g., software program instructions, routines, or services), and / or models (e.g., machine learning models) used in embodiments of the techniques described herein. In embodiments, storage 345 includes a data store (or computer data storage). Furthermore, while depicted as a single data store component, storage 345 may be implemented as one or more data stores, or may be in the cloud.

[0070] Events and user data can be received from a variety of sources, where the data can be obtained in a variety of formats. For example, in some embodiments, events and user data received via user data collection component 310 can be received via one or more sensors (such as Figure 1The sensor 103a and 107 of the embodiment of the present invention may be determined by sensors 103a and 107 of the present invention, which may be on or associated with one or more user devices (e.g., user device 102a), servers (e.g., server 106), and / or other computing devices. As used herein, a sensor may include functions, routines, components, or combinations thereof for sensing, detecting, or otherwise obtaining information (e.g., events and user data) from a data source 104a, and may be embodied in hardware, software, or both. By way of example and not limitation, events and user data may include: data sensed or determined from one or more sensors (referred to herein as sensor data), such as location information of a mobile device, smartphone data (e.g., phone status, billing data, date / time, or other information obtained from a smartphone), user activity information (e.g., application usage; online activity; searches; voice data, such as automatic voice recognition; activity logs; communication data, including calls, texts, instant messages, and emails; website posts; other user data associated with events, etc.), including user activity occurring on more than one user device, user history, session logs, application data, contact data, calendar and schedule data, notification data, social network data, news (including popular or trending items on search engines or social networks), home sensor data, appliance data, GPS data, vehicle signal data, traffic In some aspects, calendar event data may be provided in a user signal. A user signal may be a subscription to event data from a corresponding data source. In some aspects, calendar event data may be provided in a user signal. A user signal may be a subscription to event data from a corresponding data source. For example, the user signal can come from a smartphone, a home sensor device, a GPS device (e.g., for location coordinates), a vehicle sensor device, a wearable device, a user device, a gyroscope sensor, an accelerometer sensor, a calendar service, an email account, a credit card account, or other data source. In some embodiments, the user data collection component 210 receives or accesses data continuously, periodically, or when needed.

[0071] The natural language sequence normalizer 312 parses or tags event content and / or other external information (e.g., information received by the user data collection component 310) and reconstructs the information. In some embodiments, the event content is or includes a document or transcription of the order and content of everything said in the event written in a natural language. For example, the event content can be a written transcription of everything said during the entire duration of the meeting. In some embodiments, the event content can alternatively or additionally include audio content of everything said during the event. In some embodiments, the natural language sequence normalizer 312 processes the event content in near real time (e.g., as each statement is spoken during the meeting or shortly after each statement is spoken). Alternatively or additionally, in some embodiments, the natural language sequence normalizer 312 processes the event content after each event or meeting ends.

[0072] The parsing and connection component 314 tokenizes and decomposes character sequences (e.g., sentences, words) from the content, and connects other character sequences (e.g., semantically similar words) and / or tags (e.g., parts of speech) to the tokenized character sequences. In some embodiments, the connected character sequences are part of an NLP component or library, where each content is parsed into its words, and some or each word is tagged with a part of speech identifier.

[0073] "Tokenization" or parsing in various embodiments corresponds to a computer-implemented process of segmenting content into words, sentences, symbols, character sequences, and / or other elements of content. "Grammar" or grammatical properties refer to the structure of character sequences of content (as opposed to semantics or meaning), such as the structure of sentences. This can include a set of rules for parsing messages, such as word and / or POS order. For example, for the sentence "the girl jumped happily (girl happily jumped up)", the grammar can correspond to a word order with a structure of subject-verb-adverb (or subject, verb, object, etc.). In various embodiments, the POS of the message elements are tagged.

[0074] In some embodiments, NLP derives the semantic and grammatical content of semi-structured or unstructured data (e.g., data in image files, data in blogs, posts, websites, text descriptions of classifier models). This is in contrast to analyzing "structured" data (e.g., data in databases). NLP can be configured to: parse content to determine semantic context (e.g., the meaning of words determined by analyzing each word in the content in comparison with each other and with training data) and grammatical context (e.g., a set of rules governing sentence structure in a given language). NLP is configured to identify keywords, contextual information, and metadata tags associated with one or more parts of a data set. In some embodiments, NLP analyzes summary information, keywords, and text descriptions contained in a data set, and uses grammatical and semantic elements that appear in the information to identify contexts of interest. Grammatical and semantic elements may include information such as word frequency, word meaning, text font, italics, hyperlinks, proper names, noun phrases, parts of speech (e.g., nouns, adverbs, adjectives, etc.), and / or the context of surrounding words. Other grammatical and semantic elements are also possible.

[0075] The cleaning component 316 removes redundant or repeated character sequences from the content. This takes into account that a conference participant stutters, stammers, mumbles, accidentally says the same thing, or otherwise provides redundant or repeated data. For example, a conference participant may say: "I will... [pause because someone interjects]... Ok I will send them mm file (I will... [pause, because someone interjects]... OK, I will send mm file)". In this example, the cleaning component 316 can remove the "I will" phrase from the character sequence because it is redundant (e.g., it has already been said).

[0076] The sequence structure component 318 replaces, removes and / or adds words or phrases to the character sequence in the content to make the content easier to process. In some embodiments, the component uses the semantic and grammatical analysis described above with respect to NLP. Alternatively or additionally, in some embodiments, the component uses a word embedding model (e.g., WORD2VEC, GloVe, etc.) to determine semantically similar words. In some embodiments, the mapping of the character sequence of the content to the conference participants as described above can also be used to add or replace words via the sequence structure component 318. For example, the sequence structure component 318 can modify the phrase "I'm going to make sure you aren't blocked" to "[person][will] make sure that [person][is not] blocked". Therefore, the sequence structure component 318 can determine that "I'm going to" is semantically similar to "I will", and "aren't" is semantically similar to "is not", and make corresponding replacements. Additionally, the sequence structure component 318 can determine that "person" is speaking or is the subject and / or object of the sentence.

[0077] Action item candidate determiner 321 determines which parts of content are candidates for becoming action items. In certain embodiments, this occurs after the processing of natural language sequence normalizer 312 or is the response to this processing. Candidate identifier 323 identifies the pattern or grammatical relationship in the character sequence in the content to define the boundary or cutoff of the action item. For example, using the above description, after the natural language sequence normalizer determines that the character sequence is "person will make sure that person is not blocked (someone will ensure that someone is not blocked)", it can be identified as a candidate for further processing by the candidate identifier. In certain embodiments, the language or natural interruption in the candidate identifier 323 identification content, such as a period, a semicolon, when to issue a command, etc., is a candidate for further processing to determine that the content between these characters or commands. For example, if the event content has 4 sentences in total, and 3 of the 4 sentences are commands, then the candidate identifier 323 identifies these 3 sentence commands as separate action item candidates for further processing.

[0078] The feature extractor 325 extracts feature values ​​associated with the content in preparation for prediction or classification (e.g., collected by the user data collection component 310). For example, for each action item candidate, the feature extractor 325 can extract who is speaking, all people or participants in the meeting (e.g., event participant data 376), participant roles 344 of all participants in the meeting, event name 372, participant preferences 342 of all participants in the meeting, participant behavior history 346 of all participants in the meeting, event location 374, and external data 378. The participant preferences 342 correspond to the personal preferences of one or more participants and can be used to determine whether the candidate action item is indeed an action item, set an expiration date for the action item, deliver the action item to the user, etc. For example, such personal preferences can be or include: a specific user-selected communication channel (e.g., SMS text, instant chat, email, video, etc.) for sending the action item through it. Additionally or alternatively, personal preferences can include entertainment, hobbies, or other interests. For example, if it is known that a meeting participant likes car manufacturer X with model number Y, any mention of this by the participant in the event can be weighted as less likely to be an action item. In another example, a participant may have specific event preferences, such as a preference that each action item be completed within a time window. Thus, for example, when predicting or sending an action item, additional data (such as the time to complete the action item) may also be coupled with the action item based on the participant preferences 342.

[0079] Participant role 344 corresponds to event participant title or other ID. For example, participant role 344 can be or include one or more positions of a participant, such as software engineer, marketing director, CEO, CIO, management software engineer, deputy general counsel, vice president of internal affairs, etc. In some embodiments, user profile 340 includes the participant role of each participant in the event. Participant role 344 can be used as a signal to help determine whether an action item candidate is indeed an action item. This is because some statements are more likely or less likely to become action items, depending on the role of the participant who said the statement. For example, if a senior engineer or management engineer mentions "we should create a different design for this customer", the statement may be an action item, and if a new employee says the same statement, it is not necessarily an action item, because the typical employment structure requires that action items are either said by an authority or approved by an authority.

[0080] Participant behavior history 346 corresponds to identified patterns of historical behavior of one or more participants of an event. Such patterns may be or include: patterns in the way participants speak (e.g., participant A always jokes even when issuing commands, which makes the candidate less likely to be an action item), a tendency to mention action items at specific times or time windows of an event, a pattern of never owning or publishing any action items, a pattern of always owning or publishing any action items, a pattern of how people express action items or interact with others (e.g., person A, who is a manager, always says "homeworkassignment number[x],..." before an action item, which is a signal that the following sentence is an action item), the language and style of the user, a history of conversations as previous communications between participants (e.g., chats, emails, and documents), and the like. Historical patterns can be used to predict whether a candidate action item is indeed an action item. For example, if the conversation history between two participants shows that the next upcoming meeting Y will be where they decide the location of the upcoming event Z, then the phrase "Ok John, let's have it at Glendale" during meeting Y can be determined to be an action item based at least in part on the conversation history about events and event locations. In some embodiments, the patterns identified as participant behavior history 346 are determined by one or more machine learning models (e.g., Bayesian graphs, random forests, linear regression, etc.).

[0081] In some embodiments, user profile 340 includes alternative or additional information to help determine whether a candidate action item is indeed an action item, such as the participant's age, the participant's family members and related information, the participant's address, etc. For example, if speaker A is known to have a child, Jane Doe, at school, and speaker A says "I have to pick up Jane Doe today," it may be selectively not selected as an action item based on the family member data.

[0082] Event profile 370 corresponds to an event (e.g., a meeting) and associated details (e.g., collected by user data collection component 310). Event name 372 corresponds to the title or topic (or subtopic) of the event. Action items can be determined based at least in part on event name 372. This is because certain phrases or statements are more or less likely to be action items for certain events and associated topics. For example, for a meeting where the topic is the accuracy of a machine learning model, any statement about model details (e.g., providing more test data, reducing error rates, etc.) is more likely to be an action item than, for example, a meeting where the topic is sales strategies based on gestures and other body language habits.

[0083] Event location 374 corresponds to the geographic location or type of event. For example, event location 374 can indicate the physical address of the meeting or the building / room identifier of the meeting location. Event location 374 may alternatively or additionally indicate that the meeting is a virtual or online meeting or an on-site meeting. Event location 374 can also be a signal for determining whether a candidate action item is indeed an action item. This is because certain event locations are associated with certain topics, and based at least in part on the location or topic, the content of the event is unlikely or likely to be considered an action item. For example, if it is determined that the meeting is at building B, and building B is a building where engineering testing occurs, then certain statements are more likely to be action items than other statements (e.g., those statements that are commands for testing, buildings, etc.).

[0084] Event participant data 376 indicates the name or other identifier of the participant of a particular event. In some embodiments, event participant data 376 includes the relationship between users in the event. For example, event participant data 376 may include a graphical view or hierarchical tree structure indicating the highest management position at the top or root, the middle manager at the branch below the manager position, and the senior worker at the leaf level below the middle manager. In some embodiments, the name or other identifier of the event participant is automatically or nearly determined in real time when the user speaks (e.g., based on a speech recognition algorithm), or can be determined based on the manual input of the attendee, invitee, or conference administrator. In some embodiments, in response to determining event participant data 376, system 300 then retrieves or generates a user profile (e.g., user profile 340) for each participant of the event.

[0085] External data 378 corresponds to any other suitable information that is not the actual meeting content itself and can be used to predict whether the content is an action item. In some embodiments, external data 378 includes any non-personalized data that can still be used to make predictions. For example, even if the current participant pool of the current event is different from the participant pool that participated in historical events, external data 378 can also include information learned about human habits at several meetings / events. This information can be obtained via remote resources (such as blogs, social media platforms, or other data sources unrelated to the current event). In an illustrative example, it can be determined over time that for a particular organization or business unit, action events are always listed in the last 10 minutes of the meeting. Therefore, for the last 10 minutes of the current meeting that its particular participant pool has never encountered before, the candidate is likely to be predicted as an action item based on the history of the particular organization or business unit.

[0086] Candidate exclusion component 327 excludes or removes the action item candidate that is marked as action item.In some embodiments, this exclusion is based on the fact that the candidate may be completed during the event or is clearly removed as an action item.For example, if a participant declares "I will explain it in the next slide (I will explain it in the next slide)", this is a signal sent to system 200, indicating that the candidate is not a real action item, because the candidate will be completed at a certain time during the meeting.Or the participant may say, "thanks for all the input everyone...after further thinking, let's not do Y (thank you for all the input...after further thinking, we don't do Y)." Therefore, Y as an action item can be removed.Therefore, the embodiment of system 300 can use a rule set, which indicates that only when the action item candidate to be completed will still occur at a certain time after the end of the meeting, the action item candidate is indicated to be an action item.In some embodiments, this exclusion is based on such a candidate: although they appear to be action items, they are irrelevant or unimportant for a particular event. For example, a meeting participant might say "Jake, can you pick up Robert today?" However, the meeting may be about sales or other unrelated topics, and the speaker (e.g., Dan) may have a son, Robert, who needs to be picked up.

[0087] The classification component 330 classifies or otherwise makes a prediction about whether an action item candidate is an actual action item. Such a component can use any suitable model or set of models or techniques to make predictions, such as a random forest model, a deep neural network, a Bayesian network, or any other suitable machine learning model. In some embodiments where the model is supervised, for example, the label or classification target is whether the candidate is an action item. For example, in a random forest embodiment, each leaf node can indicate whether the candidate is an action item, which is based on decisions at the branch nodes and the root node (e.g., based on values ​​extracted within the user profile 340 and the event profile 370).

[0088] Event type detector 332 detects or classifies written or audio transcriptions according to the type of meeting. For example, event type detector 332 can classify whether an event is a broadcast meeting, a tutorial, a board meeting, etc. (e.g., based on information in event profile 370). Alternatively or additionally, event type detector 332 also calculates the probability that an action item will be present within the content based at least in part on the type of event and / or other information within event profile 370. In some embodiments, the probability is used to determine an additional probability to indicate whether a particular candidate is indeed an action item. For example, if it is determined that the event is a tutorial or a speech, with little feedback or dialogue between the speaker and the attendees, it can be predicted that the probability that the event will contain an action item is low (e.g., a low confidence level). Therefore, anything that is specifically stated or said during the event may be automatically weighted or otherwise scored toward a low probability.

[0089] In some embodiments, the event type detector 332 uses Naive Bayes or other Bayesian network models for prediction. The Naive Bayes algorithm is a set of probabilistic algorithms that use probability theory and Bayes' theorem to predict labels for text. For example, if the event name 372 does not automatically provide the name or topic of the meeting, the event type detector 332 can use the Naive Bayes algorithm to infer or predict the topic or name of the event. In an illustrative example, the statement "let's do more rounds of tracing and debugging" can be labeled "software performance engineering" and the like. Bayesian network graphs map the relationships between nodes (e.g., events) based on probabilities. These graphs show how the occurrence of a particular event affects the probability of other events occurring. Each node is also conditionally independent of its non-descendants. These graphs follow the basic principles of Bayes' theorem, expressed as:

[0090]

[0091] Where A and B are events, and P(B)≠0. That is, the probability of A given B (P) = the probability of B given A multiplied by the probability of (A) over B. The directed acyclic graph includes various nodes, directed edges, and conditional probability tables. For example, each node can represent a feature within the event profile 370, such that the probabilities are interdependent, such as because the probability of an even position at X is high, the probability that the subject or event name 372 of the event is Z is Y, and because of the probability Y, the probability that participants 1, 2, and 3 will attend the meeting is T.

[0092] The event part probability component 334 classifies parts of the content (e.g., action item candidates, paragraphs, sentences, pages, etc.) as specific (the part is more likely to be an action item) or descriptive (the part is less likely to be an action item). The event part probability component 334 therefore parses the content and maps these parts in a memory as being likely or unlikely to be action items (e.g., based on grammatical structure). For example, sentences or phrases without any commands or requests (e.g., sentences that are not imperative sentences), such as "item X was great for our customers" or "we have much fun at the last meeting" or "we made lots of progress since the last meeting," are less likely to be action items. In contrast, sentences or phrases that are imperative sentences or otherwise include commands or requests are more likely to be action items. Imperative sentences are used to issue commands, make requests, or provide suggestions. Imperative sentences typically end with a period or exclamation point or have other grammatical properties, such as a subject, followed by a verb, and then an object (e.g., "John [subject], please join [verb] in on the project [object]"). In some embodiments, the event portion probability component 334 can distinguish between action items and those that appear to be action items but are not action items based on, for example, grammatical properties and / or history (e.g., participant behavior history). For example, the sentence "you can press on it" may be marked as not an action item based on a dictionary or other data structure that lists spoken colloquial terms or phrases rather than action items.

[0093] In some embodiments, the event partial probability component 334 uses a predictive model, such as a hidden Markov model (HMM). HMM is a computational tool for representing probability distributions over observation sequences. These tools model time series data. HMM augments Markov chains. Markov chains are models that provide insight into the probabilities of sequences of random variables or sequences of states, each of which takes values ​​from a data set. These data sets can be text, labels, symbols, etc. The assumption of the Markov chain is that any prediction is based only on the current state, not the state before the current state. The state before the current state has no effect on the future state. In various cases, the events of interest are hidden because they cannot be directly observed. For example, POS tags are not usually observed in the text. Instead, the words themselves are directly observed and the POS tags (hidden states) are inferred.

[0094] HMM allows the model to use both observed events and hidden events, which are essentially causal factors in probabilistic algorithms. HMM is represented by the following components: The set of N states Q = q1q2...q N , the transition probability matrix AA = a 11 ……a ij ……a NN , each a ij represents the probability of moving from state i to state j, A sequence of T observations O = o1o2...o T , each from the vocabulary V = v1, v2, ... v T , the sequence of observation possibilities B = b i (o t ), also called emission probabilities, each represents the probability of observing o t The probability of being generated from state i, and the initial probability distribution over the states π=π1π2...π N π i is the probability that the Markov chain starts in state i. Some states j can have π j = 0, which means they cannot be the initial state.

[0095] The probability of a particular state depends only on the previous state, so the Markov assumption is introduced: P(q i |q1...q i-1 )=P(q i |q i-1 ). Output observation o i The probability of generating an observation q depends only on i The state of , does not depend on any other state or any other observation, thus leading to output independence O(o i |q1...q i ...,qr ,o1,...,o i ,...o T )=P(o i |q i ). This allows the component to state that given an observation o (e.g., a sentence is in the form of a command, a type of meeting, or any other value within event profile 370), the algorithm can find a hidden sequence of Q states (e.g., whether a portion of the content is likely to be an action item).

[0096] In some embodiments, the action item determiner 336 takes as input all calculated features obtained by the feature extractor 325, the results of the action item candidates of the candidate identifier 323 (e.g., determined by the action item candidate determiner 321), and / or other character sequences or contexts, and classifies or otherwise predicts whether each candidate action item is an action item or not an action item. In some embodiments, the "context" described herein is the surrounding text of a specific character sequence. For example, the context of the first sentence can be the second sentence before the first sentence and the third sentence after the first sentence.

[0097] In an illustrative example of how the action item determiner 336 makes predictions, first the natural language sequence normalizer 312 can take the entire transcription as input to reconstruct the transcription as described above. In response, the action item candidate determiner 321 can identify candidate action items based on sentence grammar, exclusion rules, and / or can extract various features from the user profile 340 and the event profile 370 as described above. In response, the classification component 330 can classify the event (e.g., via naive Bayes), determine the probability of whether the part or action item candidate in the transcription is an action item (e.g., via HMM), and then finally classify whether the candidate action item is indeed an action item. For example, the action item determiner 336 can determine whether the candidate is an action item with a certain confidence level based at least in part on the participant behavior history 346, the participant role 344, and the participant preference 342 of each participant in a specific meeting. The action item determiner 336 can additionally or alternatively determine whether the candidate is an action item based at least in part on the event name 372, the event location 374, the event participant data 376, and / or the external data 378.

[0098] In some embodiments, the action item determiner 336 uses a machine learning model, such as a deep learning classification neural network (e.g., CNN). Using CNN, a tag, category, or target classification, such as "action item" or "not an action item" can be first identified. The neural network can include a convolutional layer, a pooling layer, and a fully connected layer. The machine learning model neural network can be fed or received at the convolutional layer by the output provided by the natural language sequence normalizer 312, the action item candidate determiner 321, and / or the event type detector 332 / event partial probability component 334 as input. Each input can be converted into a two-dimensional input vector array of values, such as integers 1 and 0. Each value represents or describes the specific value of the input and the intensity of the value. For example, a certain word or character sequence can be represented by one, while other words or character sequences can be represented by zero. The convolutional layer uses one or more filter maps, each of which represents a feature of the input (e.g., participant role 344). Various features of the input can exist, so various linearly stacked filter maps can exist for a given input. A filter map is also an array of values ​​representing sets of values ​​and weights, where a value is weighted higher when it matches a corresponding value in a corresponding portion of the input (e.g., training data indicating participant behavior history 346) or a set of values. The convolutional layer includes an algorithm that uses each filter map to scan or analyze each portion of the input. Thus, each value of each filter map is compared and matched relative to the corresponding value in each portion of the input and weighted according to the similarity. In some embodiments, the convolutional layer performs a linear function or operation to obtain a filter map by multiplying each value by its own value and then performing a sum function of each product, which is then divided by the total amount of values ​​in the input feature.

[0099] In a particular embodiment, the pooling layer reduces the dimension or compresses each feature map by selecting a window size (i.e., the number of dimensional values ​​to be analyzed in the feature map) and selecting the maximum value of all values ​​in the feature map as the only output of the modified feature map. In some embodiments, the fully connected layer maps or votes for each value of each modified feature to each classification (i.e., whether the candidate is an "action item" or "not an action item"). The strength of the vote for each value is based on its weight or value score. The output is a score (e.g., a floating point value where 1 is a 100% match) that indicates the probability that a given input or set of modified features fits a particular defined classification (is an "action item" or "not an action item"). For example, the input may include a text collection that reads "let's get this meeting started after I tell a joke....Jake tell John to finish the product by Monday". The classification type may be "action item" or "not an action item". After the first character sequence "let's get this meeting started after I tell a joke..." is fed through each layer, the output may include a floating point value score for each corruption classification type indicating "action item: 0.21" and "not an action item: 0.90", indicating that there is a 90% probability that the character sequence is not an action item. Training or tuning may include minimizing a loss function between a target variable or output (e.g., 0.90) and an expected output (e.g., 100%). Thus, it may be desirable to get as close to 100% confidence in a particular classification as possible to reduce prediction error. This may occur over time as more training inputs and baseline datasets are fed into the learning model, so that classifications with higher predicted probabilities may emerge.

[0100] The presentation component 320 is used to present various interfaces and various other outputs generated by the calendar item enrichment system 300 and its components in a format that can be displayed on a user device. For example, the presentation component 320 can present the interface of the action item event (rather than the entire event content) to those appropriate individuals who own the action item.

[0101] Now turn to Figure 4 , Figure 4 FIG. 4 is a schematic diagram showing an example model flow 400 for determining one or more action items according to some embodiments. In some embodiments, the model flow 400 is composed of Figure 2 and / or Figure 3One or more components of the present invention are used. The model stream includes an input layer 404, several intermediate or hidden layers 406, 408 and 410, and an output layer 412 for performing a classification 414 indicating whether a character sequence is an action item. Transcription 402 is a written or audio content corresponding to the entire body of discourse or natural language speech occurring within an event. For example, transcription 402 may include all conversations between participants occurring during a meeting, including questions, commands or action items, pauses, sneezes, coughs, laughs, explanations, other sounds, etc. Stream 400 uses at least a portion of transcription 402 as input, such as a separate character sequence 402-1 (e.g., 3 ordered sentences that appear one after another in time). In some embodiments, layer 404 ingests the entire transcription 402 as input at a single moment. Alternatively or additionally, layer 404 ingests parts of transcription 402 at different moments (e.g., in near real time when each discourse or participant speech occurs). For example, a first set of character sequences 402 - 1 may be ingested at a first time, and a second set of character sequences 402 - 2 may be ingested at a second time after the first time, until all character sequences of the transcription 402 are ingested.

[0102] In some embodiments, the character sequence (e.g., 402-1) includes context before the sentence or sentence fragment and context after the same sentence or sentence fragment, and / or action item candidates (e.g., determined by the action item candidate determiner 321). For example, a portion of the transcription may read "...I'm feeling like we can accomplish task X. So John do task X by Friday. That should give us enough time to accomplish task Y." Therefore, "so John do task X by Friday" may be an action item candidate, while "I'm feeling like we can accomplish task X" and "That should give us enough time to accomplish task Y" are the context of the action item candidate. In some embodiments, input layer 404 (or any of layers 406 , 408 , 410 , and / or 412 ) includes artificial input neurons that may be weighted or unweighted to capture some or every information within transcript 402 .

[0103] Layer 406 receives the output of layer 404 (a character sequence with its context), and identifies (e.g., via action item candidate identifier 262) action item candidates, specific character sequences (e.g., sentences), and backward context (e.g., context before a specific character sequence) and forward context (e.g., context after a specific character sequence). In some embodiments, layer 408 takes this as input and converts the information into vectors or determines the semantic similarity between sentences. In some embodiments, layer 408 is or includes a word embedding vector model, such as WORD2VEC or GloVe. In these embodiments, the input to layer 408 is a natural language text sequence (character sequence, action item candidate, forward and backward context), and layer 408 converts the natural language text into a vector (e.g., a real number) and maps it into a vector space, in preparation for a machine learning model to make a prediction. For example, the WORD2VEC word embedding vector model can be used to map each input to a set of vectors. Then, all vectors representing, for example, part 402-1 are averaged to form a single vector representation in the vector space, which will be described in more detail herein. This way, other machine learning models can use these vectors as input to make predictions.

[0104] In some embodiments, the word embedding vector model is a WORD2VEC model. The WORD2VEC model is a two-layer network model that runs one or more input vectors (e.g., which represent message elements) through a hidden layer (i.e., a column-row matrix) and a projection layer (e.g., a softmax classifier). The WORD2VEC model predicts a target string from a source context word (i.e., via a continuous bag of words (CBOW) algorithm) or reversely predicts a source context word from a target word (i.e., via a skip-gram algorithm).

[0105] In various cases, vectors are mapped in vector space according to their semantic similarity to other vectors representing other texts. In this way, for example, the input can be normalized or otherwise cleaned (e.g., by cleaning component 316) to reconstruct text or action item candidates. In order to draw the input vector in the vector space, the model is trained using training data. In various embodiments, the training data includes a large corpus with unstructured data (e.g., documents, news articles, social media posts, news subscriptions or blogs), semi-structured data and / or structured data (e.g., database values). The training data can also be the input of the word embedding vector model. In some embodiments, the vector space represents a "pre-trained" embedding. The pre-trained embedding is a static model that is generated without feedback, retraining or reference to a data set fed through it. For example, a user can download a static word embedding vector model from an online resource, which has been trained and includes vectors or data points that have been mapped to the vector space according to the semantic similarity between words. In other embodiments, the vector space represents a "retrained" or trained embedding. A retrained or trained word embedding model is an embedding that receives training feedback after receiving an initial training session and is optimized or generated for a particular data set. For example, after mapping initial data points to one or more word embedding vector models, some embodiments may "retrain" one or more word embedding vector models a second time so that any vector or word in the future data set will always map to its nearest neighbor or other words according to the implemented strategy. In some embodiments, retraining includes issuing feedback to ensure that the correct data point pairings are used.

[0106] In order to map each word to a point in the vector space that is appropriate for its context, a training algorithm is used. For example, in some embodiments, a word embedding vector model is trained using the maximum likelihood (ML) principle to make the next word w given the previous word h (i.e., "history") according to a softmax function. t (i.e., the "target") probability is maximized:

[0107]

[0108] Among them, score(w t ,h) Calculate word w t compatibility with the context h. The model is trained by maximizing its log-likelihood on the training set, i.e. maximizing .

[0109]

[0110] This produces a properly normalized probability model for language modeling. In each training step, each probability is calculated and normalized using the scores for all other words w' in the current context h. In some embodiments, some models (e.g., WORD2VEC) are trained using a binary classification objective (e.g., logistic regression) to distinguish the true target word w in the same context. t and K noise words w". Therefore, instead of the softmax classifier, a noise classifier is used.

[0111] In some word embedding embodiments, the output of the training algorithm and / or the actual data input is a word at each position in the vector space that shows groups of semantically similar words. "Semantic similarity" is the semantic distance between two or more concepts (e.g., full forms of words). In some embodiments, the "distance" between any two or more words is based on the similarity of their meaning and / or semantic content, rather than any grammatical similarity. For example, "car" and "far" are grammatically similar, but have two different definitions, so they are not semantically similar.

[0112] In some embodiments, layer 408 includes a one-hot encoding layer and a coordinate layer to represent and map words in the vector space. For example, the word "meeting" can be converted to a one-hot encoding vector representation [1,0,0,0,0]. The vector representation shows five values, each of which corresponds to an ordered input element in the input and whether the meeting element is true or exists. Because "meeting" is a word run through a word embedding vector model, the integer 1 is used to indicate its representation. "Meeting" does not contain any other words, so other vectors are represented as 0. The output embedding vector representation can then be a coordinate or other value indicating where the word should be plotted in the vector space (e.g., [1,2]). In some embodiments, the coordinate mapping is plotted in the vector space based on semantic similarity with other words and / or averaging or otherwise combining the output embedding vectors of all words in the input.

[0113] The distance between any two vectors or words is measured according to any suitable method. For example, in some embodiments, the distance is calculated using automatic cosine similarity. Cosine similarity is a measure of the similarity between two non-zero vectors in the inner product space, which is used to measure the cosine of the angle between the two non-zero vectors. No similarity is represented as a 90-degree angle, while a total similarity of 1 (i.e., the same word) is a 0-degree angle. For example, a distance of 0.98 between two words reflects a very high semantic similarity, while a distance of 0.003 reflects a very small semantic similarity. In another example, the cosine similarity between "man" and "king" and "woman" and "queen" is the same cosine distance, so in some cases, considering the input of man and woman, king is semantically similar to queen.

[0114] After the input passes through layer 408 (e.g., is converted to a vector and mapped to a vector space), layer 410 filters or weights the input it receives. For example, layer 410 can be or include a convolutional layer and a rectified linear unit (Relu) of a CNN. Relu is an activation function that can be used by certain machine learning models (e.g., neural networks, specific CNNs). In some embodiments, Relu is represented as y=max(0,x).

[0115] In some embodiments, event type detector 432 and event partial probability component 334 can use layer 412 to make predictions. In some embodiments, event type detector 432 is associated with Figure 3 The event type detector 332 is the same component and / or is included in Figure 2 4. For example, component 432 may use the naive Bayes model described above to classify a particular event. In some embodiments, event partial probability component 434 is associated with Figure 3 The event portion probability component 334 is the same component and / or is included in Figure 2 The action item classifier 264 of FIG. 264. For example, the component can map in memory the parts that may or may not be action items.

[0116] In some embodiments, layer 412 is the final prediction layer before making a classification 414 about whether an action item candidate is an action item. In some embodiments, layer 412 is a global maximum pooling layer or similar layer that takes a summed vector of multiple values ​​at layer 410 and calculates or outputs a single maximum value. In an illustrative example, with respect to event type detector 432, for each of character sequences 402-1, 402-2 and the remainder of transcription 402, a vectorized value can be generated that indicates a probability or score of whether it is a certain event (e.g., a broadcast). In some embodiments, layer 412 replaces or is otherwise used to replace a fully connected layer of a CNN. Thus, a single feature map of a corresponding category or label can be generated. Instead of adding a fully connected layer on top of the feature map, the average of each feature map can be calculated, and the resulting vector can be fed directly to a softmax layer (e.g., output 414).

[0117] Output 414 is a classification estimate or other prediction indicating whether the predicted character sequence or action item candidate is an action item. In some embodiments, output 414 is a softmax layer (e.g., as described above). A softmax layer is a squeezing function that limits the output of a function to a range of 0 to 1, where the closer the value is to 1, the greater the confidence or probability that the input falls into a certain classification, and vice versa. Softmax layers can also be multi-class sigmoids, meaning that they can be used to determine the probabilities of multiple classes at once. Softmax layers typically have the same number of nodes or neurons as the output layer (e.g., layer 410).

[0118] Now go to Figure 5 , which depicts a process 500 for determining and providing personalized action items in a meeting according to an embodiment of the present disclosure. Process 500 (and / or any function described herein) can be performed by processing logic including hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions running on a processor to perform hardware simulation), firmware, or a combination thereof. Although specific blocks described in the present disclosure are referenced in a specific order and in a specific number, it should be understood that any block can occur substantially in parallel with any other block or before or after any other block. In addition, there may be more (or fewer) blocks than those illustrated. Such added blocks may include blocks that embody any function described herein. The computer-implemented methods, systems (including at least one computing device having at least one processor and at least one computer-readable storage medium) and / or computer storage media described herein can perform or be caused to perform process 500 or any other function described herein. In some embodiments, some or each of blocks 510 to 560 can occur automatically or otherwise in a non-manual (e.g., without an affirmative user device request) manner, which improves the prior art, as described above.

[0119] According to box 510, the content of the meeting is determined (e.g., by the meeting monitor 250). For example, the meeting activity monitor 252 can use one or more sensors or other components to monitor some parts of the chat, the context of the presentation, or the transcription. The context information extractor / determiner 254 can determine the context information of the event, such as who is present or invited to the meeting, the subject of the meeting, the location of the meeting, or other context or character sequences in the transcription or the meeting content itself. Then, the meeting content assembler 256 can generate a rich meeting activity timeline, such as a timeline including each conversation and a tag or structured data indicating when the conversation starts / stops. In some embodiments of box 510, the content is determined from the meeting, which can be determined by monitoring the meeting reception information about the meeting (e.g., transcription information) or other information about the meeting (e.g., attendees, meeting topics, and / or related context information). In some embodiments, box 510 includes: monitoring the meeting to determine the meeting content set. Specifically, in one embodiment, the meeting monitor can receive an indication of each user input in the chat or other online forum, which indicates a live meeting.

[0120] According to block 520, a set of candidate action items is determined (e.g., by action item candidate identifier 262). Embodiments of step 520 determine the set of candidate action items based on the meeting content information determined in step 510. For example, based on the conversation and context information of the meeting, a set of possible action items may be determined. Some embodiments of step 520 may be combined with action item candidate identifier 262 ( Figure 2 In some embodiments of step 530, the content of the meeting may be clarified first, for example, in conjunction with Figure 2 The meeting content clarifier 261 described in conjunction with Figure 2 Additional details of step 520 are described in action item generator 260 in FIG.

[0121] According to box 530, the candidate action items determined at box 520 are classified. For example, CNN or other machine learning models can be used to classify action item candidates as being action items or not being action items, which can all be labels in a classification model, for example. In this way, action item candidates can be excluded or actually become action items. In some embodiments, the classification can be based on the content of the meeting or other contextual information, such as when the model learns contextual information that the user always indicates that a particular phrase is an action item. Therefore, when the phrase is spoken, the probability that the phrase can be classified as an action item is high. An embodiment of step 530 can be as combined with the action item candidate classifier 264 ( Figure 2 ) is performed as described in . Figure 2Additional details of step 530 are described in action item generator 260 in FIG.

[0122] According to block 540, the action items are attributed. Embodiments of step 540 attribute one or more candidate action items to a specific individual, such as a meeting attendee. For example, the action item may be attributed to the meeting participant who is responsible for completing the action item or the meeting participant who is responsible for speaking the action item. Embodiments of step 540 may be implemented in conjunction with action item attributer 266 ( Figure 2 ) is performed as described in . Figure 2 Additional details of step 540 are described in action item generator 260 in FIG.

[0123] According to block 550, the action items are compiled. Embodiments of step 550 compile the action items by relevant action item context (e.g., expiration date of the action item, importance of the action item, etc.), context information (e.g., character sequences in the transcript other than the action item), etc. For example, the documents discussed in the meeting, the portion of the meeting transcript near the action item, or other information may be compiled. Embodiments of step 550 may be implemented in conjunction with action item assembler 269 ( Figure 2 ) is performed as described in . Figure 2 Additional details of step 550 are described in action item assembler 269 in FIG.

[0124] The embodiment of frame 560 provides the action items compiled in step 550. The embodiment of frame 560 can provide the action items intended to be performed by the personnel to a specific person. The action items can also include context information (such as expiration date), and / or other relevant contexts, such as background information, explanatory information, supervision or responsibility information about the execution of the action items for whom to prepare or who is supervising, which can be determined or extracted during the meeting (or from the previous relevant meeting). In this way, the action items can be personalized for the specific person. In some embodiments, frame 560 can also provide the action item set to the supervisor or the personnel who are performing the action item for it. In this way, the embodiment of the technology described herein provides a kind of technical means for monitoring and ensuring accountability. In some cases, only the strongest / highest confidence action items are provided or displayed, and other action items are not provided. In some embodiments, the user can view more action items, including those with less confidence, or the user can configure the setting (via user preference 244) of the threshold value of which items are to be provided for how many action items are to be provided or how high the confidence about the action items is to be used to determine which items are to be provided. In some embodiments of block 560, the action item may be clarified before providing the action item, such as in conjunction with action item clarifier 268 ( Figure 2). For example, redundant or duplicated action items may be removed (deduplicated), action items may be merged (e.g., merged into a viewable list without other meeting content), and consistency of action items may be verified.

[0125] Some embodiments of block 560 may provide a personalized collection of action items to a user who is responsible for completing and / or speaking those action items via a communication message (e.g., email) or within an application (e.g., a communication application, a calendar application, a task list / to-do application, or an online meeting application). Embodiments of block 560 may be implemented as a combination of action item assembler 266 ( Figure 2 ) is performed as described in . Figure 2 Additional details of step 540 are described in action item generator 260 in FIG.

[0126] Now go to Figure 6 , Figure 6 An example process 600 for determining whether a character sequence of an event is an action item is shown according to some embodiments. Process 600 and / or any function described herein may be performed by processing logic including hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions running on a processor to perform hardware simulation), firmware, or a combination thereof. Although the specific blocks described in the present disclosure are referenced in a specific order and in a specific number, it should be understood that any block may occur substantially in parallel with any other block or before or after any other block. In addition, there may be more (or fewer) blocks than those described. Such added blocks may include blocks embodying any function described herein. The computer-implemented methods, systems (including at least one computing device having at least one processor and at least one computer-readable storage medium) and / or computer storage media described herein may perform or be caused to perform process 600 or any other function described herein. In some embodiments, some or each of blocks 602, 604, 606, and / or 608 may occur automatically or otherwise in a non-manual (e.g., without a positive user device request) manner, which improves the prior art, as described above.

[0127] According to box 602, an indication for an event or one or more parts of an event is detected (e.g., by the conference monitor 250 or the natural language sequence normalizer 312). The event includes natural language content discussed in a conference event by one or more event participants or attendees. For example, the event can be a meeting of 3 participants, and a part of the conference event that can be detected is the opening remarks of the first participant. In some embodiments, one or more user devices, sensors, or other devices are queried (e.g., by the conference monitor 250) to determine whether a meeting has started or is about to start, which can be an indication of the event. Therefore, when a conference participant begins to participate in the discussion or when a time point when the meeting will start is identified, an indication of the event can be detected in near real time. For example, using contextual data from a calendar, text, or other data, it can be determined that the meeting will start at 4 pm. Therefore, for example, the conference monitor 250 can cause a control signal to be sent to a microphone device near or online at the meeting location to become activated, at which point, in response to receiving the first speech of a conference participant of the meeting, a conference event is detected or will be detected. In some embodiments, detecting an "indication" of an event includes detecting when it starts. Alternatively or additionally, detecting an indication of an event may include detecting at least a portion of the event or a transcription of the event (eg, detecting when a person utters words to begin a meeting).

[0128] In some embodiments, when the event is completed, a complete transcription or copy of the event can be detected. For example, a device can record the entire conference event and an administrator can upload the conference event to a computing device, which causes natural language text to be output (e.g., via speech-to-text conversion), at which point a transcription can be detected. For illustrative purposes and for the example described below, a complete sample transcription of a meeting event could be as follows: "...how was your weekend. Great...wewent fishing. a goal for X amount in profit next quarter. Ok. Jake, did we ever call the event coordinator? Yes, but I have not got a hold of him. Okay, give him a text this time. that? Yes.(How was your weekend? Great...we went fishing. Looks like Bob's coming. Okay, let's get started. You remember our last meeting we talked about sales numbers. We did well this quarter, but we need to do more. John, how is your business unit doing? We're doing OK, with a profit of X. We definitely need to improve. Set a goal [phone rings, pause], set a goal of making X amount of profit next quarter. OK, Jake, did we call the event coordinator? Yes, but I didn't get through to him. OK, text him this time. Sometimes he doesn't answer the phone. In fact, I'm texting him right now.Jake, before I forget, my daughter needs to be picked up from school too. Can you do that? Yes. )”.

[0129] According to box 604, a set of action item candidates can be determined (e.g., by an action item generator 260 or an action item candidate determiner 321). Some embodiments can calculate a set of action item candidates from multiple character sequences associated with the event. An action item can be a task indicated in a conference event that is requested to be completed to facilitate a specific goal or purpose associated with the conference event. In some embodiments, box 604 includes: tokenizing the transcription of the conference event via natural language processing to clean up or otherwise provide insight into predicted action items (e.g., by a natural language sequence normalizer 312). Tokenization can include: generating part-of-speech tags for multiple character sequences. For example, using the example transcription above, each word may be parsed and tagged with a POS identifier (e.g., "how [interrogative pronoun] was [verb] your [pronoun] weekend [noun]." In this way, grammatical or other phrase structures may be used to identify action item candidates. For example, the typical grammar of an English imperative sentence uses a verb prototype at the beginning of the sentence, without a subject. Thus, in the example transcription, the only sentences or phrases with similar structures may be "Okay, give him a text this time" and "set agoal for [telephone rings, pause] set a goal for X amount in profit next quarter" and "Actually, I'll text him right now. Jake, before I forget, my daughter also needs to be picked up from school. Can you do that...," which may all be identified as candidates.

[0130] In some embodiments, block 604 additionally or alternatively includes: excluding (e.g., by candidate exclusion component 327) action item candidates that may be completed during the conference event and / or excluding action item candidates that are commands or requests that are not related to the conference event. For example, in the example transcription, "actually, I'll text him right now" can be excluded because it is likely to be completed during the meeting. In addition, "can you [pick up my daughter from school]" can also be excluded because it is not related to the meeting.

[0131] In some embodiments, block 604 additionally or alternatively includes excluding repeated or redundant language from the action item candidates (e.g., via the cleaning component 316). For example, using the example transcription above, "Set a goal for [telephone rings, pause] set a goal for X amount in profit next quarter" can be modified to "PERSON will try to sell X amount in profit next quarter" to remove "set agoal" because it is said twice.

[0132] In some embodiments, block 604 includes processing the transcribed character sequence through a word embedding vector model or a semantic model. The word embedding vector model changes natural language characters into numerical vectors (e.g., such as a semantic vector) that represent input to a machine learning model. Figure 4As described). In this way, the vector can be used as input to a machine learning model. A semantic model (e.g., WORD2VEC) can include reconstructing a set of action item candidates by removing, adding, or replacing specific words. The reconstruction indicates what the action item candidates are. In this way, language is cleaned up or otherwise added, removed, or replaced based on the semantic context. For example, using the example transcription, "Set a goal for [telephone rings, pause] set a goal for Xamount in profit next quarter (set a goal for [telephone rings, pause], set a goal for Xamount in profit next quarter)" can be modified to "PERSON will try to sell X amount in profit next quarter (personnel will try to sell X amount in profit next quarter)". This reconstruction can be completed in various embodiments for all text of the above example transcription (or only for identified action item candidates) to clean up the text. As described above, in some embodiments, the reconstruction is based on using a natural language processing component to identify POS tags and semantic context and reconstructing a set of action items based at least in part on the POS tags and semantic context.

[0133] According to box 606, one or more characteristics (also referred to as attributes) of the event and / or event participants are detected (e.g., by the meeting monitor 250 via the user data collection component 210). For example, some or all of the data within the user profile 240, 340 and / or the event profile 370 may be detected, or any other data may be detected as described with reference to the meeting monitor 250. In an illustrative example, one or more user profile characteristics from a set of characteristics may be detected, such as participant preferences of one or more event participants, a participant role of each of the one or more event participants, and a past behavior history of each of the one or more event participants. In some embodiments, the past behavior history includes characteristics such as a tendency to mention action items at specific times or time windows in the event, and patterns in the way participants talk, as described above with respect to Figure 3 In some embodiments, participant preferences include entertainment, hobbies, or other interests.

[0134] In some embodiments, the event profile features may include one or more features of the group including, for example, the name or subject of the event, the location of the event, and event participant data, such as information about Figure 3In some embodiments, the event participant data includes the names of one or more participants in the event and the relationships between them. In some embodiments, the relationships indicate the role hierarchy of one or more participants within the organization, such as described with respect to event profile 370.

[0135] According to box 608, a prediction estimate is indicated as to whether an action item candidate has been determined (e.g., by the action item generator 260 or the classification component 330). The prediction estimate may indicate whether the set of action item candidates is an action item or not an action item. In some embodiments, the prediction at box 608 is automatically generated based at least in part on the detection of one or more features / attributes of the conference event and at least a portion of one or more conference participants (box 606). In some embodiments, a classification estimate including at least a first label and a second label is generated (e.g., via a CNN machine learning model) based at least in part on the detection of one or more attributes of the event or one or more event participants (e.g., via box 606). The first label indicates whether the action item candidate is an action item. The second label indicates whether the action item candidate is not an action item, for example, regarding Figure 4 as described.

[0136] In some embodiments, block 608 may be preceded by or may include: determining a type or classification of the event via a naive Bayes model, thereby further generating a prediction estimate based at least in part on the determination of the type or classification of the event, for example, as to Figure 3 In some embodiments, block 608 may be preceded by or may include: determining via the HMM whether the action item candidate set has a lower probability or a higher probability of being an action item, for example, as described with respect to Figure 3 and / or Figure 4 In some embodiments, automatically generating a prediction estimate includes: using a CNN and also based at least in part on content that is ordered before and after the corresponding action item candidate within the event transcript, for example, as described with respect to Figure 4 In some embodiments, automatically generating a prediction estimate includes: using at least three probability models, such as Figure 3 The naive Bayes model, CNN model, and HMM model of the classification component 330 of the embodiment of the present invention are described. For example, block 608 may include or may be preceded by: determining the type or classification of the event via the naive Bayes model, thereby further generating a classification estimate based at least in part on the determination of the type or classification of the event (e.g., as described with respect to the event type detector 332 and the event partial probability component 334).

[0137] In some embodiments, the prediction estimate or determination of the classification estimate is based at least in part on one or more user profile features as described above with respect to block 606. For example, the classification may be based on participant preferences of one or more event participants, participant roles of each of the one or more event participants, and / or past behavioral history of each of the one or more event participants. For example, using the example transcription above, because "set a goal for X amount of profit next quarter" is determined to be Bob, who is John and Jake's manager, the statement is predicted to be an action item. However, if, for example, past user history shows that the participant always had to set a goal before the end of the meeting each time it was asked in the past, it may not be an action item.

[0138] In some embodiments, the determination of the predictive estimate at block 608 is additionally or alternatively based at least in part on one or more event profile features (e.g., within event profile 370), such as the name or subject of the event, the location of the event, and event participant data. For example, it may be determined that a conference event includes the subject sales, and contextual data (e.g., as described with respect to conference monitor 250) reveals that the discussion about "event coordinator" described in the example transcription is about personal topics unrelated to the conference. Therefore, it may be predicted that the action item candidate of "Okay, give him a text this time" is not an action item based on the conference subject and the contextual data.

[0139] At block 610, a set of action items is generated for one or more conference participants. Embodiments of block 610 generate a set of one or more action items. Action items may be specific to a particular conference participant, in that a particular participant is responsible for performing the action items or seeing that they have been completed. In some embodiments, action items may be generated for users who are not necessarily present at the conference. (The non-present user may be considered a participant because she is still participating in performing the action items from the conference.)

[0140] In some embodiments of box 610, the action item set is generated based at least in part on the prediction estimate of box 608. For example, if there is a high probability or confidence level exceeding a threshold, the action item can be provided to the user, such as via an application or computing service on a user device associated with the user. In some embodiments, the generation of the action item can occur within a user interface or as a part thereof or within or as a part of other data presented to a user device of a specific user. In this way, the generated action item can be displayed on the user device (e.g., by presenting component 220), such as via a user interface, which can be a part of an application or service. In various embodiments, the output can include a displayed list of all predicted or possible action items. For example, using all examples and example transcriptions above, the only action item candidate with a threshold probability of becoming an action item can be "set a goal for X amount of profit next quarter (setting the target of the profit amount of the next quarter to be X)". Therefore, all other parts of the example transcription have been removed or otherwise excluded from becoming action items displayed or otherwise presented. In this way, the amount of content transmitted, presented (e.g., displayed) or stored to the device can be reduced to action items. In various embodiments, possible or predicted candidates may be attributed, sent, clarified, and / or sent to one or more computing applications or computing services associated with a user, and / or a user device associated with a user, e.g., such as described with respect to action item attributer 266, action item clarifier 268, presentation component 220, action item assembler 269, and / or as described above. Figure 8 859. In contrast, non-possible action items or candidates that are not predicted as action items may not be attributed, sent, clarified and / or sent to the user device, which can reduce computing resource consumption, such as memory, CPU, disk I / O, etc. as described above.

[0141] Having described various implementations, an exemplary computing environment suitable for implementing embodiments of the present disclosure is now described. Figure 7 , an exemplary computing device is provided and generally referred to as computing device 700. Computing device 700 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the present disclosure. Neither should computing device 700 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.

[0142] Embodiments of the present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-usable or computer-executable instructions, such as program modules executed by computers or other machines such as personal data assistants, smart phones, tablet PCs or other handheld devices. Generally speaking, program modules including routines, programs, objects, components, data structures, etc. refer to codes that perform specific tasks or implement specific abstract data types. Embodiments of the present disclosure may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. Embodiments of the present disclosure may also be implemented in a distributed computing environment, wherein tasks are performed by remote processing devices linked by a communication network. In a distributed computing environment, program modules may be located in both local and remote computer storage media (including memory storage devices).

[0143] refer to Figure 7 , computing device 700 includes: bus 10 that directly or indirectly couples the following devices: memory 12, one or more processors 14, one or more presentation components 16, one or more input / output (I / O) ports 18, one or more I / O components 20, and an illustrative power supply 22. Bus 10 represents what may be one or more buses (such as an address bus, a data bus, or a combination thereof). Although for clarity, the lines are shown Figure 7 , but in reality, these boxes represent logical, not necessarily actual components. For example, a presentation component (such as a display device) can be considered an I / O component. In addition, a processor has a memory. The inventors recognize that this is a feature of the art and reiterate: Figure 7 The illustrations are merely illustrations of exemplary computing devices that may be used in conjunction with one or more embodiments of the present disclosure. No distinction is made between categories such as "workstation," "server," "laptop," "handheld device," etc., and it is assumed that these categories are Figure 6 and refers to a “computing device”.

[0144] The computing device 700 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 700, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to: RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and can be accessed by the computing device 700. Computer storage media do not include the signal itself. Communication media are typically embodied as computer-readable instructions, data structures, program modules or other data in a modulated data signal (such as a carrier waveform) or other transmission mechanism and include any information delivery medium. The term "modulated data signal" means a signal that has one or more characteristics in its set of characteristics or is changed in a way that encodes the information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0145] Memory 12 includes computer storage media in the form of volatile and / or non-volatile memory. Memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Computing device 700 includes one or more processors 14 that read data from various entities such as memory 12 or I / O components 20. Presentation component 16 presents data indications to a user or other device. In some implementations, presentation component 220 (or 320) of system 200 may be embodied as presentation component 16. Other examples of presentation components may include a display device, a speaker, a printing component, a vibration component, etc.

[0146] I / O port 18 allows computing device 700 to be logically coupled to other devices, including I / O components 620, some of which may be built-in. Illustrative components include microphones, joysticks, game pads, satellite dishes, scanners, printers, wireless devices, and the like. I / O components 20 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, the input may be sent to a suitable network unit for further processing. NUI may implement any combination of the following: voice recognition, touch and stylus recognition, facial recognition, biometric recognition, gesture recognition on and adjacent to the screen, air gestures, head and eye tracking, and touch recognition associated with a display on computing device 600. Computing device 700 may be equipped with a depth camera (such as a stereo camera system, an infrared camera system, an RGB camera system, and a combination of these) for gesture detection and recognition. In addition, computing device 700 may be equipped with an accelerometer or gyroscope capable of motion detection. The output of the accelerometer or gyroscope may be provided to a display of computing device 700 to present an immersive augmented reality or virtual reality.

[0147] Some embodiments of the computing device 700 may include one or more radio units 24 (or similar wireless communication components). The radio unit 24 sends and receives radio or wireless communications. The computing device 700 may be a wireless terminal suitable for receiving communications and media over various wireless networks. The computing device 700 may communicate via wireless protocols such as code division multiple access ("CDMA"), global system for mobile ("GSM"), or time division multiple access ("TDMA") to communicate with other devices. Radio communications may be short-range connections, long-range connections, or a combination of both short-range and long-range wireless telecommunication connections. When we refer to "short" and "long" connection types, we do not mean the spatial relationship between two devices. Rather, we generally refer to short range and long range as different categories or types of connections (i.e., primary and secondary connections). By way of example and not limitation, a short-range connection may include a connection to a device (e.g., a mobile hotspot) or a wireless connection to a wireless device (e.g., a wireless hotspot). A connection that provides access to a wireless communication network (e.g., a WLAN connection using the 802.11 protocol); a Bluetooth connection to another computing device is a second example of a short-range connection, or a near field communication connection. By way of example and not limitation, a long-range connection may include a connection using one or more of the CDMA, GPRS, GSM, TDMA, and 802.16 protocols.

[0148] Many different arrangements of the various components depicted and components not shown are possible without departing from the scope of the appended claims. The embodiments of the present disclosure have been described in a manner intended to be illustrative rather than restrictive. Alternative embodiments will become apparent to readers of the present disclosure after and as a result of reading the present disclosure. Alternative means for achieving the foregoing may be accomplished without departing from the scope of the appended claims. Certain features and sub-combinations are useful and may be used without reference to other features and sub-combinations and are considered to be within the scope of the claims.

[0149] The following embodiments represent exemplary aspects of the concepts contemplated herein. Any of the following embodiments may be combined in a multi-dependency manner to be dependent on one or more other clauses. Furthermore, any combination of dependent embodiments (e.g., clauses that explicitly depend on previous clauses) may be combined while remaining within the scope of the aspects contemplated herein. The following clauses are exemplary and non-limiting in nature:

[0150] Clause 1. A computerized system comprising: one or more processors; and a computer storage memory having computer executable instructions stored thereon, which, when executed by the one or more processors, implement a method comprising: detecting an indication of a conference event, wherein the conference event includes natural language content provided by one or more conference participants in the conference event; calculating or determining a set of action item candidates from multiple character sequences associated with the natural language content, wherein an action item is a task indicated in the conference event and requested to be completed to promote a specific goal or purpose associated with the conference event; detecting one or more features of the conference event and attributes of at least a portion of the one or more conference participants; automatically generating a predictive estimate based at least in part on the detection of the one or more features of the conference event and attributes of at least a portion of the one or more conference participants, the predictive estimate indicating that the set of action item candidates is an action item or is not an action item; and generating an action item set based at least in part on the automatic generation of the predictive estimate and presenting the action item set to one or more user devices associated with the one or more participants.

[0151] Clause 2. A system according to clause 1, wherein the method includes: tokenizing the transcription of the conference event via natural language processing, wherein the tokenizing includes generating part-of-speech tags for the plurality of character sequences.

[0152] Clause 3. According to the system described in Clause 1, the method also includes: excluding action item candidates that may be completed during the conference event, and excluding action item candidates that are commands or requests unrelated to the conference event.

[0153] Clause 4. According to the system of Clause 1, the method also includes: determining the type or classification of the conference event via a naive Bayes model, and wherein the predictive estimate is further generated based at least in part on the determination of the type or classification of the conference event.

[0154] Clause 5. The system of clause 1, further comprising: determining via a hidden Markov model whether the action item candidate set has a low probability or a high probability of being an action item.

[0155] Clause 6. A system according to clause 1, wherein automatically generating the predictive estimate includes using a convolutional neural network and is also based at least in part on content within the transcription of the conference event that is ordered before and after the corresponding action item candidate.

[0156] Clause 7. The system of clause 1, wherein automatically generating the predictive estimate comprises using at least three probability models.

[0157] Clause 8. A computer-implemented method comprising: detecting at least one transcribed portion of an event, the transcription comprising natural language content discussed by one or more event participants in the event; determining a set of action item candidates from multiple character sequences within the transcription, wherein an action item is a task indicated in the event that is requested to be completed to facilitate a specific goal or purpose associated with the event; detecting one or more attributes of the event or the one or more event participants; and generating, via a machine learning model, a classification estimate comprising at least a first label and a second label based at least in part on the detection of the one or more attributes of the event or the one or more event participants, the first label indicating whether the action item candidate is an action item and the second label indicating whether the action item candidate is not an action item.

[0158] Clause 9. The method according to Clause 8 further includes: excluding repeated or redundant language from action item candidates.

[0159] Clause 10. A method according to clause 8, wherein generating the classification estimate is based at least in part on one or more user profile features from a set of features, the features comprising: participant preferences of the one or more event participants, a participant role of each of the one or more event participants, and past behavioral history of each of the one or more event participants.

[0160] Clause 11. A method according to clause 10, wherein the past behavior history includes a tendency to mention action items at specific times or time windows in an event and patterns in the way participants talk.

[0161] Clause 12. The method of clause 10, wherein the attendee preferences include entertainment, hobbies, or other interests, and wherein generating the classification estimate is further based on the entertainment, hobbies, or other interests of the one or more event attendees.

[0162] Clause 13. The method according to Clause 8 further includes: processing the multiple character sequences through a word embedding vector model that converts natural language characters into digital vectors, wherein the digital vectors represent input to the machine learning model.

[0163] Clause 14. The method of clause 8, further comprising determining a type or classification of the event via a naive Bayes model, and wherein the classification estimate is further generated based at least in part on the determination of the type or classification of the event.

[0164] Item 15. One or more computer storage media having computer executable instructions embodied thereon, which, when executed by one or more processors, cause the one or more processors to perform a method comprising: determining meeting content, the meeting content comprising natural language content provided by one or more meeting participants in the meeting; determining a set of action item candidates based on the meeting content; classifying the set of action item candidates as action items or not action items; attributing the classified action items to a set of meeting participants among the one or more meeting participants who are responsible for completing the classified action items or a set of meeting participants who speak the classified action items; compiling the classified action items based at least in part on at least one of: the meeting content and contextual information; and providing the compiled classified action items to one or more user devices associated with the one or more meeting participants.

[0165] Clause 16. The computer storage medium of clause 15, the method further comprising: refactoring the set of action item candidates by removing, adding, or replacing specific words, the refactoring indicating clarification of what the action item candidates are.

[0166] Clause 17. The computer storage medium of clause 15, wherein the reconstructing is based on using a natural language processing component to identify part-of-speech tags and semantic context and reconstructing the set of action items based at least in part on the part-of-speech tags and the semantic context.

[0167] Clause 18. A computer storage medium as described in clause 15, wherein the attribution is based at least in part on one or more event profile features from a set of features, the set of features comprising: a name or subject of the event, a location of the event, and event participant data.

[0168] Clause 19. The computer storage medium of clause 18, wherein the event participant data comprises names and relationships between the one or more participants in the event, wherein the relationships indicate a role hierarchy of the one or more participants within an organization.

[0169] Clause 20. The computer storage medium of clause 15, wherein the method further comprises tokenizing the transcription of the meeting content and generating part-of-speech tags for the plurality of character sequences in response.

Claims

1. A computerized system comprising: one or more processors; as well as A computer storage memory having computer executable instructions stored thereon, which when executed by the one or more processors implement a method comprising: determining a set of action item candidates from natural language content in a transcription of content spoken by one or more conference participants during a conference event, wherein individual action item candidates from the set of action item candidates include a textual description of a task to be completed; identifying a specific participant who spoke the natural language content by machine analysis of meeting characteristics; Reconstructing the text description by removing, adding or replacing specific words in the text description to automatically generate reconstructed action item candidates via a model, wherein the model uses the specific participant as input to generate the reconstructed action item candidates; detecting one or more characteristics of the one or more conference participants; providing the one or more features and the reconstruction action item candidates as input to a machine learning model; processing the input by the machine learning model to automatically generate a prediction estimate indicating whether the reconstructed action item candidate is an action item or not an action item; and Based at least in part on the predictive estimate indicating that the reconstructed action item candidate is an action item, the reconstructed action item candidate is presented to one or more user devices associated with the one or more conference participants.

2. The system according to claim 1, wherein: The method includes generating a set of action items, the set of action items including reconstructed action item candidates having a prediction estimate above a threshold.

3. The system according to claim 1, wherein: The determination of the action item candidate set includes: excluding action item candidates that may be completed during the conference event, and excluding the following action item candidates: the action item candidates are commands or requests that are irrelevant to the conference event.

4. The system of claim 1, wherein the method further comprises: A trained prediction model is received, and the machine learning model is generated by retraining the trained prediction model using historical meeting data associated with one or more of the meeting participants.

5. The system according to claim 1, wherein: The machine learning model is a hidden Markov model.

6. The system according to claim 1, wherein: The machine learning model is a convolutional neural network and is also based at least in part on content within the transcription of the conference event that is ordered before and after the corresponding action item candidate.

7. The system according to claim 1, wherein: The automatically generating the predictive estimate includes using at least three different types of probability models.

8. A computer-implemented method comprising: detecting a transcription comprising natural language content spoken by an event participant during a conference event; Marking the natural language content via a natural language processing (NLP) component; identifying an event participant who spoke the natural language content by machine analysis of meeting features; Responsively generating one or more tags for a plurality of character sequences of the natural language content; determining a candidate set of action items from the plurality of character sequences within the transcription, wherein an action item is a task indicated in the natural language content and requested to be completed; responsive to the determining, generating, via a model, a set of reconstructed action item candidates by automatically reconstructing the set of action item candidates by removing, adding, or replacing specific words from the set of action item candidates, the reconstructing indicating what the set of action item candidates are, wherein the model uses the event participants as input to generate the reconstructed action item candidates; detecting one or more attributes of participants of the event; training a machine learning model to exclude action items describing tasks completed in said meeting event; providing the one or more attributes and the set of reconstruction action item candidates as input to the machine learning model; processing the input via the machine learning model to generate a classification estimate indicating whether a reconstructed action item candidate is an action item; generating a set of action items from a set of reconstructed action item candidates associated with classification estimates above a threshold; and The set of action items is presented to one or more user devices associated with the event participants.

9. The method according to claim 8, further comprising: Eliminate repeated or redundant language from action item candidates.

10. The method according to claim 8, wherein: The generating the classification estimate is based at least in part on the following user profile characteristics: Participant preferences of the one or more event participants, a participant role of each of the one or more event participants, and a past behavior history of each of the one or more event participants.

11. The method according to claim 10, wherein: The past behavior history includes the tendency to mention action items at specific times or time windows during previous events and patterns in how participants talk.

12. The method according to claim 10, wherein: The attendee preferences include recreation, hobbies, or other interests, and wherein the generating the classification estimate is further based on the recreation, hobbies, or other interests of the one or more event attendees.

13. The method according to claim 8, further comprising: The plurality of character sequences are processed by a word embedding vector model that converts natural language characters into digital vectors, wherein the digital vectors represent inputs to the machine learning model.

14. The method according to claim 8, further comprising: A trained prediction model is received, and the machine learning model is generated by retraining the trained prediction model using historical event data associated with one or more of the event participants.

15. One or more computer storage media having computer executable instructions embodied thereon, which when executed by one or more processors cause the one or more processors to perform a method comprising: Determining conference content, wherein the conference content includes natural language content spoken by one or more conference participants in the conference; tagging the transcription of the meeting content via a natural language processing (NLP) component; Responsively generating one or more tags for a plurality of character sequences of the natural language content; associating the participant with one or more of the plurality of character sequences by machine analysis of the conference characteristics; In response to the marking, determining a candidate set of action items based on the plurality of character sequences and participants associated with one or more character sequences of the plurality of character sequences; using a machine learning model to classify the set of action item candidates as action items or not action items, wherein the classification is based on comments made by the participants in past meetings; attributing the classified action item to a set of conference participants among the one or more conference participants responsible for completing the classified action item; assembling the categorized action items based at least in part on: the meeting content, the attribution and contextual information; and Based on the attribution, the compiled categorized action items are transmitted to one or more user devices associated with the set of conference participants, and any other portion of the transcript of the conference content is excluded from transmission to the one or more user devices.

Images