Automatically capture information from audio data and computer operation context

Abstract

The computer system provides an automated tool that processes audio data and data about computer operation context to automatically capture information (such as information from conversations and meetings). The computer operation context may include, for example, environmental data sensed by the computer, or other contextual information maintained or tracked by the computer. The audio data and computer operation context are processed by the computer to detect the starting conditions for data extraction. After the starting conditions are detected, the processor initiates voice processing to the audio information to generate corresponding text data. The processor processes the text data by using a text filter that defines a prominent pattern in the text data. Such a prominent pattern may correspond to, for example, tasks to be performed, subsequent appointments and other events, messages to be sent, key points or notes, etc. Such automatic processing improves user productivity when tracking valuable information from meetings and other conversations.

Description

[0001] This application is a divisional application of the invention patent application with an international application date of February 9, 2016, a priority date of March 1, 2015, an international application number of PCT / US2016 / 017047, a national application number of 201680013377.8, and an invention name of “Automatically capturing information from audio data and computer operation context”. Background Art

[0002] Actively listening to a conversation (e.g., during a meeting) while taking notes is a difficult skill that most people take years to learn how to do well. Without perfect note-taking skills, people often identify what appear to be important moments in a conversation and then jot down short, vague notes. Such notes often don't make sense out of context later. Other important moments are also missed. At the other extreme, the conversation can be recorded and transcribed. However, searching through the transcript of the conversation from beginning to end to identify important information can be tedious. The data storage requirements for such recordings and transcriptions can also be prohibitive for everyday use in business. Summary of the invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features, nor is it intended to limit the scope of the claimed subject matter.

[0004] The computer system provides an automated tool that processes audio data and data about computer operation context to automatically capture information (such as information from conversations and meetings). The computer operation context may include, for example, environmental data sensed by the computer, or other contextual information maintained or tracked by the computer. The audio data and computer operation context are processed by the computer to detect the starting conditions for data extraction. After detecting the starting conditions, the processor initiates voice processing to the audio information to generate corresponding text data. The processor uses a text filter that defines a prominent pattern in the text data to process the text data. Such a prominent pattern may correspond to, for example, tasks to be performed, subsequent appointments and other events, messages to be sent, key points or notes, etc. Such automatic processing improves user productivity when tracking valuable information from meetings and other conversations.

[0005] In the following description, reference is made to the accompanying drawings, which form a part of the specification and in which are shown by way of illustration specific example implementations of the present technology. It is to be appreciated that other embodiments may be used and structural changes may be made without departing from the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a data flow diagram of an example computer system for automatically capturing information from audio data and computer operating context.

[0007] Figure 2 is a flow diagram describing the operation of an example implementation of a computer system.

[0008] Figure 3 is a diagram of example data structures for starting conditions, and rule structures that may define starting conditions.

[0009] Figure 4 is a flow diagram describing the operation of an example implementation of an activity detection module.

[0010] Figure 5 is a sample graphical user interface for a note-taking application.

[0011] Figure 6 is a block diagram of an example computer in which components of such a system may be implemented. DETAILED DESCRIPTION

[0012] Figure 1 is a data flow diagram of an example computer system for automatically capturing information from audio data and computer operating context.

[0013] Typically, such a computer system processes audio data and data about the computer operating context to automatically capture information (such as information from conversations and meetings). The computer operating context may include, for example, environmental data sensed by the computer or other contextual information maintained or tracked by the computer. The audio data and the computer operating context are processed by the computer to detect the starting conditions for data extraction. After the starting conditions are detected, the processor initiates voice processing of the audio information to generate corresponding text data. The processor uses a text filter that defines a prominent pattern in the text data to process the text data. Such a prominent pattern may correspond to, for example, tasks to be performed, follow-up appointments and other events, messages to be sent, highlights or notes, etc.

[0014] The computer system 100 includes a note taking application 150, which is combined with the following Figure 6The note-taking application 150 may be any application that allows a user to interactively edit an electronic document comprising at least text and optionally other related data by processing user input 155. Such note-taking applications typically allow a user to view and / or store the electronic document. Such user input 155 is received through an input device of the computer. The note-taking application 150 may also access information stored in a storage device of the computer system. The note-taking application 155 also allows a user to read, or listen to, or otherwise use information captured in the note-taking application 150 by presenting such information via an output device of the computer.

[0015] A wide variety of applications can be used as note-taking applications. For example, the note-taking application can be a word processing application. The note-taking application can be a presentation application. The note-taking application can be an application specifically designed to assist in note-taking. The note-taking application can be a simple text editor.

[0016] The note taking application provides note data 152. Note data 152 may be any form of text, image, or other data temporarily stored in memory that a user may direct the note taking application to store in a data file. Note data 152 may include a timestamp, where the timestamp indicates the time when the data was entered into the note taking application by the user.

[0017] The computer system 100 includes one or more audio devices 102 that capture live audio signals and provide audio data 104. The one or more audio devices may be one or more microphones and / or one or more other devices (such as an audio or audio-visual communication system) that can provide audio data to the computer system.

[0018] The computer system 100 also includes one or more sources 106 of computer operating context 110. The computer operating environment helps the computer system identify situations in which the capture of notes may be useful. Specifically, if the computer system can detect that it is likely to be used in a meeting, lecture, talk, or other important conversation, the computer system can activate automatic information capture.

[0019] There are many kinds of sources 106 of computer operating context. One source 106 is a sensor. For example, a location sensor such as a global positioning sensor can provide the location of the computer. As another example, a light sensor can provide information about the ambient light in which the computer is being used. As another example, the sensor is a microphone or other source of audio data. Characteristics of such audio data, such as voice characteristics, can be used.

[0020] Another source of computer context information is an application. As an example, a calendar application may provide information about the date, time, and / or location of a meeting. As another example, the note-taking application 150 or the computer system 100 may also store historical data for a user indicating the times and locations at which the user previously used the computer system for automatic information capture, such as in previous meetings. Such historical data aggregated over multiple users may be provided. As another example, an audio communication application may provide information indicating any active phone calls supported by the application. As another example, the note-taking application 150 may be a source of computer operational context. If the note-taking application 150 is actively used by the user, information capture may be activated.

[0021] To identify situations in which automatic information capture can be activated, the computer system 100 has an activity detection module 108 having a first input for receiving audio data 104 and at least a second input for receiving a computer operating context 110. The activity detection module 108 has an output 112 indicating that a start condition has been detected. The start condition indicates to other modules of the computer system whether those modules can begin processing audio data and other data for a certain event. The activity detection module processes the received computer operating context 110, such as by applying defined rules or filters, to detect such a start condition. This will be described below in conjunction with the present invention. Figure 3 The operation of an example implementation of an activity detection module and example rules are described in further detail.

[0022] The computer system 100 has a speech processing module 114 having a first input for receiving the audio data 104 and a second input for receiving the output of the activity detection module. The speech processing module 114 processes the audio data in response to a second input indicating that a start condition has been detected. The speech processing module 114 has an output 116 that provides a sequence of text data such as words and phrases. The speech processing module can be implemented using a variety of speech recognition technologies. In an example implementation, the output is time-stamped text data. The text data of the output can be provided for further processing by storing the text data of the output in a data file and / or by temporarily storing the text data of the output in a memory.

[0023] The pattern detection module 120 has an input for receiving text data output by the speech processing module. The pattern detection module processes the text data using a text filter 122 that defines prominent patterns in the text data. The pattern detection module may also be responsive to input from the note taking application 150. Such prominent patterns may correspond to, for example, tasks to be performed, follow-up appointments and other events, messages to be sent, highlights or notes, etc. Prominent patterns may be defined, for example, using a grammatical representation such as a rule or other data structure that may be interpreted by the pattern detection module. Figure 4 Examples of such patterns are described in more detail. The output of the pattern detection module is a set of occurrences 124 of prominent patterns. Each occurrence may include an indication of the type of pattern detected and data related to the pattern. Such prominent pattern occurrences may include a timestamp or other data indicating when such a pattern occurs in the audio data or corresponding text data.

[0024] The note taking application 150 may be configured to receive text data from a speech recognition application, and / or to receive the set of occurrences of the salient pattern from a pattern detection module. Each occurrence is represented by data structured in a data format specific to the type of pattern detected (e.g., calendar event, task, message, note, etc.). The data format may be a data format used by a personal information management tool that handles calendar events, task lists, notes, and messages (such as email) for a user. A message such as the following Figure 5 As shown, the note-taking application 150 may include various graphical user interface features that allow the user to view and manipulate the user's own input in conjunction with the received text and highlighting mode. For example, the note-taking application 150 may correlate the user input, the received text, and the received highlighting mode through their respective timestamps.

[0025] Figure 1 The computer system 100 in the embodiment may use one or more of the following Figure 6 There are various deployments in which a first computer (such as a computer used by an individual participating in a meeting) includes at least a note-taking application 150. Other portions of the computer system may reside on one or more other computers, such as a server computer with which the user's computer communicates over a computer network to which both the server computer and the user's computer are connected.

[0026] In one example deployment, the activity detection module and the note-taking application reside on a first computer. When the activity detection module detects a start condition, it begins transmitting audio data to at least one second computer. The second computer includes a speech processing module and a pattern detection module, and the second computer returns the results to the first computer. The source of the computer operating context may reside only on the first computer, or may also include data received from one or more other computers.

[0027] In another example deployment, Figure 1 All units of the system reside on a single computer.

[0028] In another example deployment, only the note-taking application 150 resides on a first computer. The remaining components of the computer system reside on one or more second computers, such as a server computer with which the user's computer communicates over a computer network to which both the server computer and the user's computer are connected.

[0029] Reference now Figure 2 , a flowchart depicting the overall system operation in one implementation of a computer system will now be described.

[0030] First, an activity detection module is activated 200. The activity detection module may be a computer program that typically runs, for example, in a background mode and responds to computer operation context information as it becomes available and / or changes. The activity detection module processes 202 the computer operation context information until a start condition is detected.

[0031] After detecting the start condition, the speech recognition module begins 206 processing the audio data and outputting text data. After the speech recognition generates the text data, the pattern detection module can detect 208 the prominent pattern in the text data. The text data and / or the prominent pattern can then be provided 210 to the note-taking application. In some implementations, such processing can occur in real time as the conversation is occurring; in other implementations, there can be a delay between the system receiving the audio data and processing the audio data to produce the text and prominent pattern output.

[0032] Activity Detection Module

[0033] Now go to Figure 3 , an example implementation of an activity detection module will now be described. In this example implementation, the activity detection module has a set of data 300 that it maintains based on information it receives from a source of computer operating context. Figure 3The data structure shown in is just an example. In any implementation, more or less data or different data can be stored, and the data can be stored in multiple data structures. In general, data 300 can be implemented as a collection of name-value pairs. Depending on the data source, the activity detection module can update the data structure when receiving data from the source of the data or by periodically retrieving and processing the data.

[0034] exist Figure 3 In the example, such computer operation context data may include, for example: Enable data extraction flag 301 may indicate whether a start condition has been detected and whether data extraction is currently in progress. For example, the flag may be set by an activity detector when a start condition is detected. Note-taking application in use flag 302 may indicate whether a note-taking application is currently in use. In one implementation, when a note-taking application is first launched, the note-taking application may set the flag. Other applications may detect that there is an active process for the note-taking application.

[0035] The user's location 304 may be stored, such as with GPS coordinates or other information, such as a conference room name. The ambient light level 306 of the room may be stored. Voice level data 308 extracted from the audio signal may be stored, such as the number of people detected, the amount of time each person spoke, etc. The history of user locations 310 may be a collection of one or more locations, such as with GPS coordinates, conference room names, etc. Next appointment date / time 312 and next appointment location 314 are examples of appointment data that may be stored.

[0036] The active phone call flag 316 may indicate whether another application or device of the user is involved in an active phone call. Other information that may be stored may be business-related information for the user, such as typical work hours and identifiers of co-workers.

[0037] It is also conceivable that the activity detection module has access to the current date and time, but such data is likely to be provided by the operating system of the computer running the module and will not be stored as part of the context data.

[0038] Figure 3 The data shown in can be stored in a single data structure, or can be stored in multiple data structures. The data structure can reside in different types of memory, such as processor registers, input / output registers, main memory, files, etc.

[0039] If data is maintained by the activity detection module, it may typically process such data when changes to such data occur to detect whether a start condition has occurred.The start condition may be detected by applying rules to the stored data. Figure 3An example rule set 350 is shown in FIG. Figure 3 These are examples only and not limitations, as any of a wide variety of rules may be defined depending on the nature of the data maintained by the activity detection module and the conditions under which the system designer intends the system to operate.

[0040] Typically, in a rule set 350, a rule 352 has a set of conditions 354. Each condition 354 includes one or more values ​​358 and one or more operations 360. The operation 360 can be defined, for example, by one or more operators and one or more other values. The simplest rule specifies a variable (e.g., the note-taking application is in use) and an operator (e.g., equals) and another value (e.g., true). If all conditions are met, the rule "fires" or indicates that a start condition has been detected. The activity detection module can process the rules in many ways, such as by testing all rules periodically, or by testing the rules when data changes.

[0041] like Figure 3 As shown in , example rule 352 may have the following condition: If the note-taking application is in use, and the ambient light is less than a threshold, and the user is at a location in the location history. As another example, a rule may have the following condition: The date / time of the next meeting on the user's calendar includes the current time.

[0042] Activity detection can be based on other ways of processing computer operation context data. For example, a scoring algorithm can be applied to generate a score based on the available data. If the score exceeds a threshold, a starting condition can be detected. Such scoring can be defined within the computer program, and more weight can be given to some context data than other context data. For example, low ambient light in the room will increase the score. Whether the current time is within the current working hours can increase the score.

[0043] Other examples of indications that a starting condition may be detected include, but are not limited to: the user is in a location where many people have captured notes in the past; an active phone call is on the user's machine, particularly during the user's normal working hours; the continued presence of one or more intelligible voices is detected simultaneously with an open note page or document, as might occur in an ad-hoc document discussion or review; the continued presence of a single intelligible voice, as might occur in a lecture; any such condition, particularly where ambient light levels are dim, as might occur during a presentation.

[0044] Pattern Detection Module

[0045] Reference now Figure 4, an example implementation of a pattern detection module will now be described in more detail. The pattern detection module receives text data output from the speech recognition module and identifies a prominent pattern. Such a prominent pattern may correspond to, for example, tasks to be performed, follow-up appointments and other events, messages to be sent, highlights or notes, etc. The pattern detection module has pattern data 400 describing the prominent pattern. The pattern data specifies a pattern of one or more text elements 402 that may appear in the text and an action 404 corresponding to the pattern of the text element.

[0046] The specified pattern can be defined, for example, using regular expressions, rules, filters, or using other programming logic. The text element can be an actual word or word type or word category. For example, the word "Monday" is a kind of "day". For example, the expression "[day] near [time]" can be used to detect possible appointments. Such an expression will be triggered by the appearance of a phrase (such as "Let's meet again at 10 am next Wednesday") in the text. The expression can also be written to detect specific phrases, such as: "This is key", "important", "homework", "next time", "follow-up", "action item", "please", etc.

[0047] An action may be any data indicating what is important for a detected pattern and may be used by the pattern detection module to determine what to do with a detected pattern. For example, a pattern may indicate the likelihood of an appointment, event, or other item that may be on a calendar to be scheduled, a task to be performed or other item that may be on a "to do" list, an email to be sent or other item that may be a message to be sent, a note to be stored, or other information item (such as a document) that is important and may be stored.

[0048] In one implementation, if there is text that matches the pattern, the output of the pattern recognition module can be generated based on the text that matches the pattern and the timestamp of the text, and a template corresponding to the type of action associated with the pattern. The template can be a data structure used by other computer programs to represent an item corresponding to the action. For example, a data structure such as an email message, a calendar item, a task or to-do list item, a text document, etc. commonly used in personal information management applications can be used as a template. Such a template can be filled with information based on the text that matches the pattern. For example, a calendar item can be generated with a date and time completed based on the extracted data. The pattern recognition module can output the data structure including the filled template and the timestamp to the note-taking application.

[0049] Note taking application graphical user interface

[0050] Reference now Figure 5 , an example of one aspect of a graphical user interface for a note-taking application will now be described.

[0051] The note taking application may receive data including at least user input, text from a speech recognition module, and information from a pattern recognition module. Each of these inputs may be timestamped. There are a variety of ways in which such information may be communicated to the user. In one implementation of a graphical user interface, such elements may be displayed in different portions of a display, such as a user notes portion, a transcript portion, and an event portion. In one implementation of a graphical user interface, such elements may be displayed on a timeline based on timestamps. A search interface may also be provided to receive user input for searching for content.

[0052] exist Figure 5 In the illustrated implementation, the graphical user interface of the note-taking application includes a display area 500. The display area 500 includes a meeting information portion 502 that provides information about the meeting. The information can be obtained, for example, from a calendar entry stored for the meeting. The note portion 504 is an area that the user can select and, when selected, the user can enter data in. The event portion 506 allows information from the pattern detection module to be displayed. Such events can be displayed in a variety of ways. In this implementation, the icon 508 indicates the type of event, the time 510 indicates the time in the transcript with which the event is associated, and the content 512 includes the following information: such as text extracted from the transcript or other information related to the event. The type of event can be, for example: task, calendar event, message, note, etc. The user can manipulate the event through user input via one or more input devices (such as by manipulating the displayed icon), in response to which the note-taking application can display more information about the event. For example, for a calendar event, the user can be prompted to complete a meeting invitation, which can then be sent to the participants through the calendar application.

[0053] Timeline pane 520 displays a representation of a timeline for the meeting. The user can manipulate the timeline through user input via one or more input devices (such as by selecting points in the timeline). In response, the note-taking application can display a transcript and other information associated with the meeting in another view (not shown), such as user notes and detected events, based on timestamps for such data.

[0054] Such a graphical user interface allows users to take notes and view events and effectively process meeting-related data, thereby improving their productivity. Such storage of information from note-taking applications of multiple participants in a meeting will also improve the productivity of the work group.

[0055] It should be appreciated that the above description is merely one example of an implementation of a computer system that processes audio data and computer operating context to capture information from conversations and conferences. A wide variety of implementations are possible.

[0056] By having a computer system providing an automated tool process audio data and computer operation context to automatically capture information from conversations and meetings, user productivity is improved. Specifically, automatic identification of tasks to be performed, follow-up appointments and other events, messages to be sent, highlights or notes, etc., improves workgroup productivity.

[0057] Accordingly, in one aspect, a computer system includes: at least one audio input device, at least one computer storage medium, and at least one processor, wherein the at least one audio input device and the computer storage medium are configured to receive and store live audio data for processing by the computer system. The computer system is also configured by one or more computer programs to: access data in the computer storage medium that defines a computer operating context of the computer system; detect a start condition by processing the computer operating context from the computer storage medium; after detecting the start condition, process the live audio data to extract at least text and store the text in the at least one computer storage medium; process the extracted text to generate information identifying a prominent pattern in the extracted text in the at least one computer storage medium; and provide the extracted text and the information identifying the prominent pattern to a note-taking application running on the computer system, wherein the note-taking application is configured to receive user input to edit an electronic document in the at least one computer storage medium that incorporates the information identifying the prominent pattern and at least a portion of the extracted text.

[0058] In another aspect, an article of manufacture includes: at least one computer storage medium including at least one of a memory device and a storage device; and computer program instructions stored on the at least one computer storage medium, the computer program instructions configuring the computer system when processed by a computer system. The computer system is configured to: access data in the computer storage medium defining a computer operating context of the computer system; detect a start condition by processing the computer operating context from the computer storage medium; after detecting the start condition, process live audio data to extract at least text and store the text in the at least one computer storage medium; process the extracted text to generate information identifying a prominent pattern in the extracted text in the at least one computer storage medium; provide the extracted text and the information identifying the prominent pattern to a note-taking application running on the computer system, wherein the note-taking application is configured to receive user input to edit an electronic document in the at least one computer storage medium incorporating the information identifying the prominent pattern and at least a portion of the extracted text.

[0059] In another aspect, a process is performed by a computer system comprising at least one audio input device, at least one computer storage medium, and at least one processor. The at least one audio input device and the at least one computer storage medium are configured to receive live audio data and store it in the at least one computer storage medium for processing. The process includes: accessing data in the computer storage medium that defines a computer operating context of the computer system; detecting a start condition by processing the computer operating context from the computer storage medium; after detecting the start condition, processing the live audio data to extract at least text and storing the text in the at least one computer storage medium; processing the extracted text to generate information identifying a highlight pattern in the extracted text in the at least one computer storage medium; providing the extracted text and the information identifying the highlight pattern to a note-taking application running on the computer system, wherein the note-taking application is configured to receive user input to edit an electronic document in the at least one computer storage medium that incorporates the information identifying the highlight pattern and at least a portion of the extracted text.

[0060] In another aspect, a computer system includes means for storing live audio data from an audio input device and means for storing data defining a computer operating context of the computer system. The computer system also includes means for detecting a start condition from the computer operating context. The means for processing the live audio data is operable to extract text from the live audio data in response to detecting the start condition by the means for detecting. The means for processing the extracted text provides information identifying a prominent pattern in the extracted text. The output means provides the extracted text and the information identifying the prominent event to a note-taking application, the note-taking application being configured to receive user input to edit an electronic document incorporating the information identifying the prominent pattern and at least a portion of the extracted text.

[0061] In another aspect, a computer system includes a computer program defining a note-taking application that configures the computer system to at least receive user input, receive extracted text from a speech recognition process applied to live audio data, and receive information identifying salient events in the extracted text. The note-taking application further configures the computer system to allow a user to interactively edit an electronic document incorporating the information identifying salient patterns and at least a portion of the extracted text. The computer system is further configured to provide a computer operating context that controls initiation of the speech recognition process.

[0062] In any of the foregoing aspects, the data defining the computer operating context may include environmental data from sensors (such as ambient light) and / or data indicative of the location of a computer (such as a computer running a note-taking application) and / or other sensor data.

[0063] In any of the foregoing aspects, the data defining the computer operating context may include meeting data, such as including the location of the meeting.

[0064] In any of the foregoing aspects, the note-taking application may have a graphical user interface for presenting extracted text, highlight mode, and user input.

[0065] In any of the foregoing aspects, to automatically process the extracted text to identify salient patterns, the computer system is further configured to apply one or more filters to the extracted text, each of the one or more filters defining a pattern of text corresponding to an action. The action may be any one or more of at least a calendar event, a task, a note, and a message.

[0066] Any of the foregoing aspects may be embodied as a computer system, as any individual component of such a computer system, as a process performed by such a computer system or any individual component of such a computer system, or as an article of manufacture comprising a computer storage device in which computer program instructions are stored and which, when executed by one or more computers, configure the one or more computers to provide such a computer system or any individual component of such a computer system.

[0067] refer to Figure 6 , an example implementation of a general purpose computer will now be described. A general purpose computer is computer hardware configured with a computer program that provides instructions to be executed by one or more processors or other logic devices in the computer. Computer programs on a general purpose computer typically include an operating system and applications. An operating system is a computer program running on a computer that manages access by applications and the operating system to various resources of the computer. The various resources typically include one or more processors or logic devices, memory, storage, communication interfaces, input devices, and output devices.

[0068] Examples of general purpose computers include, but are not limited to, personal computers, game consoles, set-top boxes, handheld or laptop devices (e.g., media players, notebook computers, tablet computers, cellular telephones, personal data assistants, voice recorders), server computers, multiprocessor systems, microprocessor-based systems, programmable consumer electronics devices, networked personal computers, minicomputers, mainframe computers, and distributed computing environments that include any of the above types of computers or devices, and the like.

[0069] Figure 6 An example of computer hardware for a computer is illustrated. Example computer 600 includes at least one processing unit 602 and memory 604. A computer may have multiple processing units 602 and multiple devices implementing memory 604. Processing unit 602 may include one or more processing cores (not shown) that operate independently of each other. Additional co-processing units (such as graphics processing unit 620 or other logic devices) may also be present in the computer; such logic devices may be located in the computer and in Figure 6 Some other components are shown in FIG.

[0070] The memory 604 may include volatile devices (such as dynamic random access memory (DRAM) or other random access memory devices) and non-volatile devices (such as read-only memory, flash memory, etc.), or some combination of the two. The computer 600 may include additional storage devices (removable and / or non-removable), including but not limited to magnetic or optical recording disks or tapes. Figure 6 Such additional storage is illustrated in FIG. 6 by removable storage 608 and non-removable storage 610 . Figure 6 The various components in are generally interconnected by an interconnection mechanism such as one or more buses 630 .

[0071] Computer storage media is any medium in which data can be stored by a computer and retrieved from an addressable physical storage location. Computer storage media include volatile and non-volatile storage devices, and removable and non-removable storage media. Memories 604 and 606, removable storage 608, and non-removable storage 610 are all examples of computer storage media. Some examples of computer storage media are RAM, ROM, EEPROM, flash memory, processor registers or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical or magneto-optical recording storage devices, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Computer storage media and communication media are mutually exclusive categories of media.

[0072] The computer 600 may also include (one or more) communication connections 612 that allow the computer to communicate with other devices through a communication medium. The communication medium transmits data, such as computer program instructions, data structures, program modules, or other data, through wired or wireless materials by propagating a modulated data signal such as a carrier wave or other transport mechanism on the material. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in a manner such as encoding information in the signal, thereby changing the configuration or state of a receiving device of the signal. By way of example and not limitation, communication media includes wired media such as a wired network or a direct wired connection, and wireless media includes any non-wired communication media that allows signals (such as acoustic, electromagnetic, electrical, optical, infrared, radio frequency, and other signals) to propagate. The communication connection 612 is a device such as a wired network interface, a wireless network interface, a radio frequency transceiver (e.g., a Wi-Fi, cellular, Long Term Evolution (LTE) or Bluetooth transceiver), a navigation transceiver (e.g., a Global Positioning System (GPS) or a Global Navigation Satellite System (GLONASS)), or a transceiver that interfaces with a communication medium to send data on the communication medium and receive data from the communication medium.

[0073] In the computer, example communication connections include, but are not limited to, wireless communication interfaces for wireless connection to a computer network, and one or more radio transmitters for telephone communication over a cellular telephone network, and / or. For example, there may be a WiFi connection 672, a Bluetooth connection 674, a cellular connection 670, and other connections 676 in the computer. Such connections support communication with other devices. One or more processes may be run on the computer and managed by the operating system to enable voice or data communication over such connections.

[0074] The computer 600 may have various (one or more) input devices 614, such as touch-based input devices, pens, cameras, microphones, sensors such as accelerometers, gyroscopes, thermometers, light sensors, etc., and the like. Output devices (one or more) 616, such as displays, speakers, etc., may also be included. All of these devices are well known in the art and need not be discussed exhaustively here. The various input and output devices may implement a natural user interface (NUI), which is any interface technology that enables a user to interact with a device in a "natural" manner without the artificial constraints imposed by input devices such as a mouse, keyboard, remote control, etc.

[0075] Examples of NUI methods include those that rely on speech recognition, touch and stylus recognition, gesture recognition on and adjacent to the screen, mid-air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence, and may include use of touch-sensitive displays, voice and speech recognition, intent and goal understanding, motion gesture detection using depth cameras (such as stereo camera systems, infrared camera systems, and other camera systems and combinations of these), motion gesture detection using accelerometers or gyroscopes, facial recognition, three-dimensional displays, head, eye and gaze tracking, immersive augmented reality and virtual reality systems (all of which provide more natural interfaces), and techniques for sensing brain activity using electric field sensing electrodes (EEG and related methods).

[0076] The various storage devices 610, communication connections 612, output devices 616, and input devices 614 may be integrated into the housing along with the rest of the computer hardware, or may be connected via various input / output interface devices on the computer, in which case reference numerals 610, 612, 614, and 616 may indicate either interfaces for connecting to the devices or the devices themselves, as appropriate.

[0077] Each component of a computer system such as described herein and operating on a computer (which may also be referred to as a "module" or "engine", etc.) may be implemented using one or more processing units or logic devices of a computer and one or more computer programs processed by the computer. In general, such a module has inputs and outputs through locations in a memory or processor register, from which data can be read or to which data can be written when the module is executed by a computer. A computer program includes computer-executable instructions and / or computer-interpreted instructions, such as program modules, which are processed by a computer. Typically, such instructions define routines, programs, objects, components, data structures, etc., which, when processed by a computer, command the computer to perform operations on data or configure the computer to implement various components or data structures.

[0078] Alternatively or additionally, the functionality of one or more of the various components described herein may be performed at least in part by one or more hardware logic components. For example and without limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), program-specific integrated circuits (ASICs), program-specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0079] It should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific embodiments described above. The above-described specific embodiments are disclosed by way of example only.

Claims

1. A computer system comprising: at least one audio input device, at least one computer storage medium, and at least one processor, wherein the at least one audio input device and the computer storage medium are configured to receive and store live audio data for processing by the computer system; The computer system is further configured by a computer program in a computer storage medium to: using the processor to determine when to begin processing the live audio data and other data derived from the live audio data; After you decide to start processing: processing the live audio data using the processor to extract at least text and storing the text in the at least one computer storage medium; processing the extracted text using the processor to generate information in the at least one computer storage medium identifying salient patterns in the extracted text; as well as The extracted text and the information identifying the highlighting pattern in the computer storage medium are provided for access by a note-taking application running on the computer system, wherein the note-taking application running on the computer system is configured to receive user input to edit an electronic document in the at least one computer storage medium, the electronic document incorporating the extracted text and the information identifying the highlighting pattern. 2 . The computer system of claim 1 , wherein to determine when to start processing, the computer system is further configured to detect a start condition based on data defining a computer operating context.

3. The computer system of claim 2, wherein the data defining a computer operating context includes data indicative of ambient light.

4. The computer system of claim 2, wherein the data defining a computer operating context includes data of a computer location.

5. The computer system of claim 2, wherein the data defining the computer operating context comprises meeting data including a location of a meeting.

6. The computer system of claim 1, wherein the note-taking application has a graphical user interface for presenting the extracted text, highlight mode, and user input.

7. A computer system according to claim 1, wherein in order to automatically process the extracted text to identify prominent patterns, the computer system is also configured to apply one or more filters to the extracted text, each of the one or more filters defining a pattern of text corresponding to an action.

8. At least one computer storage medium storing computer program instructions that, when processed by a computer system, configure the computer system to perform the following actions, the computer system comprising at least one audio input device, the at least one computer storage medium, and at least one processor, wherein the at least one audio input device and the at least one computer storage medium are configured to receive live audio data and store the live audio data in the at least one computer storage medium for processing: determining when to begin processing the live audio data and other data obtained from the live audio data; After determining to start processing the live audio data: processing the live audio data using the processor to extract at least text and storing the text in the at least one computer storage medium; processing the extracted text using the processor to generate information in the at least one computer storage medium identifying salient patterns in the extracted text; as well as The extracted text and the information identifying the highlighting pattern in the computer storage medium are provided for access by a note-taking application running on the computer system, wherein the note-taking application running on the computer system is configured to receive user input to edit an electronic document in the at least one computer storage medium, the electronic document incorporating the extracted text and the information identifying the highlighting pattern.

9. The medium of claim 8, wherein to determine when to begin processing, the computer system is further configured to detect a start condition based on data defining a computer operating context.

10. The medium of claim 9, wherein the data defining a computer operating context includes data indicative of ambient light.

11. The medium of claim 9, wherein the data defining a computer operating context includes data of a computer location.

12. The medium of claim 9, wherein the data defining the computer operating context comprises meeting data including a location of a meeting.

13. The medium of claim 8, wherein the application includes a graphical user interface for presenting the extracted text, highlight mode, and user input.

14. The medium of claim 8, wherein to automatically process the extracted text to identify salient patterns, the computer system is configured to apply one or more filters to the extracted text, each of the one or more filters defining a pattern of text corresponding to an action.

15. A method performed by a computer system, the computer system comprising at least one audio input device, at least one computer storage medium, and at least one processor, wherein the at least one audio input device and the at least one computer storage medium are configured to receive live audio data and store it in the at least one computer storage medium for processing, the method comprising: determining when to begin processing the live audio data and other data obtained from the live audio data; After you decide to start processing: processing the live audio data to extract at least text and storing the text in the at least one computer storage medium; processing the extracted text to generate in the at least one computer storage medium information identifying salient patterns in the extracted text; as well as The extracted text and the information identifying the highlighting pattern are provided to a note-taking application running on the computer system, wherein the note-taking application is configured to receive user input to edit an electronic document in the at least one computer storage medium, the electronic document incorporating the extracted text and at least a portion of the information identifying the highlighting pattern.

16. The method of claim 15, wherein determining when to begin processing comprises: A starting condition is detected based on data defining a context in which the computer operates.

17. The method of claim 16, wherein the data defining a computer operating context includes data indicative of ambient light.

18. The method of claim 16, wherein the data defining a computer operating context includes data of a computer location.

19. The method of claim 16, wherein the data defining the computer operating context comprises meeting data including a location of a meeting.

20. The method of claim 15, wherein to automatically process the extracted text to identify salient patterns, the method further comprises: One or more filters are applied to the extracted text, each of the one or more filters defining a pattern of text corresponding to an action.

Images