Update and implement documentation from audio transcripts

The method partitions and compares audio and text documents to update text segments with relevant audio content, addressing the inefficiencies of manual updates and enabling automated device control from audio feedback.

JP7590086B2Active Publication Date: 2024-11-26INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2022522011
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-10-12
Filing Date
2020-10-09
Publication Date
2024-11-26
Estimated Expiration
2040-10-09

AI Technical Summary

Technical Problem

Manually updating long text documents such as source code listings and meeting transcripts is time-consuming and prone to errors, and there is no effective method to incorporate verbal changes from subsequent conversations into these documents.

Method used

A method and system that partitions text documents into topic-based segments, converts audio recordings to text, segments the audio into corresponding topics, matches and identifies differences between the text and audio segments, and updates the text document with relevant audio content to generate an updated version, which can be used to modify the behavior of controlled devices.

Benefits of technology

Facilitates quick and accurate incorporation of verbal updates into text documents, improving efficiency and reducing errors in document maintenance, and enabling automated device operation modifications based on audio feedback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590086000001
    Figure 0007590086000001
  • Figure 0007590086000002
    Figure 0007590086000002
  • Figure 0007590086000003
    Figure 0007590086000003
Patent Text Reader

Abstract

The method modifies and utilizes a text document based on an audio file having a captured audio transcript associated with the document text. One or more processors and / or a user segment the text document into topic-based text segments. The processor(s) receive an audio file associated with the document text, convert the audio file to text, and match a specific topic-based text segment from the topic-based text segments to a specific topic-based audio segment from the topic-based audio segments. In response to identifying a difference between the content of the specific topic-based text segment and the content of the specific topic-based audio segment, the processor(s) and / or the user update the content of the specific topic-based text segment with the content of the specific topic-based audio segment to generate an updated version of the text document.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to the field of documents updated with audio recordings of associated minutes, more specifically, the field of updating documents used to control devices based on audio files of audio minutes. [Background technology]

[0002] Text documents, such as source code listings, meeting transcripts, equipment operating manuals, and the like, are often very long. Manually updating such text documents is a time-consuming and error-prone endeavor. Furthermore, when changes are based on subsequent verbal conversation, there is currently no useful method for adequately incorporating such verbal changes into the text document. Accordingly, one or more embodiments of the present invention update text documents with updates from verbal minutes (e.g., from an audio recording of the verbal minutes) quickly and / or automatically.

[0003] Therefore, there is a need in the art to address the above problems. Summary of the Invention

[0004] Viewed from a first aspect, the present invention provides a method for updating a document, the method comprising the steps of: partitioning a text document into topic-based text segments; receiving, by one or more processors, an audio file associated with the text document; converting, by the one or more processors, the audio file to text to generate a text representation of the audio file; partitioning the text representation of the audio file into topic-based audio segments; matching a particular topic-based text segment from the topic-based text segments to a particular topic-based audio segment from the topic-based audio segments, where the particular topic-based text segment from the topic-based text segments and the particular topic-based audio segment from the topic-based audio segments are of the same topic; identifying a difference between content of the particular topic-based text segment and content of the particular topic-based audio segment; determining that a new criterion is satisfied by the particular topic-based audio segment based on the difference between the content of the particular topic-based text segment and the content of the particular topic-based audio segment; and responsive to determining that the new criterion is satisfied by the particular topic-based audio segment, updating the content of the particular topic-based text segment with the content of the particular topic-based audio segment to generate an updated version of the text document.

[0005] Viewed from a further aspect, the invention provides a computer system for updating a document, the computer system including one or more processors, one or more computer readable memories, one or more computer readable non-transitory storage media, and program instructions stored in at least one of the one or more computer readable non-transitory storage media and for execution by at least one of the one or more processors via the at least one of the one or more computer readable memories, the stored program instructions comprising: segmenting a text document into topic-based text segments, the text document controlling operation of a device; receiving an audio file associated with the text document; converting the audio file to text and generating a text representation of the audio file; segmenting the text representation of the audio file into topic-based audio segments; and identifying particular topic-based text segments from the topic-based text segments by a topic-based text representation. The system is executed to perform a method including matching a specific topic-based audio segment from an audio segment, where the specific topic-based text segment from the topic-based text segment and the specific topic-based audio segment from the topic-based audio segment are of the same topic; identifying a difference between content of the specific topic-based text segment and content of the specific topic-based audio segment; determining that a new criterion is satisfied by the specific topic-based audio segment based on the difference between the content of the specific topic-based text segment and the content of the specific topic-based audio segment; in response to determining that the new criterion is satisfied by the specific topic-based audio segment, updating the content of the specific topic-based text segment with the content of the specific topic-based audio segment to generate an updated version of the text document; and performing an action to modify operation of the device utilizing the updated version of the text document.

[0006] Viewed from a further aspect, the present invention provides a computer program product for updating a document, the computer program product comprising a computer readable recording medium readable by a processing circuit and storing instructions for causing the processing circuit to perform a method for performing the steps of the present invention.

[0007] Viewed from a further aspect, the present invention provides a computer program stored on a computer readable medium and readable into an internal memory of a digital computer, the computer program comprising software code portions for performing the steps of the present invention when the program is run on the computer.

[0008] Viewed from a further aspect, the present invention provides a computer program product including a computer readable storage medium having program code embodied thereon, the computer readable storage medium being not itself a transitory signal, the program code being readable by a processor and executable to perform a method comprising the steps of: segmenting a text document into topic based text segments, the text document controlling operation of a device; receiving an audio file associated with the text document; converting the audio file to text and generating a text representation of the audio file; segmenting the text representation of the audio file into topic based audio segments; and matching a particular topic based text segment from the topic based text segments to a particular topic based audio segment from the topic based audio segments, The method includes matching, where the particular topic-based text segment from the topic-based text segment and the particular topic-based audio segment from the topic-based audio segment are of the same topic; identifying a difference between content of the particular topic-based text segment and content of the particular topic-based audio segment; determining that a new criterion is satisfied by the particular topic-based audio segment based on the difference between the content of the particular topic-based text segment and the content of the particular topic-based audio segment; in response to determining that the particular topic-based audio segment satisfies the new criterion, updating the content of the particular topic-based text segment with the content of the particular topic-based audio segment to generate an updated version of the text document; and performing an action to modify operation of the device utilizing the updated version of the text document.

[0009] In an embodiment of the invention, a method modifies and utilizes a text document based on an audio file having a captured audio transcript of a device that the text document controls. One or more processors and / or a user partition the text document into topic-based text segments, where the text document controls the operation of the device. The processor(s) receive an audio file associated with the text document and convert the audio file to text to generate a text representation of the audio file. The processor(s) and / or a user partition the text representation of the audio file into topic-based audio segments, and then match a particular topic-based text segment from the topic-based text segments to a particular topic-based audio segment from the topic-based audio segments, where the particular topic-based text segment from the topic-based text segments and the particular topic-based audio segment from the topic-based audio segments are of the same topic. The processor(s) and / or a user identify differences between content of the particular topic-based text segment and content of the particular topic-based audio segment, and determine that a novel criterion is satisfied by the particular topic-based audio segment based on the differences between content of the particular topic-based text segment and content of the particular topic-based audio segment. In response to identifying differences between the content of the particular topic-based text segment and the content of the particular topic-based audio segment, and in response to determining that the particular topic-based audio segment satisfies the new criteria, the processor(s) and / or user updates the content of the particular topic-based text segment with the content of the particular topic-based audio segment to generate an updated version of the text document. The processor(s) and / or user then perform an action utilizing the updated version of the text document to modify the operation of the device.

[0010] In one or more embodiments, the method(s) described herein are performed by execution of a computer program product and / or a computer system.

[0011] The invention will now be described, by way of example only, with reference to preferred embodiments thereof, as illustrated in the following drawings: [Brief description of the drawings]

[0012] [Figure 1] 1 illustrates an exemplary system and network in which the present invention may be implemented in various embodiments. [Diagram 2] 1 illustrates a high-level overview of one or more components of the present invention. [Diagram 3] FIG. 1 illustrates an exemplary deep neural network for use in one or more embodiments of the present invention. [Figure 4] 1 is a high-level flowchart of one or more steps performed in accordance with one or more embodiments of the present invention. [Diagram 5] FIG. 1 illustrates a cloud computing environment in accordance with an embodiment of the present invention. [Figure 6] FIG. 2 illustrates an abstract model layer of a cloud computing environment according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In one or more embodiments, the invention is a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. In one or more embodiments, the computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0014] A computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. Computer-readable storage media, as used herein, is not to be construed as a transitory signal per se, such as an electric wave, a freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0015] The computer readable program instructions described herein may be downloaded to each computing / processing device from a computer readable storage medium or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in the computer readable storage medium within the respective computing / processing device.

[0016] In one or more embodiments, the computer readable program instructions for carrying out the operations of the present invention include assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented languages ​​such as Java, Smalltalk, C++ or the like, traditional procedural languages ​​such as the C programming language or similar programming languages. In one or more embodiments, the computer readable program instructions execute as a stand-alone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, and in one or more embodiments, the remote computer is connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection is made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, the electrical circuitry executes the computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electrical circuitry, which may include, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), to perform aspects of the invention.

[0017] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer readable program instructions.

[0018] In one or more embodiments, these computer readable program instructions are provided to a general purpose computer, special purpose computer processor or other programmable data processing device to generate a machine such that the instructions, executed via the computer processor or other programmable data processing device, create means for implementing the functions / acts identified in a block or blocks of the flowchart illustrations and / or block diagrams. In one or more embodiments, these computer readable program instructions are also stored on a computer readable storage medium that can, in one or more embodiments, direct a computer, programmable data processing device or other device or combination thereof to function in a particular manner, such that the computer readable storage medium having instructions stored thereon includes an article of manufacture including instructions that implement aspects of the functions / acts identified in a block or blocks of the flowchart illustrations and / or block diagrams.

[0019] In one or more embodiments, the computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to generate a computer-implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement aspects of the functionality / actions identified in a block or blocks of the flowchart and / or blocks.

[0020] The flowchart diagrams and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams represents a module, segment, or portion of instructions, including one or more executable instructions for implementing a particular logic function(s). In some alternative implementations, the functions noted in the blocks occur out of the order depicted in the drawings. For example, two blocks shown in succession are in fact executed substantially simultaneously, or the blocks are executed in reverse order depending on the functionality involved. It should be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams in one or more embodiments of the present invention are implemented by a special-purpose hardware-based system that performs a particular function or action, or executes a combination of special-purpose hardware and computer instructions.

[0021] Referring now to the drawings, and in particular to FIG. 1, there is shown a block diagram of an exemplary system and network that may be utilized by and / or in the implementation of the present invention. It should be noted that some or all of the exemplary architecture, including both the illustrated hardware and software, shown for and within computer 102 may be utilized by artificial intelligence 124 and / or software deployment server 150 and / or text document server(s) 152 and / or physical device 154 and / or controller 145 and / or audio file server(s) 158 and / or Text Updating and Utilization Logic (TUUL) shown in FIG. 2 and / or neurons / nodes shown within deep neural network 324 shown in FIG. 3.

[0022] The exemplary computer 102 includes a processor 104 coupled to a system bus 106. The processor 104 may utilize one or more processors, each having one or more processor cores. A video adapter 108 that drives / supports a display 110 is also coupled to the system bus 106. The system bus 106 is coupled to an input / output (I / O) bus 114 via a bus bridge 112. An I / O interface 116 is coupled to the I / O bus 114. The I / O interface 116 provides communication with various I / O devices including a keyboard 118, a mouse 120, a media tray 122 (which may include storage devices such as CD-ROM drives, multimedia interfaces, etc.), an artificial intelligence 124, and an external USB port(s) 126. The type of ports connected to the I / O interface 116 may be any type known to those skilled in the art of computer architecture, and in one embodiment, some or all of these ports are Universal Serial Bus (USB) ports.

[0023] As shown, the computer 102 can also communicate with the artificial intelligence 124 and / or the software deployment server 150 and / or the text document server(s) 152 and / or the physical device 154 (via the controller 145) and / or the audio file server(s) 158 using a network interface 130 with the network 128. The network interface 130 is a hardware network interface such as a network interface card (NIC). The network 128 may be an external network such as the Internet, or may be an internal network such as an Ethernet, a virtual private network (VPN), etc. One or more examples of the physical device 1543 are provided below.

[0024] A hard drive interface 132 is also coupled to the system bus 106. The hard drive interface 132 interfaces to a hard drive 134. In one embodiment, the hard drive 134 populates a system memory 136, which is also coupled to the system bus 106. The system memory is defined as the lowest level of volatile memory in the computer 102. This volatile memory may include additional higher levels of volatile memory (not shown), including, but not limited to, cache memory, registers, and buffers. Data populated in the system memory 136 includes the operating system (OS) 138 and application programs 144 of the computer 102.

[0025] The OS 138 includes a shell 140 for providing transparent user access to resources such as application programs 144. Generally, the shell 140 is a program that provides an interpreter and interface between a user and the operating system. More specifically, the shell 140 executes commands that are entered into a command line user interface or from a file. As such, the shell 140 is also referred to as a command processor and is generally the highest level of the operating system software hierarchy and functions as a command interpreter. The shell provides the system prompt, interprets commands entered by keyboard, mouse or other user input media, and sends the interpreted command(s) to the appropriate lower level of the operating system (e.g., kernel 142) for processing. It should be noted that while the shell 140 is a text-based, line-oriented user interface, the present invention equally well supports other modes of user interfaces such as graphical, voice, gesture, etc.

[0026] As shown, OS 138 also includes a kernel 142, which includes the lower level functionality of OS 138, including providing essential services required by other portions of OS 138 and application programs 144, where the essential services include memory management, process and task management, disk management, and mouse and keyboard management.

[0027] The application programs 144 include a renderer, shown in an exemplary manner as a browser 146. The browser 146 includes program modules and instructions that enable a World Wide Web (WWW) client (i.e., computer 102) to send and receive network messages to and from the Internet using HyperText Transfer Protocol (HTTP) messaging, thus enabling communication with a software deployment server 150 and other computer systems.

[0028] The application programs 144 in the system memory of the computer 102 (as well as the system memory of the software deployment server 150) also include a Document Modification and Utilization Logic (DMUL) 148. The DMUL 148 includes code that implements the processes described below, including those illustrated in FIGS. 2-4. In one embodiment, the computer 102 may download the DMUL 148 from the software deployment server 150, for example, on an on-demand basis, where the code for the DMUL 148 is not downloaded until needed for execution. Additionally, it should be noted that in one embodiment of the present invention, the software deployment server 150 performs all of the functionality associated with the present invention (including execution of the DMUL 148), thus relieving the computer 102 from using its own internal computing resources to execute the DMUL 148.

[0029] It should be noted that the hardware elements illustrated within computer 102 are not intended to be exhaustive, but rather are depicted to highlight the essential components required by the present invention. For example, computer 102 may include alternative memory storage devices such as magnetic cassettes, digital versatile disks (DVDs), Bernoulli cartridges, etc. These and other variations are intended to be within the scope of the present invention.

[0030] One or more embodiments of the present invention (1) update the text document from the audio recording of the minutes by capturing the text document, which is performed by dividing the text document into a number of small units (e.g., pages or paragraphs on a particular subject), analyzing the small units for topic and keyword information, and indexing the analyzed document in a search engine.

[0031] The system (2) identifies candidate updates by tracking the audio transcripts after converting the audio transcripts using a speech-to-text (STT) program, storing windows of transcribed speech (e.g., 1-2 utterances), creating candidate updates, and analyzing the candidate updates for topics / keywords.

[0032] The system (3) determines whether the candidate update is relevant / important enough to update the document. In one or more embodiments of the invention, the overall update score is a combination of (a) speaker importance and (b) content-specific scores.

[0033] (a) Speaker importance is determined by the speaker's role. For example, if a meeting is about changes to software that controls a particular piece of equipment, meeting participants who are operators or technicians responsible for that particular piece of equipment will be ranked higher in importance than other meeting participants who are unfamiliar with and / or have never operated that particular piece of equipment.

[0034] In embodiments of the present invention, speaker identity and / or role are determined using voice recognition and / or facial recognition along with identity:role matching. For example, if an audio file is part of an audio / video file, speaker facial recognition is used by applying any known facial recognition system that maps facial feature points to a database of known people, etc. Similarly, if the audio file is simply an audio file, speaker recognition is achieved by comparing the speaker's audio clips to a database of similar words spoken by known speakers (e.g., using a Fast Fourier Transform (FFT) to create a digital file of the speaker's words) to thereby identify the particular speaker.

[0035] (b) Using content scores to identify language within a candidate update for inherent importance. For example, language spoken during a meeting such as "change the behavior of a particular device" would be scored higher than "Shall we order lunch?"

[0036] The importance of the speaker will often change if different passages are about different topics, i.e. an engineer talking about operating a particular piece of equipment will be ranked / scored higher than an accountant discussing the operation of that particular piece of equipment, but the accountant will be ranked / scored higher than the engineer if the topic is changing the price terms of a contract to purchase / maintain that particular piece of equipment.

[0037] In one or more embodiments of the invention, the update score for a particular text passage is a speaker-weighted version of the content_score: update_score=speaker_importance*content_score.

[0038] That is, the decision to update a particular passage in the original text document is based on a combination of the importance of the speaker from the audio file with respect to the subject of the passage in the text document being considered for change and the importance of the content in that particular passage. For example, if a speaker is scored as unimportant with respect to a particular passage from the text document, the system will disregard what he or she said with respect to that particular passage. Similarly, if the particular passage itself is judged to be unimportant (e.g., general marketing information in an operating manual, source code that does not affect the operation of a particular device, etc.), it will not be changed. However, if the product of the speaker's importance and the content score is high enough, the passage from the text document will be changed / updated.

[0039] Thus, if a manufacturing operator verbally recommends changes to the operational parameters of a device in a meeting (speaker_importance), and these operational parameter changes have a large impact on how the device functions (content_score), then the changes will be made due to a high update_score.

[0040] In this way, the update_score of the candidate update may be used to automatically trigger an update if it is high enough (above a threshold) or may be reviewed by a human to decide to boost the update. If a decision to update is made, the system will then identify candidate updates of relevant sections from the text document. If not, the candidate update is discarded and the system process returns to step (2) of identifying candidate updates.

[0041] The system (4) identifies relevant sections in the text document by generating a search query based on the analyzed candidate updates, submitting the query to an index for search, retrieving relevant paragraphs (subunits) from the index (passage retrieval), and presenting the retrieved paragraphs to a document updater (an automated module or a human user), i.e., each section (subunit) of the text document is identified according to its subject matter, so that it can be matched to a scored passage from an audio file spoken to the proceedings by a particular participant.

[0042] The system then (5) updates and re-indexes the text document by selecting a section to update with content (either automatically based on the confidence update score above or based on a user's manual selection), comparing the new content with the selected section to see if it represents new information, and if new information is present, adding the content to that section, and updating the index for that section with the added content (re-indexing). That is, once relevant content from the audio file is identified to update a particular passage in the text document, the relevant content is used to modify that particular passage, and the text document is re-indexed to indicate the location of the modification.

[0043] Thus, the system selects a section of the text document to update with the content based on the confidence / update score. The system compares the new content (from the audio file) with the selected section (from the text document) to see if it represents new information using techniques such as contextual analysis of the selected section, topic / phrase detection, etc., to generate an information novelty score to meet the novelty criteria (i.e., does the new information provide significantly different data, and is that significantly different data important?). If new information is present in the audio file, it is added to the selected section. This content inclusion / update is automatic if the update score mentioned above is greater than a predefined threshold. The index of the text document is updated for that selected interval. In an embodiment of the present invention, the index not only indicates which passage of the text document has been updated, but also indicates the new content (from the audio file) in the index.

[0044] In this manner, one or more embodiments of the present invention determine whether a candidate update from an audio file should result in a modification of the document, compare the candidate update to the relevant document section to identify whether it is new information, update and / or modify the relevant document section with only the new information from the candidate update, and / or modify / re-index the text document in real-time as the text document is modified, or any combination thereof.

[0045] Referring now to FIG. 2, a high level overview of one or more components of the present invention is provided.

[0046] As shown in Figure 2, a text update and utilization logic (TUUL) 202 (e.g., computer 102 shown in Figure 1) receives a text document 204 from a text document server(s) 152 described in Figure 1. The TUUL 202 partitions the text document 204 into a number of topic-based text segments, where the number of topic-based text segments includes topic-based text segment 206, topic-based text segment 208, topic-based text segment 210 and topic-based text segment 212. Here, four topic-based text segments are illustrated for illustrative purposes, although it will be understood that the number of topic-based text segments may be greater than or less than four.

[0047] In a first embodiment of the present invention, TUUL 202 first divides the text document 204 into multiple text segments and then determines the topic / subject of each text segment.

[0048] In a second embodiment of the present invention, TUUL 202 examines a section of text to determine the context / content / subject of the section and continues to add subsequent passages of the same context / content / subject to the section.

[0049] That is, in the second embodiment, TUUL202 identifies a passage (e.g., using keyword identification, natural language processing, etc.) as being about a particular topic of the passage. Follow-on passages that immediately follow the passage are then evaluated to determine whether they are about the same topic. If so, these follow-on passages are included in the originally identified particular passage.

[0050] However, in the first embodiment, TUUL 202 first divides the text document 204 into segments of a given nature (e.g., sentences, paragraphs, chapters, etc.) and determines the subject / topic of each segment. Multiple segments related to the same subject / topic are then aggregated so that TUUL 202 can make global changes / modifications to the related segments.

[0051] In a third embodiment, the present invention utilizes a combination of the first and second embodiments to (1) combine subsequent passages on the same topic, and (2) correlate each combined passage with other passages under the same topic, thereby making the correlated passages candidates for being updated with content from audio file 214.

[0052] TUUL 202 then retrieves audio file 214 from audio file server(s) 158 shown in FIG. 1 and segments audio file 214 into topic-based audio segment 216, topic-based audio segment 218, topic-based audio segment 220 and topic-based audio segment 222.

[0053] Similar to the text segments from the text document 204 described above, four topic-based audio segments are shown for illustrative purposes, although it will be understood that the number of topic-based audio segments may be greater than or less than four.

[0054] Similar to the process described for dividing the text document 204 into multiple text segments, the audio files 214 are divided into multiple audio segments based on their subject / topic, according to their topics, subsequent audio passages, predetermined sizes of audio passages, etc.

[0055] However, before the audio file 214 is segmented, it is first converted to a text file (e.g., using a speech-to-text conversion process). Thus, each of the topic-based audio segments shown as part of the audio file 214 is actually a text transcription of an audio passage. TUUL 202 thus divides these text versions of the audio file 214 into segments of a predetermined nature (e.g., sentences, paragraphs, chapters, etc.) and determines the subject / topic of each segment. Segments related to the same subject / topic are then aggregated so that global changes / modifications to the associated segments can be made by TUUL 202.

[0056] Similar to segmentation of the text document 204, the text conversion of the audio file 214 may combine subsequent passages on the same topic and / or correlate each combined passage with other passages under the same topic, thereby making the correlated audio passages candidates for updating the text segment of the text document 204.

[0057] In an embodiment of the present invention, different audio segments are associated with a given text segment based on their matching topic, relevance, importance, etc., as described above, as well as their location within the respective audio files 214 and text document 204.

[0058] For example, assume topic-based text segment 206 is text related to controlling device A (e.g., from source code, operating manuals, etc.), topic-based text segment 208 is text related to controlling device B, topic-based text segment 210 is inventory descriptive text describing the size, color, etc. of device A and / or device B, and topic-based text segment 212 describes the monetary cost of device A and / or device B.

[0059] Further, for purposes of explanation (and not to limit the scope of the invention), assume that topic-based audio segmentation 216 is a text transcription of spoken utterances by person W (e.g., a first operator or design engineer) about how to control device A and device B during the meeting / conference, topic-based audio segmentation 218 is a text transcription of spoken utterances by person X (e.g., an accountant) describing the expenses of device A, topic-based audio segmentation 220 is a text transcription of spoken utterances by person Y (e.g., a second operator or design engineer) about how to control device B, and topic-based audio segmentation 222 is a text transcription of spoken utterances by person Z (e.g., to the meeting / minutes in which audio file 214 was originally captured) asking where participants should go for lunch.

[0060] Further assume that TUUL 202 determines that the topic-based audio segment 216 and the topic-based text segment 206 and topic-based text segment 208 relate to the same subject / topic (i.e., controlling device A or device B or both). As such, TUUL 202 will determine that the topic-based audio segment 216 is a candidate for modifying the topic-based text segment 206 and topic-based text segment 208 depending on the role of person W and the importance of the topic-based text segment 206 and / or the topic-based text segment 208 and / or the topic-based audio segment 216, as discussed herein.

[0061] Similarly, TUUL202 determines whether the topic-based audio segment 218 is a worthwhile candidate for modifying the topic-based text segment 212 (based on both the topic-based audio segment 218 and the topic-based text segment 212 being associated with the cost of device A or device B, or both).

[0062] Similarly, if both the topic-based audio segment 220 and the topic-based text segment 208 are targeted at controlling device B, then assuming the novelty criteria are met (i.e., whether the change is worthwhile, the level of the proposed change, the importance of the change if altering the operation of device B and person Y's role / reputation, etc.), TUUL 202 will determine that the topic-based audio segment 220 is a good candidate for updating the topic-based text segment 208.

[0063] The topic-based text segments 210 would not be candidates for modification because the wording describing the inventory regarding sizes, colors, etc. of device A and / or device B should not be changed.

[0064] Asking who to go to lunch with at a meeting is not related to the content of the text document 204 , so the topic-based speech segmentation 222 is not a candidate for modifying any text segmentation in the text document 204 .

[0065] Now, assume that TUUL 202 determines that the topic-based text segment 206 should be modified by the content of the topic-based audio segment 216 because the content of the topic-based audio segment 216 is different from the content currently found in the topic-based text segment 206 based on the role of the speaker, the criticality of the text segment, the criticality of the spoken audio segment, etc. For example, assume that device A is a unit of rotary equipment and person W, in a meeting after document 204 is generated, says, "Device A will increase its nominal operating speed from 2000 revolutions per minute to 2500 RPM." Thus, having determined that both the topic-based text segment 206 and the topic-based audio segment 216 are related to the operating parameters of device A, TUUL 202 will automatically modify the topic-based text segment 206 to change "2000 RMP" to "2500 RPM." If text document 204 is an operation manual for device A, this change is made therein for the reference of the operator of device A. However, if the text document 204 is source code used by the controller 156 to control the operation of device A (e.g., physical device 154), then TUUL 202 will identify the appropriate source code within the text document 204 to modify the operation of device A (e.g., from comments in the source code, source code known to TUUL as being related to controlling the rate of operation, etc.) and automatically modify this portion of the source code.

[0066] Similar iterations / evaluations / modifications are selectively performed by TUUL 202 on topic-based text segment 208 and / or topic-based text segment 212 using new information (i.e., differences from what is already present in topic-based text segment 208 and / or topic-based text segment 212) obtained from topic-based audio segment 216 and / or topic-based audio segment 220 (for topic-based text segment 208) or topic-based audio segment 218 (for topic-based text segment 212) or both, respectively.

[0067] In one or more embodiments of the present invention, TUUL 202 uses natural language processing (NLP) and / or machine learning (ML) to determine which text segments from the text document 204 should be modified, if necessary, by one or more of the audio segments in the audio file 214 shown in FIG. 2.

[0068] In one or more embodiments of the present invention, TUUL 202 uses artificial intelligence 124 shown in FIG. 1 to determine which text segments from the text document 204 should be modified, if necessary, by one or more of the audio segments in the audio file 214 shown in FIG. 2.

[0069] In one or more embodiments of the present invention, the artificial intelligence 124 utilizes an electronic neural network architecture, such as that found in a deep neural network (DNN) or a convolutional neural network (CNN).

[0070] In a preferred embodiment, a deep neural network (DNN) is used to evaluate textual / numerical data in documents from a text corpus received from the text document server(s) 152, while a CNN is used to evaluate images from an image corpus.

[0071] CNNs are similar to DNNs in that they use interconnected electronic neurons. However, CNNs differ from DNNs in that (1) CNNs have neural layers whose size is based on filter size, stride value, padding value, etc., and (2) CNNs use a convolutional scheme to analyze image data. CNNs are named "convolutional" because they are based on convolution (i.e., a mathematical operation on two functions to obtain a result) of filtering and pooling pixel data to generate a predicted output (to obtain a result).

[0072] The logic units in an electronic neural network (DNN or CNN) are referred to as "neurons" or "nodes." When an electronic neural network is implemented entirely in software, each neuron / node is a separate piece of code (i.e., instructions to perform an action). When an electronic neural network is implemented entirely in hardware, each neuron / node is a separate piece of hardware logic (e.g., processor, gate array, etc.). When an electronic neural network is implemented as a combination of hardware and software, each neuron / node is a set of instructions or a piece of hardware logic, or both.

[0073] As the name implies, neural networks are loosely modeled on biological neural networks (e.g., the human brain). Biological neural networks consist of a series of interconnected neurons that influence each other. For example, a first neuron can be electrically connected to a second neuron by a synapse through the release of neurotransmitters (from the first neuron) that are received by the second neuron. These neurotransmitters can excite or inhibit the second neuron. The pattern of excited / inhibited interconnected neurons ultimately results in biological outcomes, including thoughts, muscle movements, memory retrieval, etc. This description of a biological neural network is highly simplified, but at a high level, the idea is that one or more biological neurons affect the behavior of one or more other biological neurons with which they are biologically and electrically connected.

[0074] Electronic neural networks are similarly composed of electronic neurons. However, unlike biological neurons, electronic neurons are not technically "inhibitory" but are often only varying degrees of "excitatory."

[0075] In an electronic neural network, neurons are arranged in layers known as the input layer, hidden layer(s) and output layer. The input layer contains neurons / nodes that take in input data and send it to a series of hidden layer neurons, where every neuron in one hidden layer interconnects with every neuron in the next hidden layer. The final hidden layer outputs the results of its calculations to the output layer, which is often one or more nodes to hold vector information.

[0076] Referring now to FIG. 3, a deep neural network (DNN) 324 is presented that may be used to evaluate text data (e.g., segments of text documents 204 and / or segments of audio files 214 shown in FIG. 2) for purposes of updated text document segmentation in accordance with one or more embodiments of the present invention.

[0077] For example, for purposes of illustration, assume that the inputs to the DNN 324 are the topic-based text segment 208 from FIG. 2 (referring to controlling device B), the topic-based audio segment 216 (which is a text version of the words spoken by person W about how device B should be operated / controlled), the topic-based audio segment 220 (which is a text version of the words spoken by person Y about how device B should be operated / controlled), and novel criteria 301 (e.g., criteria that must be met to change or update the topic-based text segment 208). Examples of elements of novel criteria 301 include, but are not limited to, common themes found in a particular topic-based text segment and a particular topic-based audio segment, who spoke the words found in the topic-based audio segment, the amount of difference between the content of the topic-based text segment and the topic-based audio segment, the importance of the content found in the topic-based text segment, or the content of the topic-based audio segment, or a combination thereof.

[0078] As shown in FIG. 3, electronic neurons in the DNN 324 are arranged in layers known as input layer 303, hidden layer 305, and output layer 307. The input layer 303 includes neurons / nodes that take in input data and send it to a hidden layer of neurons (e.g., hidden layer 305) where neurons from one layer in the hidden layer are interconnected to all of the neurons in the next layer in the hidden layer 305. The final layer in the hidden layer 305 then outputs the results of the operation to the output layer 307, where the output layer 307 is often a number of nodes each holding vector information. In an embodiment of the present invention, each neuron in the output layer 307 is associated with a specific output to the output 302 as shown in FIG. 3.

[0079] In the example shown in FIG. 3, two outputs in output 302, a first version of an updated topic-based text segment 315 (“Text Segment 315”) and a second version of an updated topic-based text segment 315 (“Text Segment 317”), are generated by DNN 324.

[0080] For example, based on the input presented to input layer 303, text segment 315 may be a modified version of topic-based text segment 208, where source code for controlling the operation of device B (which is topic-based text segment 208 in an embodiment of the present invention), an operating manual (which is topic-based text segment 208 in other embodiments of the present invention), etc., have been modified to reflect changes found in topic-based audio segment 216. Similarly, text segment 317 may be the result of modifying topic-based text segment 208 with the content of topic-based audio segment 220.

[0081] That is, using the operations described herein for the DNN 324, the DNN 324 can determine (based on how well the novelty criteria 301 are met by the topic-based audio segment 216 relative to the topic-based audio segment 220) that content from the topic-based audio segment 216 or content from the topic-based audio segment 220, or a combination of content from the topic-based audio segment 216 and the topic-based audio segment 220, is the best candidate for modifying the topic-based text segment 208.

[0082] Thus, if the DNN 324 determines that who person W is and what he / she said in the topic-based audio segment 216 is more relevant / important to the topic-based text segment 208 than what person Y said in the topic-based audio segment 220, then the topic-based text segment 208 is updated with the content of the topic-based audio segment 216. However, if the DNN 324 determines that who person Y is and what he / she said in the topic-based audio segment 220 is more relevant / important to the topic-based text segment 208 than what person W said in the topic-based audio segment 216, then the topic-based text segment 208 is updated with the content of the topic-based audio segment 220. If the DNN 324 determines that person W and person Y are equally important to the topic-based text segment 208, and what they both said is equally relevant, and what they said does not contradict each other, then content from both the topic-based audio segment 216 and the topic-based audio segment 220 is incorporated into the topic-based text segment 208.

[0083] As noted above, each node in the illustrated DNN 324 represents an electronic neuron, such as the illustrated neuron 309. As shown in block 311, each neuron (including neuron 309) includes a number of features, such as a mathematical function, an output value, and weights and bias values.

[0084] A mathematical function is a mathematical method for processing data from one or more upstream neurons. For example, assume that one or more of the neurons shown in the intermediate hidden layer 305 send data values ​​to neuron 309. Neuron 309 then processes these data values ​​by performing the mathematical function shown in block 311 to create one or more output values, which are sent to another neuron, such as other neurons in the hidden layer 305 or neurons in the output layer 307. Each neuron also has a weight specific to that neuron and / or other neurons connected to it. In addition, bias value(s) are added to the output value(s), where the bias value increases or decreases the output value, allowing the DNN 324 to be further fine-tuned.

[0085] For example, assume that neuron 313 is sending the analysis result of its data to neuron 309. Neuron 309 has a first weight that defines how important the data coming specifically from neuron 313 is. If the data is important, the data coming from neuron 313 is weighted more heavily and / or increased by a bias value, so that the mathematical function(s) in neuron 309 will generate a higher output, which will have a heavier influence on the neurons in the output layer 307. Similarly, if neuron 313 is determined to be important to the operation of neuron 309, the weight in neuron 313 is increased, so that neuron 309 receives a higher value for the output of the mathematical function in neuron 313. Alternatively, the output of neuron 309 may be minimized by decreasing the weight and / or bias used to influence the output of neuron 309. These weights / biases can be adjusted for one, some or all of the neurons in DNN 324, resulting in a confidence output from the output layer 307. Such adjustments are alternatively performed manually or automatically.

[0086] When manually adjusted, the mathematical function(s), output value(s), weight(s) or bias value(s), or combinations thereof, are iteratively adjusted by the user until the output from the output layer 307 matches the expected values. For example, assume that the text segmentation 315 uses content from the topic-based speech segmentation 216 to update the topic-based text segmentation 208, while the text segmentation 317 uses content from the topic-based speech segmentation 220 to update the topic-based text segmentation 208. Further assume that the output 302 outputs the updated text segments in a more reliable / appropriate order. Thus, because the DNN 324 interprets the topic-based speech segmentation 216 as more relevant and important to the topic-based text segmentation 209 than the topic-based speech segmentation 220, the DNN 324 ranks and outputs the text segmentation 315 above the text segmentation 317. However, suppose further that a reviewer of the DNN 324 determines that the keywords, speaker roles, etc. from the topic-based audio segment 220 are indeed better at meeting the novelty criteria 301 than those found in the topic-based audio segment 216. Thus, the mathematical function(s), output value(s), weight(s) or bias value(s), or combinations thereof shown in block 311 are manually adjusted until the text segment 317 is ranked and displayed higher than the text segment 315.

[0087] If automatically adjusted, the mathematical functions, output values, weights and / or biases, or combinations thereof, are adjusted using "backpropagation," where a "Gradient Descent" method determines how each mathematical function, output value, weight and / or bias, or combinations thereof should be adjusted to provide an accurate output 302. That is, the mathematical function(s), output value(s), weight(s) and / or bias value(s), or combinations thereof shown in block 311 are recursively adjusted until the text segment 317 is ranked and displayed higher than the text segment 315, assuming that the new criteria is indeed better met by the topic-based speech segment 220 than the topic-based speech segment 216.

[0088] Referring to FIG. 4, a high level flowchart of one or more steps performed in accordance with one or more embodiments of the present invention is presented.

[0089] After start block 402, one or more processors and / or users partition the text document into topic-based text segments, as described in block 404. In an embodiment of the invention, the text document controls the operation of a device, i.e., the text document is source code that controls an electronic controller of the device, or an operating manual for controlling the device, or both.

[0090] As described in block 406, one or more processors (e.g., processor 104 shown in FIG. 1) receive an audio file associated with the text document. For example, in FIG. 2, audio file 214 is associated with text document 204 because they are both associated with device A or device B or both. In an embodiment of the invention, audio file 214 and text document 204 are considered to be associated with each other based on metadata associated with audio file 214 and text document 204 (e.g., “Device A”, “Device B”, “Operation Control”, etc.), the title of audio file 214 (e.g., “Recommended Changes to Operation of Device A and Device B”) and the title of text document 204 (e.g., “Operation of Device A and Device B”), etc.

[0091] As illustrated at block 408, the processor(s) convert the audio file to text (e.g., using known speech-to-text (STT) algorithms) and generate a text representation of the audio file.

[0092] As described in block 410, the processor(s) and / or the user partition the text representation of the audio file into a number of topic-based audio segments. For example, the text version of the audio file 314 may be partitioned into the topic-based audio segments 216, 218, 220, 222 shown in FIG. 2 based on their content, keywords used, etc., to identify the subject / content of each of the topic-based audio segments 216, 218, 220, 222.

[0093] As described in block 412, the processor(s) and / or user match a particular topic-based text segment from the plurality of topic-based text segments to a particular topic-based audio segment from the plurality of topic-based audio segments, where the particular topic-based text segment from the topic-based text segments and the particular topic-based audio segment from the topic-based audio segments are of the same topic. For example, in FIG. 2, the topic of topic-based audio segment 220 is "controlling device B," which is the same topic as topic-based text segment 208.

[0094] As described in block 414, the processor(s) and / or user identify differences between the content of a particular topic-based text segment and the content of a particular topic-based audio segment. For example, even if topic-based text segment 208 and topic-based audio segment 220 are directed to the same topic of "controlling device B," their contents may differ. That is, topic-based text segment 208 may be an utterance of "operate device B at 2500 RPM," while topic-based audio segment 220 may be an utterance of "operate device B at 2000 RPM."

[0095] The processor(s) and / or the user determine that the novelty criterion is met by the particular topic-based audio segment based on the difference between the content of the particular topic-based text segment and the content of the particular topic-based audio segment, as described in block 416. In the above example, "2000 RPM" is sufficiently different from "2500 RPM" to meet this novelty (i.e., new information) criterion.

[0096] In response to determining that the new criteria are satisfied by the particular topic-based audio segment, as described in block 418, the processor(s) and / or the DNN 324 shown in FIG. 3 updates the content of the particular topic-based text segment with the content of the particular topic-based audio segment to create an updated version of the text document (the first version of the updated topic-based text segment 315 shown in FIG. 3). If the updating is performed by a neural network or similar artificial intelligence, then the updating is performed using the process described above for the DNN 324 shown in FIG. 3. If the updating is performed by the processor 104 shown in FIG. 1, the generation of the first version of the updated topic-based text segment 315 shown in FIG. 3 is created by the processor 104 executing a program that (1) identifies differences between the content of the selected text / audio file segments, (2) determines that the differences are significant (e.g., change the operating value beyond a predetermined range), and (3) updates the text segment with the audio segment of the new content.

[0097] As illustrated in block 420, the processor utilizes the updated version of the text document to perform actions that modify the operation of the device. For example, if the text document is source code, then the TUUL 202 shown in FIG. 2 reprograms the controller 156 with this updated version of the source code to change the operation of the physical device 154 controlled by the controller 156. In the above example, the physical device 154 (e.g., a pump) would then switch from rotating at 2000 RPM to rotating at 2500 RPM. This improves the operation of the device by performing in an optimal manner under a certain set of conditions.

[0098] The flow chart ends at terminal block 422.

[0099] In an embodiment of the present invention, the audio file is generated after the text document, so the content of the audio file modifies the already existing text document.

[0100] In an embodiment of the invention, the processor(s) apply natural language processing (NLP) to the text-based representation of the audio file to identify candidate document updates to the text document. For example, assume now that person W utters, "I think we should speed up the rotation speed of device B by 25%," and the topic-based audio segment 216 is a text version of the words uttered by person W. The NLP will recognize that this utterance (1) is about device B, and (2) is to increase the rotation speed of device B by 25%. Thus, the NLP will evaluate the topic-based text segment 208 and recognize that it (1) is about the operation of device B, and (2) that it is currently operating at a rotation speed of 2000 RPM. Thus, the system will associate this information in the topic-based text segment 208 with increasing the rotation speed of device B to 2500 RPM.

[0101] In an embodiment of the present invention, the processor(s) further determine that a particular topic-based speech segment satisfies the novelty criterion based on the role of the speaker who spoke the particular topic-based speech segment. That is, it is not only the difference in information between what is found in the speech segment and the text segment that indicates that this difference in information is novel (significantly different), as discussed above, but also the role of the speaker, as discussed above, indicates whether the novel / different information is significant. That is, if a pump operator says that the pump speed should be increased by 25%, this statement is given more credence than if a person who does not have a technical or operational role related to the pump said that the pump speed should be decreased by 25%.

[0102] In an embodiment of the present invention, the processor(s) determine an importance score for the particular topic-based audio segment, where the importance score is based on the type of change the particular topic-based audio segment makes to the text document when updating the particular topic-based text segment, and the processor(s) determine that the particular topic-based audio segment satisfies the new criteria based on the importance score for the particular topic-based audio segment. For example, in FIG. 2, if person W recommends increasing the rotation speed of device B by 25% (topic-based audio segment 216) while person Y recommends painting it gray instead of white (topic-based audio segment 220), the topic-based audio segment 216 is assigned a higher level of importance than the topic-based audio segment 220 through the use of NLP or similar logic to identify what the speaker is saying. Thus, since the color of the pump (device B) does not affect its performance, the topic-based text segment 208 is modified to reflect increasing the rotation speed of device B to 2500 RPM, while person Y's suggestion to paint device B gray is ignored.

[0103] In an embodiment of the invention, the processor(s) provide the specific topic-based audio segments to the user for feedback, and based on receiving feedback from the user, update the text document and re-index the updated text document. That is, once the text document 204 shown in FIG. 2 is updated in its entirety using the processes described herein, it is sent to the user for approval feedback. Once the user reviews and approves the changes, the text document 204 is re-indexed to indicate in the index what changes were made and where they are located within the text document 204.

[0104] In an embodiment of the invention, as illustrated in Figures 1 and 2, the text document is source code that, when executed by one or more processors, causes a controller of a device to perform actions that modify the operation of the device.

[0105] In one or more embodiments, the present invention is implemented using cloud computing. Nevertheless, although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings detailed herein is not limited to a cloud computing environment. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or hereafter developed.

[0106] Cloud computing is a model of service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. The cloud model includes at least five characteristics, at least three service models, and at least four deployment models.

[0107] The characteristics are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, as they need it automatically without the need for human interaction with the service provider. Broadband Network Access: Capabilities are available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, PDAs). Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided, but there is a sense of location independence in the sense that it is still possible to specify location at a higher level of abstraction (e.g. country, state, or data center). Rapid Elasticity: Capacity can be rapidly and elastically, in some cases automatically, provisioned to quickly scale out and rapidly released to quickly scale in. To the consumer, the provisioned available capacity often appears externally unlimited, and any amount can be purchased at any time. Metered Services: Cloud systems automatically control and optimize resource usage by utilizing metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active users). Resource usage is monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used. Software as a Service (SaaS): The capability offered to the consumer is to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities with the potential exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The capability offered to a consumer is to deploy consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and possibly the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The capability provided to the consumer is to provide processing, storage, networking, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over selected networking components (e.g., host firewalls).

[0108] The deployment model is as follows: Private Cloud: Cloud infrastructure is used exclusively for one organization. In one or more embodiments, this may be managed by the organization or a third party and may exist on-premise or off-premise, or a combination of these. Community Cloud: Cloud infrastructure is shared by several organizations to support a particular community with common concerns (e.g., mission, security requirements, policy and compliance considerations). In one or more embodiments, it may be managed by the organization or a third party, or may exist on-premise or off-premise, or a combination of these. Public Cloud: The cloud infrastructure is available to the general public or large industry entities and is owned by organizations that sell cloud services. Hybrid Cloud: A cloud infrastructure is a blend of two or more clouds (private, community or public) that remain unique entities but are joined by standardized or proprietary technologies that allow data and application portability (e.g. cloud bursting for load balancing between clouds).

[0109] Cloud computing environments are service-oriented with an emphasis on statelessness, low coupling, modularity and semantic interoperability. At its core, cloud computing is an infrastructure that includes a network of interconnected nodes.

[0110] Referring now to FIG. 5, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 that communicate with local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof. The nodes 10 also communicate with each other. In one embodiment, the nodes may be physically or virtually grouped (not shown) in one or more networks, such as private, community, public, or hybrid clouds, as described above, or combinations thereof. This enables the cloud computing environment 50 to provide infrastructure, platform, or software, or combinations thereof, as a service, for which the cloud consumer does not need to maintain resources on a local computing device. The types of computing devices 54A-54N shown in FIG. 5 are for illustrative purposes only, and it will be understood that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network, a network addressable connection (e.g., using a web browser), or both.

[0111] Referring now to Figure 6, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 5) is shown. It will be understood in advance that the components, layers and functions shown in Figure 6 are for illustrative purposes only and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0112] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, servers based on RISC (Reduced Instruction Set Computing) architecture 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0113] The virtualization layer 70 provides an abstraction layer from which the following examples of virtualized entities are provided in one or more embodiments: virtualized servers 71, virtualized storage 72, virtualized networks including virtual private networks 73, virtualized applications and operating systems 74, and virtual clients 75.

[0114] In one example, the management layer 80 provides the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides tracking of costs as resources are utilized within the cloud computing environment and billing or invoicing for consumption of these resources. In one example, these resources include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and fulfillment 85 provides pre-allocation and procurement of cloud computing resources in anticipation of future demand in accordance with SLAs.

[0115] The workload layer 90 provides examples of functionality for which a cloud computing environment may be utilized in one or more embodiments. Examples of workloads and functionality provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and document update and usage processing 96 that perform one or more of the features of the invention described herein.

[0116] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, it is to be understood that the terms "comprise" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements or components, or combinations thereof, and do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0117] The equivalents of the corresponding structures, materials, acts, and all means or step-plus-function elements in the following claims are intended to include any structures, materials, or acts for performing in combination with other claimed elements as specifically claimed. The description of various embodiments of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed forms. Many changes and modifications will be apparent to those skilled in the art without departing from the scope of the invention. The embodiments have been selected and described in order to best explain the principles and practical application of the invention and to enable those skilled in the art to understand the invention in various embodiments with various modifications suitable for the particular use contemplated.

[0118] In one or more embodiments of the present invention, any of the methods described in this disclosure are implemented using VHDL (VHSIC Hardware Description Language) programs and VHDL chips. VHDL is an exemplary design entry language for field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs) and other similar electronic devices. Thus, in one or more embodiments of the present invention, any of the software-implemented methods described herein are emulated by a hardware-based VHDL program, which is then applied to a VHDL chip such as an FPGA.

[0119] Having thus described in detail the embodiments of the invention in this application and having reference to exemplary embodiments thereof, it will be apparent that modifications and variations are possible without departing from the scope of the invention as defined in the appended claims.

Claims

1. 1. A method for updating a document, comprising: Segmenting a text document into topic-based text segments; receiving, by one or more processors, an audio file associated with the text document; converting, by one or more processors, the audio file to text to generate a text representation of the audio file; segmenting the text representation of the audio file into topic-based audio segments; matching a particular topic-based text segment from the topic-based text segment to a particular topic-based audio segment from the topic-based audio segment, wherein the particular topic-based text segment from the topic-based text segment and the particular topic-based audio segment from the topic-based audio segment are of the same topic; identifying differences between the content of the particular topic-based text segment and the content of the particular topic-based audio segment; determining that a novelty criterion is satisfied by the particular topic-based audio segment based on the difference between the content of the particular topic-based text segment and the content of the particular topic-based audio segment; in response to determining that the particular topic-based audio segment satisfies the new criteria, updating content of the particular topic-based text segment with content of the particular topic-based audio segment to generate an updated version of the text document. The method includes:

2. The text document controls the operation of a device, The method comprises: utilizing the updated version of the text document to perform an action that modifies the operation of the device. The method of claim 1 , comprising:

3. The method of claim 1 or 2, wherein the audio file is generated after the text document.

4. applying, by one or more processors, natural language processing to the textual representation of the audio file to identify candidate document updates for the textual document. The method of any one of claims 1 to 3, further comprising:

5. determining whether the novelty criteria are satisfied by the particular topic-based speech segment based on the role of the speaker who spoke the particular topic-based speech segment; The method of any one of claims 1 to 4, further comprising:

6. determining an importance score for the particular topic-based speech segment, the importance score being based on a type of change that the particular topic-based speech segment would make to the text document when updating the particular topic-based text segment; determining that the novel criteria are satisfied by the particular topic-based audio segment based on the importance score for the particular topic-based audio segment; The method of any one of claims 1 to 5, further comprising:

7. providing said particular topic-based audio segment to a user for feedback; updating the text document based on receiving the feedback from the user; re-indexing the updated text document; The method according to any one of claims 1 to 6, comprising:

8. The method of any one of claims 1 to 7, wherein the text document is source code that, when executed by one or more processors, causes a controller of a device to perform actions that modify the operation of the device.

9. 1. A computer system for updating a document, the computer system including one or more processors, one or more computer readable memories, one or more computer readable non-transitory storage media, and program instructions stored in at least one of the one or more computer readable non-transitory storage media for execution by at least one of the one or more processors via at least one of the one or more computer readable memories, the stored program instructions comprising: Segmenting a text document into topic-based text segments, the text document controlling an operation of a device; receiving an audio file associated with the text document; converting the audio file to text to generate a text representation of the audio file; segmenting the text representation of the audio file into topic-based audio segments; matching a particular topic-based text segment from the topic-based text segment to a particular topic-based audio segment from the topic-based audio segment, wherein the particular topic-based text segment from the topic-based text segment and the particular topic-based audio segment from the topic-based audio segment are of the same topic; identifying differences between the content of the particular topic-based text segment and the content of the particular topic-based audio segment; determining that a novelty criterion is satisfied by the particular topic-based audio segment based on the difference between the content of the particular topic-based text segment and the content of the particular topic-based audio segment; in response to determining that the particular topic-based audio segment satisfies the new criteria, updating content of the particular topic-based text segment with content of the particular topic-based audio segment to generate an updated version of the text document; utilizing the updated version of the text document to perform an action that modifies operation of the device; A computer system configured to perform a method including:

10. 10. The computer system of claim 9, wherein the audio file is generated after the text document.

11. The method comprises: determining whether the novelty criteria are satisfied by the particular topic-based speech segment based on the role of the speaker who spoke the particular topic-based speech segment; 11. The computer system of claim 9 or 10, further comprising:

12. The method according to claim 1, determining an importance score for the particular topic-based speech segment, the importance score being based on a type of change that the particular topic-based speech segment would make to the text document when updating the particular topic-based text segment; determining that the novel criteria are satisfied by the particular topic-based audio segment based on the importance score for the particular topic-based audio segment; The computer system according to any one of claims 9 to 11, further comprising:

13. A computer system according to any one of claims 9 to 12, wherein the text document is source code that, when executed by one or more processors, causes a controller of the device to perform actions that modify the operation of the device.

14. A recording medium storing a computer program for updating a document, the computer program being for causing a computer to execute each step of the method according to any one of claims 1 to 8.

15. A computer program stored on a computer readable medium and readable into an internal memory of a digital computer, the computer program comprising software code portions for performing the method according to any one of claims 1 to 8, when said computer program is executed on said digital computer.

Citation Information

Patent Citations

  • Index creation device, index creation method and index creation program

    JP2004326404A

  • Software management device, and software management method

    JP2014203352A

  • Information processing device and information processing method

    JP2017211430A

  • Programming by voice

    US20190129695A1

  • Speech data summary reproducing device, speech data summary reproducing method, and speech data summary reproducing program

    WO2007132690A1